Skip to main content
SearchLoginLogin or Signup

Thinking with semiotic-dialogism: Re-orientating augmented reality and visual literacy

Full paper

Published onMar 14, 2022
Thinking with semiotic-dialogism: Re-orientating augmented reality and visual literacy
·

Abstract

Augmented reality (AR) apps, like Adobe’s Aero, enable users to turn Photoshop layers into interactive AR experiences and are considered promising for higher education. But what we see or do not see are mediated via histories, cultural values, ideologies, social practices and technologies. Simultaneously, the ways we receive knowledge, communicate and learn are more than ever being communicated via visual technologies. Yet, theories of visuality within educational research represent a longstanding gap within scholarship and theorising of visual technologies, including AR, is lacking. This study re-orientates conceptions of AR visual literacy through ‘thinking with’ semiotics, which is the study of signs, images, sounds or any phenomena communicating meaning (Peirce, 1908). Semiotics is synthesised with dialogism, defined as the exchange of texts, perspectives and voices (Bakhtin, 1986). The semiotic-dialogic framework is applied to a series of AR exhibits at Adobe’s (2020) Festival of the Impossible. The analysis re-orientates commercialised conceptions of AR pedagogy to reveal that, while AR experiences can be developed without coding knowledge, they still require visual literacies.

Keywords: Augmented reality; visual technologies; visual literacies; semiotics; dialogism; collaborative learning; thinking with

Part of the Special Issue Visual literacies and visual technologies for teaching, learning and inclusion

1. Introduction

It is claimed that augmented reality (AR) supports collaborative learning and overcomes the barriers of outdated teaching methods (Martín-Gutiérrez et al., 2017). However, definitions of collaborative learning activities or what is meant by pedagogic ‘barriers,’ in relation to AR learning are thin. In this study, the semiotic philosophy of the American pragmatist Charles Sanders Peirce (1839–1914) and the dialogism of Mikhail Bakhtin (1895–1975) are synthesised to help re-orientate conceptions of AR visual literacy. This offers a framework for the review of literature concerning AR pedagogy and visual literacy. It also develops a novel framework for visual inquiry of AR exhibits at Adobe’s (2020) Festival of the Impossible. Theorising AR, as semiotic and dialogic, or as ‘semiotic-dialogism’, could contribute to expanding conceptions of AR visual literacy. The term ‘semiotic-dialogism’ is utilised to convey to the reader a sense of the multiple meanings afforded by visual technologies. Moreover, it suggests that there is not one way of looking at something and even visual phenomena that appear identical could be interpreted in numerous ways. This study is important since it refines theoretical, commercialised and normative conceptions of AR visual literacy.

From a Peircean perspective learning is defined as always semiotic and as occurring, not only linguistically, but through a broad range of signs (Petrilli, 2014). Signs include, but are not limited to, written and spoken language, images, sounds, temperatures or anything communicating meaning. It could help to go beyond normative and commercialised assumptions that AR visual literacy is simply a matter of triggering AR codes; consuming visual media; or following a series of instructions to overlay stock animated images, icons and symbols (Adobe, 2020). Peircean semiotics corresponds with Bakhtin’s (1986; 1999) theory of dialogism which positions social actors as interpreters in dialogic exchange (Petrilli, 2014). Dialogic pedagogy has also been developed to conceive of learning as a range of perspectives, texts and voices in social interaction (Matusov, 2009). Through the fusion of semiotic-dialogism, the study asks: ‘To what extent could semiotic-dialogic inquiry help to open understandings of AR visual literacy?’ To consider these issues, I first begin with a thematic literature review to consider how AR has been positioned in relation to learning and the fuzzy concept of visual literacy. Second, I provide inquiry into two exhibits from Adobe’s (2020) Festival of the Impossible, as a case to illustrate semiotic-dialogic inquiry.

2. Literature review

The literature was selected from a range of pedagogic journals and publications. Key word searches included AR and education; AR pedagogy; AR collaboration; AR teaching and learning; AR visual literacy. A total of 25 articles were analysed from a semiotic-dialogic perspective which placed an emphasis on understanding the semiotic and dialogic aspects of AR pedagogy. The analysis followed eight steps proposed by Tesch (1990): (1) capturing the essence of the entire data; (2) picking one document and considering its essential sense, followed by jotting down concepts; (3) listing all topics, clustering comparable topics, and making columns to differentiate between key, exceptional, and discarded topics; (4) coding the text; (5) uncovering the most descriptive phrasing for the topics and classifying them into categories; (6) abbreviating each category and alphabetising the codes; (7) compiling the codes and making initial analysis; and (8) recoding, if needed. The results showed a diverse range of manuscripts published in the journal databases including research articles, reviews, technical notes, features, and news. Types of participants using AR discussed in these publications included consumers, university students, primary students, secondary students, teachers, and children with special needs.

The literature indicates that AR is currently configured through a range of technological semiotic and multimodal resources, which include the combination of video, text, visual effects, animated 3D objects, shapes, emojis and filters that can be overlaid on surfaces, spaces and landscapes. For example, the furniture chain IKEA have developed an AR application to allow users to model their products through augmenting items over work and living spaces via a mobile device, see Figure 1.

Figure 1: “Ikea Place App” CC BY 2.0 (Hillary: Creative Commons, 2021a)

Outside of formal education AR social media digital filters are popular with younger users, for example animal masks, glitter, make-up and hair effects on apps like Snapchat and Instagram (Eisenbrand & Peterson, 2019). However, AR researcher Speicher (2018) suggests that while AR has been hyped by technology companies there is a lack of real use in education. Nevertheless, pedagogic literature proposes a prominent role for AR in education through improvement of students’ knowledge and understanding of materials. Educational literature suggests that AR is considered effective in: supporting situated learning; creating student-centered learning; useful for peer-teaching, improved teamwork among students; and, allowing teachers to mentor (Kamarainen et al., 2013). It is noted that AR allows students to make use of mobile devices in learning and Martín-Gutiérrez et al. (2015) stated that when using AR applications, teachers do not need to repeat instructions since students enjoy AR’s ability to assist them in learning. AR materials are considered to increase students’ motivation and concentration (Yen et al., 2013). Akçayır et al., (2016) disclosed that for physics students, AR was effective in enhancing laboratory skills and creating positive attitudes to physics laboratories.

A further noticeable role of AR in relation to learner outcomes was enhancing students’ knowledge and understanding in different subjects. In mathematics, AR was viewed as facilitating students’ comprehension since it offered a more interesting visualization and interface (Coimbra et al., 2015). More broadly, it is claimed that AR enhances learners’ enjoyment (Akçayır & Akçayır, 2017); offers conveniences during the learning process (Zhu et al., 2012); and engages students in learning (Akçayır & Akçayır 2017; Kamarainen et al., 2013). The literature suggests that AR has a lot of potential in education but also several challenges, such as technical problems related to AR operation (Sungkur et al., 2016), the new development of AR (Zhu et al., 2014), and students’ lack of skills, experiences, and tools required to operate AR (Akçayır et al., 2016).

Yet, a number of papers suggest that AR presentations are increasingly easy to create, share and consume via cloud-based platforms. Martín-Gutiérrez et al., (2014), for example, suggest that AR contributes to computer supported collaborative learning which is a pedagogical approach that can be used for deploying educational apps based on AR in higher education. They suggest that, “outdated teaching creates barriers for some students that are used to interacting with modern technological gadgets and computers” (Martín-Gutiérrez et al., 2014, p. 760). But, despite these bold claims, their definition of collaborative learning, why AR could support it, or what they mean by pedagogic ‘barriers’ remain arguably thin.

Other AR pedagogic literature emphasises the psychological and motivational factors of AR (Solak & Cakir, 2015; Di Serio et al., 2013). This indicates AR’s immersive entertainment of learners through play, technology and the novelty value of AR visuality. These AR studies indicate the pedagogic affordances for increasing reading comprehension, concretizing abstract concepts (Dori & Belcher, 2005) and the development of critical thinking (Dunleavy, et al., 2008). However, what is meant by critical thinking is vague and presented in normative and arguably non-critical terms. AR pedagogists, Mahadzir and Phung (2013, p. 34) discuss AR’s affordances for academic reading and suggest it increases students’ performances by providing an inspiring learning environment. They state AR contributes to, perceptual arousal, variability, goal orientation, motive matching, familiarity, learning requirements, success opportunities, personal control, intrinsic reinforcement, extrinsic rewards, and equity. But the AR contribution to perceptual arousal could potentially lead to semiotic-dialogic question formations, yet Mahadzir and Phung’s (2013) focus on fixed affordances leaves little scrutiny or discussion of the limitations, cultural or individual differences.

Dunleavy (2014) perspective of AR rectifies this to a certain extent. He suggests that AR can be viewed as a cognitive tool, when combined with pedagogical approaches, and situated within constructivist learning theory. However, although Dunleavy claims that AR has cognitive and constructivist affordances, its limitations include cognitive overload and the challenge of integrating and managing the overall AR experience from teachers’ perspectives. I therefore suggest that this does not provide a convincing rationale for their constructivist stance and a privileging of the teacher’s role in pedagogy cannot be aligned with dialogic pedagogy (Matusov, 2009). Furthermore, understandings of AR’s configurations in terms of semiotic resources, including audio, video, text and filters, technologies, represent a gap in pedagogic scholarship and research.

Subsequently, ways in which AR provides learners with a mixture of communication tools, and the visual literacies involved, require clearer articulations of the impacts upon learning. This brief review indicates that scholarship of new epistemologies and ontologies occurring visually via AR represent a significant lacuna despite the positive assumptions of AR technologies as a solution for enhancing learning. Next, I briefly discuss some of the varying conceptions of visual literacy that might provide more nuanced insights into AR.

A review of visual literacy scholarship indicates that it is a broad multidisciplinary field with a variety of definitions. This is hardly surprising considering visual artefacts have been used throughout human histories and concepts of ‘visual literacy,’ despite the terms’ modern connotations, are nothing new (Pettersson, 1989). Kędra (2018) suggests that, for those who can see, seeing is one of the most natural acts since socialising and learning occurs by observation and acting. But within mediated visuality, our gaze is trained in terms of scopic regimes. Scopic regimes occur in psychological, cultural and historical terms and visual literacy is not a naturally occurring competency, acquired through frequently encountering images of various kinds (Mirzoeff, 2006).

Arguably, our experience and perception of the world, choices that we make, individual preferences and fears are greatly organized by what we see, do not or cannot see, but ways of seeing are underpinned by cultural and historical practices (Berger, 1972). While visual literacy definitions mostly use a metaphor of visual reading and writing, not all visual skills easily undergo this categorization (Kędra, 2018). This tendency may further indicate that visual literacy is a failed metaphor, as already suggested by Cassidy and Knowlton (1983). It is also rooted in essentialist notions of what visual literacy already is, could and should be.

In this paper the concept of visual literacy is developed to consider how visual literacy involves visual meanings and learning in terms of signs via semiotic-dialogic interactions. Bakhtin and Peirce both viewed meaning making as occurring through interconnections of words, form, content and further chains of meaning. Peirce’s philosophy of signs extends far beyond words, speech acts, linguistics, literary genres, and/or indeed human activity, to include all elements, or signs, that communicate meaning. I draw on Peirce’s (1908) theory of semiosis, which is the notion of how signs, or representamen (elements, words, images, etc.); objects (the meanings to which signs refer); and interpretants (processes of interpreting and creating further chains of meaning) occur simultaneously. This triadic theory of meaning is developed as a semiotic-dialogic perspective. It provides a promising framework for understanding visual thinking and learning, beyond ‘reading’ to include affective, interpretative, social and conceptual dimensions of AR visual literacies. In light of the literature review, in the next section I discuss the semiotic-dialogic inquiry of the study.

3. Folds of inquiry

In this paper, the semiotic-dialogic framework has been developed through drawing on the literature and theory underpinning semiotics and dialogic inquiry. As the author of the study, I draw on past studies as well as my own previous fieldwork carried out while working with students using the now obsolete AR app ‘Aurasma’ (Hurley, 2016). My earlier research iterations were concerned with exploring how the implementation of AR to pedagogy could promote collaboration between learners. This concern was motivated by critical pedagogy and theorists like Freire (1970) who suggest that education can only be considered dialogic, and thereby having significance, when learning provides collaborative openings for transformation and praxis. Freire (1970) views the social systems and processes of learning as ontologically inseparable to the individual and learning and subjectivity are inter-subjective. But this constructivist position views learning as optimal when social actors collaborate to construct meaning via multidirectional dialogue.

However, my field research, involving English second language (ESL) speakers at a university in Dubai (Hurley, 2016), revealed participants’ difficulties in collaborating and using AR. Subsequently, the semiotic-dialogic framework in this study has been designed to offer alternative pathways into understanding AR visual literacies and problematises assumed positive affordances mentioned in previous literature. This theorising could therefore go beyond transcendental perspectives of what ‘true’ dialogic pedagogy or ‘true’ AR visual literacy should be while situating variations of weak and strong dialogisms in terms of their cultural historical context (Matusov, 2009). The study is also informed by the principle of researcher self-reflexivity and a co-construction of knowledge via theoretical processes designed to be interpretative and performative (Denzin, 2001). In terms of my own positionality, as mentioned, I initially came to AR as an ESL teacher. During my doctoral studies in technology enhanced learning, I began researching AR via an explicitly semiotic-dialogic framework since I was interested in developing a visually orientated approach to AR in order to help ESL speakers communicate in their second language. However, the semiotic-dialogic approach does not exclude partially sighted or blind students since visuality, from a Peircean perspective, includes the meanings of signs at symbolic and conceptual levels in an interpreter’s mind. Although beyond the scope of this article, theorising AR from a semiotic-dialogic standpoint could therefore have potential for facilitating inclusivity of students with varying visual abilities. More generally, semiotic-dialogism considers AR engagement in terms of collaborative interpretation of the meaning of signs at material, symbolic and conceptual levels.

In terms of this study, I am also self-reflexive that what I present is derived from my subjective interpretations, individual application of the semiotic-dialogic instrument and underpinned by the way I ‘see’ things. But, due to theoretical insights of semiotic-dialogism, it is argued that even the research of an individual author operates within a broader sociocultural context and in terms of intertextual perspectives (Lather, 2016). The conceptual framework of semiotic-dialogism as well as the methods to carry out the analysis are designed to integrate a range of perspectives. The thinking, findings and reflections of this study were also presented at the online international ‘Conference on Visual Literacies and Visual Technologies for Teaching, Learning and Inclusion’ (CIELL, 2020). This helped to gather feedback from other researchers in the field and language teachers who might be interested in using AR in their classrooms. The analytic categories of semiotic-dialogic inquiry are illustrated in Figure 2.

Figure 2: Semiotic-dialogic folds of inquiry

In Figure 2, folds of the AR semiotic-dialogic inquiry amplify the following entities:

  1. signs/sign sequences (specific elements, features and aspects of the AR content, including static and dynamic image sequences),

  2. objects (meanings the AR content refers to),

  3. dialogic interpretants (series of possible interpretations and dialogue around interpretative meanings),

  4. interpreters/learners’ interpretive practices (range of interpretations), and

  5. AR visual literacies (interpreter’s elements in conjunction with sociocultural aspects of interpretation) in relation to addressing the research question (research object).

These analytic categories are illustrated above as discrete units, but also as occurring simultaneously in semiosis. Application of semiotic-dialogic inquiry for analysis of specific AR images is also outlined as a series of analytical steps. These steps illustrate further dimensions of the specific image analysis embedded within the broader folds of AR semiotic-dialogic inquiry. This involved the following:

  1. selection of AR exhibits from the Festival of the Impossible,

  2. selection of specific AR images and dynamic image sequences from the exhibits,

  3. visual inquiry of AR exhibits’ signs, objects and interpretations occurring simultaneously,

  4. integrative inquiry of the AR images/image sequences as folds of sociocultural meanings and subjective author/learner interpretation, and

  5. implications/reflections on all of the above for analysing AR as a pedagogic tool.

These analytic steps are also laid out in Figure 3 to convey that, when thinking with semiotic-dialogism, visual meanings are considered as unfolding in motion, negotiable, becoming, on behalf of visual researchers, teachers, learners or any interpreters according to sociocultural context and subject matter. Figure 3 amplifies the broad aspects of focused semiotic-dialogic AR visual analysis.

Figure 3: Semiotic-dialogic AR image analysis

Nevertheless, since the application of the semiotic-dialogic framework in this article involves an individual instrument (the author) it is not anticipated that the visual inquiry will generate a broad range of interpretative and empirical data. The findings are also not intended to offer extensive representational validity but rather to highlight for the reader of the article the theoretical scope of semiotic-dialogic analysis for co-construction of knowledge and also a framework that teachers could use for visual inquiry into AR with their students. This (post)qualitative perspective could enable pathways for thinking about the specific elements of the visual meanings and learning offered by AR content, through the objects of reference and range of potential meanings.

The concept of the compound-sign is useful for understanding how AR images bundle concentrated histories, geographies and cultures of scopic regimes. As the inquiry graphics scholar Lacković (2018, 2020) points out, a Peircean perspective enables theorising of compound-signs and that denotation (what signs represent) as occurring simultaneously with connotation (the meaning of signs at individual and sociocultural levels). Semiotic-dialogic theorisations of how an image is a compound sign, made-up of an icon-symbol-index, means the observable features of AR can be identified and understood in relation to the broader sociocultural context as well as pointing to meanings in the interpreter’s mind. Its image-concept object (point of reference) provides a vehicle for bringing together the material and abstract aspects of thinking and knowledge (Lacković, 2020).

Furthermore, the point of reference or meanings of the AR content can be material, abstract, conceptual, ideological and/or imaginary while occurring in dialogue. The dialogic aspect of the framework facilitates a constant chain of questions concerning the intersecting and intertextual compound-signs of AR as integral to unfolding visual literacies. It is hoped that thinking with the assemblage of semiotic-dialogism re-orientates the pitfalls of objectivism and constructivism, rendering the visually literate sign-user as either a passive recipient or an omnipotent creator of meaning (Bergman, 2009). Alternatively, semiotic-dialogic inquiry involves critical questions about signs in process, as dynamic image sequences in interpretation and their role in creating new meanings. Semiotic-dialogic inquiry and folds of analysis provide an explicit framework for viewing AR within sociocultural contexts. It helps to question how AR visual literacies are entangled within a series of cultural, historical, political, material and technological scopic regimes. Exploring the sign assemblage of AR could therefore help to envisage a broader spectrum of interpreters/learners’ visual literacies, across a variety of sociocultural environments rather than exclusively in idealist terms or outcomes that are already optimised for dialogism.

The central question of the paper asks, ‘To what extent could semiotic-dialogic inquiry help to open understandings of AR visual literacy?’ I have so far suggested semiotic-dialogism may not only contribute to understandings of AR technologies, but also how it might expand conceptions of visual literacy, in terms of semiotic-dialogism, as a process of questions about visual meaning making. However, just as there is no fixed sense of what ‘true’ dialogic pedagogy should be in this paper, there is also no definitive version of visual literacies being presented. This is because visual regimes are not considered as merely representational but, conversely, they are affective, experiential, conceptual and sociocultural. Representational discourses, concerned with what an image or visual experience ‘means’ and how it can be ‘read’ or ‘captured’ is a transcendental perspective going against dialogism and fallibilism of Peircean semiotic philosophy (Bergman, 2009).

Building on insights from inquiry graphics, the semiotic-dialogic framework in this study could enable theoretical unfolding of the varying dynamics of AR dynamic image sequences including, technological, sociocultural, intersubjective and situated meanings occurring in tandem. This marks a shift in thinking about visual literacies as representing, capturing and naming visual processes in linguistic terms. Thinking with semiotic-dialogism helps to consider multiple social meanings being facilitated within semiotic-dialogic visual interplays. Moreover, the framework offers a version of semiotics that goes beyond linguistic and cultural nodes into a diverse universe of signs.

To consider specific AR exhibits, I take Adobe’s (2020) Festival of the Impossible as a case to discuss some of the semiotic-dialogic findings emerging from the folds of semiotic-dialogic inquiry. To facilitate coding of the AR exhibits, dynamic image sequences were summarised in text form, coded with key sign descriptors, and saved with a free-form description of what the post considered relatable, who and what was visible as well as what was inferential. In the next section I present the findings emerging from this inquiry.

4. Semiotic-dialogic findings

In the findings I present two exhibits. First, the analytical framework of semiotic-dialogic inquiry was applied to the Adobe AR exhibit, ‘White Noise’ (Landa, 2020) and followed the folds of semiotic-dialogic inquiry:

  • The selection of the AR film by Anna Landa was chosen based on my interest in the artist’s use of visual collage and fragments of varying cinematic styles.

  • I focused on two specific dynamic image sequences from Landa’s AR film. These sequences were the external view of windows of an apartment block that frame different women, one in colour and the other in black and white.

  • Possible interpretations and dialogue surrounding these elements include the combination of various objects and aesthetic styles for representing women across cinematic and photographic histories.

  • The next stage was to consider the objects and meanings of these representamina. The elements of the AR image thus included interpreting the visual meanings or object of the collage of different sized and celluloid styled women as objects in the windows.

  • Following this initial semiotic-dialogic inquiry, the next stage of exploration involved the focused semiotic-dialogic AR image analysis.

Next, I will guide the reader through the five folds of AR image analysis.

  1. At sign level, Landa’s AR theatrical exhibit conveys fragmented representations of women in different celluloid styles, mise en scène, shapes and sizes.

  2. At object level, this complex image is understood as a compound-sign that incorporates a range of visual styles, colours, and fragments of cinematic representations of celluloid women, see Figure 4.

Figure 4: Landa’s AR Windows (Landa, 2020)

  1. Semiotic-dialogic inquiry involved self-reflexivity concerning my reading of the image as interpreter. I reflected that, as viewer, I am positioned to peep into the apartment windows (of Landa’s AR exhibit) to view women in a scene reminiscent of Alfred Hitchcock’s (1954) ‘Rear Window’, see Figure 5.

Figure 5: “Rear Window Loop” CC BY-NC-ND 2.0
(Ars Electronica: Creative Commons, 2021b)

  1. The fourth integrative fold of the inquiry involved sociocultural reading of the AR image sequence. In terms of sociocultural meanings, the feminist film theorist Modleski (1988) called Hitchcock’s female characters ‘The women who knew too much’, since they are positioned as self-conscious objects of the (male) cinematic gaze. But, in ‘White Noise,’ rather than the male gaze through the lens of Hitchcock’s camera, Landa’s (2020) AR-interpreters are positioned, via symbolic-indexical signs, to experience and possibly question the foreboding of technological scopophilia, magnifying the intense visual surveillance of women in media (Soukup, 2009).

  1. At the conceptual level of the interpretant, Landa’s AR film indexes cinematic and visual trends, proceeding AR, that have produced layered voyeuristic gazes, in a number of mediums. Landa’s use of AR amplifies the voyeuristic act of looking at women via the fetishizing surveillance of various technological media. The image sequence involves complex icon-symbol-index sign compounds, that draw on historical signs of cinematic representation, while configuring new meanings about AR’s visualities.

Next, I discuss inquiry into the second dynamic image sequence from Landa’s ‘White Noise.’

  1. Sign elements within the frame of another window were selected. In this montage, a young woman grows and shrinks before us, much like ‘Alice in Wonderland’ (Carroll, 1865) see Figure 6.

Figure 6: AR Women (Landa, 2020)

  1. In addition to displaying the sexualized female body, the AR object here could be technology itself which occurs as the lens for fantasy and pleasure that continues to position women as objects who also look at themselves and back at the camera. The object of the growing/shrinking woman (see Figure 6) seems to be negating eroticism in exchange for the gaze of techno-scopophilia (Soukup, 2009).

  2. At conceptual levels, the semiotic-dialogic instrument enables pathways into analysing the AR image sequence’s elements; the meanings resulting from their combination; and possible sociocultural implications of the meanings. This also enables consideration of the research object, concerned with semiotic-dialogic inquiry into AR visual literacies.

  3. As author of the study, the research object (question) informed integrative subjective and sociocultural interpretations of the visual meanings. In my opinion, the AR artist indexes questions of how visual subjectivities involve complex systemic, cultural, historical and aesthetic collage for positioning women as the continuing object of spectacle within scopic regimes of technological surveillance. For interpreters, these elements of the spectacle could oscillate between claustrophobia and alienation, while possibly amplifying women’s anxieties surrounding body image, body dysmorphia and visual anxieties within techno-scopophilia and visual regimes.

  4. In view of the above integrative meanings, we can become aware of the AR artist’s reference to historical media styles, that despite industry hype surrounding AR as a novel medium, visual spectacle is nothing new and goes back to classic Hollywood, a cinema of attractions and much older histories of gendered visual representations.

In terms of implications for semiotic-dialogic inquiry of all the above, the two dynamic image sequences analysed via the framework of semiotic-dialogism, illustrate an example for teachers wishing to lead learners through the steps of visual inquiry into AR texts. These steps require learners to reflect on their choice of AR texts for inquiry; to list sign elements within dynamic AR visual sequences; to develop dialogues and reflections on their interpretations of the images in relation to their own sociocultural context. This framework could re-orientate learners to view AR within histories of visual culture which can be understood as a genealogy, rather than fragmented into disciplinary units, such as film, television, art, video, AR (Mirzoeff, 2006).

To illustrate this further, semiotic-dialogic inquiry was also applied to the Festival of the Impossible exhibit ‘The Masked City’ (Ritchie, 2020). Visual inquiry and the embedded unfolding of focused semiotic-dialogic dynamic image analysis occurred to reveal the following broad points:

  1. The AR story of Aislar, a character whose face is subsumed by an AR headset, was selected and key elements of the narrative were recorded. Aislar’s story begins when she emerges from a sketch inside a book, next to a keyboard and screen. The audio voice over, a further modal element, tells the audience she is a “traveller” in “this time of isolation.” Aislar wanders lonely through desolate, graffitied, dirty cityscapes and the debris of urban dystopia. The grey tone and brutalist aesthetic reflect deliberate design choices, by the AR artist, to convey a particular social ambience via compound-sign meanings. Beside the sketchbook is a copy of Houghton’s (1882) ‘Chronicles of the Photographs of Spiritual Beings and Phenomena Invisible to the Material Eye.’ This is an obscure reference by the AR artist to a nineteenth century text which includes alleged ‘spirit photographs’ of mediums and reminds us that we cannot always believe what we see. At the end of the AR film, Aislar walks into the sea, passing through a digitally imposed screen (within the screen).

  2. The semiotic-dialogic inquiry offered pathways to consider the complex elements of the dynamic compound-sign image sequence. Following the selection of this AR film, I zoomed in on specific AR images or frames within the dynamic sequence.

  3. Visual inquiry of the signs, objects and interpretations occurred simultaneously. This enabled identification of specific elements or details of the images to be listed. For example, the use of black and white, the protagonist Aislar dressed androgynously and wearing a VR headset.

  4. This led to the integrative reading of the AR images’ sociocultural meanings and subjective author/learner interpretation. In my interpretation, the scenes of the deserted cityscape pointed to the sense of desolate sociocultural alienation. Aislar’s face submerged behind a VR headset conveyed the character’s posthuman and digital persona.

  5. Analysis of the AR dynamic image sequence, ‘The Masked City,’ through application of the semiotic-dialogism instrument, reveals tacit criticisms and questions raised by the AR artist of the alienating onto-epistemologies of AR. In ‘The Masked City,’ Adobe’s AR app Aero is applied to demonstrate AR, not necessarily as bringing people together, but as an individualising technology of dystopian contexts. This contradicts themes in the AR pedagogic literature suggesting AR’s collaborative affordances. It also brings into question Adobe’s promotion of its Aero app in terms of its collaborative enhancements.

In regard to implications for learners and/or learning, the semiotic-dialogic dynamic image sequence analysis reveals some pivotal tensions concerning AR’s role in pedagogy. The sense of alienation, emerging from AR, is in sharp contrast to the views of Martín-Gutiérrez et al. (2014) who suggest that it is traditional pedagogy, rather than AR, which creates barriers to collaborative learning. At the end of the story, Aislar’s lonely dissent into the sea and then a screen could be interpreted as an object of cleansing but also an index of surrender or questioning of AR’s intensifying techno-scopophilic visual tides. Adobe, owned by Adobe.Inc, are not content with software, apps and graphics tools, and are keen to keep-up the momentum of cloud-based computing. Although the AR exhibits were funded by Adobe, the texts tacitly embed critiques of the corporate driven medium. This could suggest to learners that AR will not necessarily or automatically facilitate collaboration with peers but conversely the technological positioning might be alienating. In the next section, I offer further discussion of the study’s theoretical implications and possible application of the semiotic-dialogic framework for teaching and learning.

5. Discussion

The semiotic-dialogic inquiry reveals findings that ‘The Masked City’ occupies a comparable stance to ‘White Noise’ in indicating AR’s onto-epistemologies that are underpinned by tensions, anxieties and alienation. In terms of the research object, semiotic-dialogic analysis therefore problematises the normative perspective in the literature that AR promotes collaborative experiences even when these interpretations could occur collectively. Nevertheless, the application of the semiotic-dialogic instrument helps to reveal the critical meanings (objects) that the AR artists embedded in the AR exhibits. Semiotic-dialogic inquiry helped to go deeper than surface level description of the AR visual spectacle and to consider integral sociocultural interpretations and meanings. Theorising also suggests that the AR visual regimes, on display at the Adobe industry event, are not necessarily the exclusive domain of artistic expression, but, like other artists, performers and filmmakers, AR content creators work within the constraints of sociomaterial practices and the political economy.

In a similar vein, higher education is not necessarily the exclusive domain of teachers, learners and educators. Conversely, learning is regarded as a prospective ‘market’ by platform capitalists, like Adobe, who have been steadily moving into universities and schools around the globe (Means, 2018). Simultaneously, it is crucial to remember that Adobe’s pedagogy is motivated by profit as it positions learners as consumers and universities as clients. Even though Adobe’s Creative Cloud for Schools (2020) state: “Adobe tools empower students to communicate and think creatively so they can graduate with the digital skills needed for future career opportunities”, as higher education journalist Joshua Kim (2020) reminds us, “selling software is not analogous to educating students.”

In terms of teaching and learning, the semiotic-dialogic framework is generalisable as a theory of learning and also for pedagogic inquiry into AR dynamic image sequences. The framework enables a step-by-step approach for carrying out semiotic-dialogic inquiry into AR exhibits’ sign elements, the objects of meaning and interpretations unfolding through dialogic dynamic images sequences and in relation to sociocultural context. In this study, semiotic-dialogic inquiry considered AR exhibits that were shown during Adobe’s (2020) Festival of the Impossible. As an example of visual inquiry, analysing the two AR artists’ exhibits illustrates strategies for developing descriptions of dynamic image sequences and interpretive insights into the positionality of AR texts and the meanings being generated by AR artists at an industry sponsored event (Adobe, 2020).

The rationale was that asking questions about AR’s positionality, in terms of how it is being positioned by industry leaders and AR artists, could provide deeper insights that learners and teachers could consider through dialogue and visual reflection. Semiotic-dialogic inquiry helps to develop theoretical insights into AR visual literacies and refines considerations of the AR medium. Inquiry is concerned with externally observable image sequences and also the symbolic and conceptual affordances mediating dialogic sociocultural interpretations of signs. Simultaneously, this offers further planes of questioning into the discursive practices of the AR medium and its possible pedagogic uses for learners as a conceptual learning tool.

In terms of reflections on the commercial aspects of AR’s role in higher education, the Adobe (2020) sponsored Festival of the Impossible facilitates examples of AR resources that are available online. The Festival of the Impossible enabled Adobe to showcase artists’ uses of Aero AR tools. Adobe (2020) called the festival a “collective hallucination,” and emphasised AR’s scope for collaboration and interaction between users. Unlike virtual reality (VR), engulfing users within a headset, Adobe are building on the perception that AR users can interact with one another more easily than via the individualised experiences of VR. But, although these AR exhibits are accessible online and freely available, the Adobe event is motivated by promoting the AR Aero app as a marketable commodity. Adobe are making these resources available as part of their broader marketing push, aiming to convince potential consumers that Aero is an accessible and intuitive learning tool for users. However, just as AR’s collaborative affordances are questioned by the artists who created AR exhibits, the accessibility and/or desirability of Adobe Aero as an intuitive technology requires further inquiry. In the final section, I offer some end points, limitations and implications of the study for further research and policy.

6. Conclusions

In terms of addressing the central research question, of the extent to which semiotic-dialogic inquiry could open understandings of AR visual literacy, this study re-orientates inquiry into AR pedagogy. The literature review and case study of the Festival of the Impossible illustrates how semiotic-dialogism facilitates a series of questions within dialogue (Bakhtin, 1986) as well as perpetual inquiry as a process of ongoing meaning making via sign semiosis (Peirce, 1878). Previous literature surrounding AR pedagogy indicates that learners are motivated by the novel visual appeal of AR technologies (Solak & Cakır, 2015; Di Serio et al., 2013). Adobe’s Festival of the Impossible also tries to market its AR Aero software in terms of novel spectacle. However, this assumes that the tantalizing thrill of a new visual medium like AR can be sustained and generates new visual literacies. Yet, the history of obsolete media, for example analogue television or video home-recording systems (VHS), indicate otherwise. But, while Mahadzir and Phung (2013) emphasise the lure of AR’s perceptual arousal for learners, there is a limited conception of how AR draws on older modes and media of visual meaning making.

As counterpoint, the semiotic-dialogic perspective enables inquiry into how AR content and the objects for meaning build on previous historical, gendered and aesthetic meanings within the broader contexts of sociocultural scopic regimes. Semiotic-dialogic inquiry is a critical framework, for both AR dynamic image analysis and pedagogy, that could be applied to explore AR exhibits as compound-signs of densely bundled histories, ideologies and creation of new meanings occurring simultaneously. Furthermore, semiotic-dialogic inquiry provides openings for new ways of thinking about AR’s conflations with higher education, learning and platform capitalism. It suggests that despite the aggressive marketing by Adobe Aero, to establish AR as a collaborative learning tool, there is a lack of evident research or discussion of what is meant by collaboration or learning beyond novel spectacle.

Nevertheless, visual inquiry into AR exhibits, created by artists at the Festival of the Impossible, reveal tacit contradictions concerning the alienation and gendered histories of technological surveillance. The inquiry reveals that commercial platforms’ broad-brush promises concerning learning, for example Adobe’s claims for AR, cannot be taken at face value. These findings have important ramifications for higher education policy. The inquiry suggests that so-called learning technologies, and their assumed affordances for learning do not necessarily facilitate software that scaffold learning. In the case of Adobe Aero, despite the platform’s claims that the AR app is intuitive and affords accessible templates for creating content, this type of modelling activity is not synonymous with the conceptions of learning from a semiotic-dialogic perspective.

Semiotic-dialogism, viewing learning as dialogue and creation of new meanings, problematises the commercialised and technocratic narratives surrounding AR visual literacy. It does not accept technological solutionism as inevitable or as the only chapter of the AR pedagogic story. Inquiry advocates that research into AR learning technologies needs to be considered by pedagogic research entities who do not have a vested interest in the platforms’ financial profit. Consequently, matters of policy surrounding educational technologies are also questions of ethics and power concerning which entities are positioned to define ‘learning’ in the age of platform capitalism. This is an important future object of research.

Further future studies, thinking with semiotic-dialogism about AR and/or visual literacies, might explore the folds and bundled meanings of dynamic image sequences to consider their ethical, political and gendered positionings within scopic regimes. Semiotic-dialogism could be developed to position visual literacy as interdisciplinary inquiry involving a range of fields including: feminism; queer studies; critical discourse analysis; critical race theory; film theory; art and design; art history and other onto-epistemologies.

A possible limitation of the study is that, through synthesising semiotics, dialogism and visual literacy, it is overly theoretical and resists grand representational engagement. However, representation was not the goal of the semiotic-dialogism. The foci instead have been on re-orientation of conceptions of AR visual literacy to offer insights into the complexity of AR visual meanings and visual learning. In a similar vein, semiotic-dialogism is not orientated around validity since the standards of excellence for considering AR visual literacy do not necessarily have to be those of social sciences and/or positivism. Alternatively, semiotic-dialogic inquiry offers a model for re-orientating AR visual literacies in terms of the theories of learning, communication, visual arts, affect, impression, expression and thinking with. This is a valuable philosophical re-orientation especially considering neoliberal trends within technocratic societies are leading to the underfunding of arts and humanities despite their important contribution to critical research, the staging of political and ethical questions (Jandrić et al., 2018).

Finally, semiotic-dialogism suggests visual literacies, rather than being top-down or as exclusively driven by technology corporations, are the visual inquiries embedded within the broad genealogies of scopic regimes. Semiotic-dialogic analysis of dynamic AR image sequences reiterates that visual literacies should reflect diverse expressions, experiences, interpretations to facilitate inquiry and creation of new meanings. Thus, AR visual literacies could be developed through semiotic-dialogism to open learners’ hybrid and critical responses to AR, new meanings, visualities and visual onto-epistemologies. The AR artists’ exhibits discussed in this study, from the Festival of the Impossible, are revealed as posing difficult aesthetic questions about AR. Teachers, learners, researchers and educational policymakers also have an important role to play in developing semiotic-dialogic inquiry. Only as a result of critical questioning will technology platforms, like Adobe, be made more accountable for their broad-brush claims concerning learning and technological solutionism.

Acknowledgements

This paper draws on research undertaken as part of the Doctoral Programme in E-Research and Technology Enhanced Learning in the Department of Educational Research at Lancaster University. I also acknowledge the work of the AR artist Anna Landa whose critical application of AR informed the research and who graciously granted permission for images from her AR film ‘White Noise’ to be included. I would also like to thank the guest editors of this special edition of Studies in Technology Enhanced Learning — Julie-Ann Sime and Chryssa Themelis — for their efforts in getting this article ready for publication.


About the author

Zoe Hurley, Zayed University, Dubai, United Arab Emirates.

Zoe Hurley

Zoe Hurley currently teaches in the College of Communication and Media Sciences at Zayed University, in Dubai, on undergraduate courses focusing on social media and new media writing. Her research has involved developing qualitative visual/multimodal approaches to learning and communication from gender and critical theory perspectives. She is particularly interested in participant centred methodologies for understanding social actors’ uses of technology for learning within the broader intersections of historical processes and structural inequalities.

Zoe has published articles in Social Media + Society focusing on Gulf-Arab women’s visual social media as well as technology’s emotional temperatures and postdigital entanglements. Since becoming a PhD researcher, Zoe has delivered conference paper presentations on social media learning, social media influencers, multimodality, design thinking and the Change Laboratory as well as Gulf-Arab women’s empowerment through visual social media.

Email: [email protected]

ORCID: 0000-0002-9870-8677

Twitter: @ZoeHurleyDubai

Article information

Article type: Full paper, double-blind peer review.

Publication history: Received: 15 December 2020. Revised: 20 February 2021. Accepted: 03 June 2021. Published: 14 March 2022.

Cover image: Stefan Keller via Pixabay.


References

Adobe. (2020). Festival of the impossible: A digital art exhibition exploring identity, and the meaning of home. https://www.festivaloftheimpossible.com/

Adobe Creative Cloud for Schools. (2020). Creative Cloud for schools, universities, institutions | Adobe. https://www.adobe.com/mena_en/creativecloud/buy/education.html?sdid=B16P4299&mv=search&ef_id=Cj0KCQjwufn8BRCwARIsAKzP697bnnh8RZdfObBxPvmRriCvrNoB2MTZYpgUwyel5akOwc-dmINUZUYaAiTIEALw_wcB:G:s&s_kwcid=AL!3085!3!397323857730!e!!g!!adobe%20creative%20cloud%20education!1677056358!71951618824

Akçayır, M., & Akçayır, G. (2017). Advantages and challenges associated with augmented reality for education: A systematic review of the literature. Educational Research Review, 20, 1-11. https://doi.org/10.1016/j.edurev.2016.11.002

Akçayır, M., Akçayır, G., Pektaş, H., & Ocak, M. (2016). Augmented reality in science laboratories: The effects of augmented reality on university students’ laboratory skills and attitudes toward science laboratories. Computers In Human Behavior, 57, 334-342. https://doi.org/10.1016/j.chb.2015.12.054

Bakhtin, M. (1986). Rabelais and his world (H. Iswolsky, Trans). Indiana University Press.

Bakhtin, M. (1999). The problem of speech genres. In A. Jaworski & N. Coupland, The discourse reader (pp. 121-132). Routledge.

Berger, J. (1972). Ways of seeing. British Broadcasting Corp.

Bergman, M. (2009). Peirce's philosophy of communication. The Rhetorical Underpinnings of the Theory of Signs. Continuum.

Carroll, L. (1865). Alice in Wonderland. MacMillan.

Cassidy, M., & Knowlton, J. (1983). Visual literacy: A failed metaphor?. ECTJ, 31(Michael F. Cassidy & James Q. Knowlton), 67-90.

CIELL (2020). Conference on Visual literacies and visual technologies for teaching, learning and inclusion. https://ciell.eu/conferenceeng/

Creative Commons (2021a). "Ikea place app" by markhillary is licensed under CC BY 2.0 [Image]. https://creativecommons.org/licenses/by/2.0/?ref=ccsearch&atype=rich

Creative Commons (2021b). "Rear window loop" CC BY-NC-ND 2.0 by Ars Electronica [Image]. https://ars.electronica.art/center/en/rear-window-loop/

Coimbra, M., Cardoso, T., & Mateus, A. (2015). Augmented reality: An enhancer for higher education students in math's learning?. Procedia Computer Science, 67, 332-339. http://doi.org/10.1016/j.procs.2015.09.277

Denzin, N. (2001). The reflexive interview and a performative social science. Qualitative Research, 1(1), 23-46. https://doi.org/10.1177/146879410100100102

Di Serio, A., Ibáñez, M., & Kloos, C. (2013). Impact of an augmented reality system on students' motivation for a visual art course. Computers & Education, 68, 586-596. https://doi.org/10.1016/j.compedu.2012.03.002

Dori, Y., & Belcher, J. (2005). How does technology-enabled active learning affect undergraduate students' understanding of electromagnetism concepts?. Journal of The Learning Sciences, 14(2), 243-279. https://doi.org/10.1207/s15327809jls1402_3

Dunleavy, M., Dede, C., & Mitchell, R. (2008). Affordances and limitations of immersive participatory augmented reality simulations for teaching and learning. Journal of Science Education And Technology, 18(1), 7-22. https://doi.org/10.1007/s10956-008-9119-1

Dunleavy, M. (2014). Design principles for augmented reality learning. TechTrends, 58(1), 28-34. http://doi.org/10.1007/s11528-013-0717-2

Eisenbrand, R., & Peterson, S. (2019). Augmented reality filters are coming—and so are impressions in the billions. https://omr.com/en/ar-effects-instagram-and-snapchat/

Freire, P. (1970). Pedagogy of the oppressed (11th ed.). Continuum.

Hitchcock, A. (1954). Rear window [Film]. Paramount Pictures.

Houghton, G. (1882). Chronicles of the photographs of spiritual beings and phenomena. Elibron Classics.

Hurley, Z. (2016). WoW! Window on the World: IPADS and Augmented Reality for Richer Readings. In Academic Oasis International Multidisciplinary Academic Conference. Miami: Academic Oasis International Multidisciplinary Academic Conference.

Jandrić, P., Ryberg, T., Knox, J., Lacković, N., Hayes, S., & Suoranta, J. et al. (2018). Postdigital dialogue. Postdigital Science and Education, 1(1). https://doi.org/10.1007/s42438-018-0011-x

Kamarainen, A., Metcalf, S., Grotzer,, T., Browne, A., Mazzuca, D., Tutwiler, M., & Dede, C. (2013). EcoMOBILE: Integrating augmented reality and probeware with environmental education field trips. https://dash.harvard.edu/bitstream/handle/1/37231546/EcoMOBILE_Integrating_Augmented_Reality_and_Probeware_with_Environmental_Education_Field_Trips.pdf?sequence=1&isAllowed=y

Kędra, J. (2018). What does it mean to be visually literate? Examination of visual literacy definitions in a context of higher education. Journal of Visual Literacy, 37(2), 67-84. https://doi.org/10.1080/1051144x.2018.1492234

Kim, J. (2020). Why behaving like big tech will not ‘save’ higher education | Inside Higher Ed. https://www.insidehighered.com/blogs/learning-innovation/why-behaving-big-tech-will-not-%E2%80%98save%E2%80%99-higher-education

Lacković, N. (2018). Analysing video in educational research: an "inquiry graphics" model for multimodal, Peircean semiotic coding of video data. Video Journal of Education And Pedagogy, 3(6).

Lacković, N. (2020). Inquiry graphics in higher education. Springer Nature.

Landa, A. (2020). White Noise [AR Film]. Adobe Festival of the Impossible: https://www.festivaloftheimpossible.com/.

Lather, P. (2016). Top Ten+ List. Cultural Studies ↔ Critical Methodologies, 16(2), 125-131. doi: 10.1177/1532708616634734

Mahadzir, N., & Phung, L. (2013). The use of augmented reality pop-up book to increase motivation in English language learning for national primary school. IOSR Journal Of Research & Method In Education (IOSRJRME), 1(1), 26-38. https://doi.org/10.9790/7388-0112638

Martín-Gutiérrez, J., Mora, C., Añorbe-Díaz, B., & González-Marrero, A. (2017). Virtual Technologies Trends in Education. EURASIA Journal of Mathematics, Science And Technology Education, 13(2). https://doi.org/10.12973/eurasia.2017.00626a

Matusov, E. (2009). Journey into dialogic pedagogy. Nova Science.

Means, A. (2018). Learning to save the future. Routledge.

Mirzoeff, N. (2006). On visuality. Journal of Visual Culture, 5(1), 53-79. https://doi.org/10.1177/1470412906062285

Modleski, T. (1988). The women who knew too much. Routledge.

Peirce, C. (1878). How to Make Our Ideas Clear. Popular Science Monthly, 12(Jan), 286-302.

Peirce, C. (1908). The essential Peirce. Indiana University Press.

Petrilli, S. (2014). Sign studies and semioethics. De Gruyter Mouton.

Pettersson, R. (1989). Visuals for information, research and practice (p. 146). Educational Technology Publications.

Ritchie, M. (2020). The Masked City [AR Film]. Adobe Festival of the Impossible: https://www.festivaloftheimpossible.com/.

Solak, E., & Cakır, R. (2015). Exploring the effect of materials designed with augmented reality on language learners' vocabulary learning. The Journal of Educators Online, 13(2),50-72. http://files.eric.ed.gov/fulltext/EJ1068381.pdf

Soukup, C. (2009). Techno-scopophilia: The semiotics of technological pleasure in film. Critical Studies In Media Communication, 26(1), 19-35. https://doi.org/10.1080/15295030802684026

Speicher, M. (2018, November 16). What is augmented reality anyway? The Conversation. https://theconversation.com/what-is-augmented-reality-anyway-99827

Sungkur, R., Panchoo, A., & Bhoyroo, N. (2016). Augmented reality, the future of contextual mobile learning. Interactive Technology and Smart Education, 13(2), 123-146. https://doi.org/10.1108/itse-07-2015-0017

Tesch, R. (1990). Qualitative research: Analysis types and software tools. Routledge.

Yen, J., Tsai, C., & Wu, M. (2013). Augmented reality in the higher education: Students’ science concept learning and academic achievement in astronomy. Procedia - Social And Behavioral Sciences, 103, 165-173. https://doi.org/10.1016/j.sbspro.2013.10.322

Zhu, E., Hadadgar, A., Masiello, I., & Zary, N. (2014). Augmented reality in healthcare education: An integrative review. Peerj, 2, Article e469. https://doi.org/10.7717/peerj.469

Zhu, P., Dudhia, J., Field, P., Wapler, K., Fridlind, A., & Varble, A. (2012). A limited area model (LAM) intercomparison study of a TWP-ICE active monsoon mesoscale convective event. Journal of Geophysical Research: Atmospheres, 117(D11). https://doi.org/10.1029/2011jd016447

Comments
0
comment
No comments here
Why not start the discussion?