topical media & game development

talk show tell print


PANORAMA -- explorations in the aesthetics of social awareness

A. Eliëns & D. Vyas
VU University Amsterdam



In this paper we reflect on our experiences in developing PANORAMA, a playful application meant to promote and support social awareness in a work environment, through art-inspired visualisations of social processes and personal contributions. With respect to the design of PANORAMA, we found common notions of visual semiotics helpful in determining the overall composition of the screen layout. More in general, however, the development of PANORAMA proved to be an exercise in interaction aesthetics, which as we will argue in this paper may greatly benefit from common notions in interactive video game play. In this paper we will only briefly discuss technical and deployment issues, since our main contribution here is to establish the relation between the aesthetics of interaction and game play.
Keywords and phrases: social awareness, visual semotics, interaction aesthetics, video games


Since the 1970's, Dutch universities have enormously grown in size, due to the ever larger number of students that aim at having university level education. As departments become bigger, however, staff members no longer know eachother personally. The impersonal and anonymous atmosphere is increasingly an issue of concern for the management, and various initiatives have been taken, including collective trips into nature, as well as cultural events, not to much avail for that matter. An additional problem is that more and more members of the staff come from different countries and cultures, often only with a temporal contract and residence permit. Yet, during their stay, they also have the desire to communicate and learn about the other staff members and their culture(s).

In september 2006, the idea came up to use a large screen display in one of the public spaces in our department, to present, one way or another, the 'liveliness' of the work place, and to look for ways that staff members might communicate directly or indirectly with eachother through this display. Observing that communications often took place during casual encounters at the coffee machine or printer, we decided that monitoring the interactions at such places might give a clue about the liveliness of the work place. In addition, we noted that the door and one of the walls in the room where the coffee machine stood, was used by staff members to display personal items, such as birth announcement cards or sport trophees. In that same room, mostly during lunch time, staff members also gathered to play cards.

Taking these observations together, we decided to develop a system, nicknamed PANORAMA, to present these ongoing activities and interactions on a large display, which was to be positioned in the coffee room. The name of our system is derived from the famous Mesdag Panorama in The Hague, which gives a view on (even in that time nostalgic rendering of) Scheveningen. However, it was explicitly not our intention to give an in any sense realistic/naturalistic remdering of the work place, but rather, inspired by artistic interpretations of panoramic applications as presented in  [Grau (2003)], to find a more art-ful way of visualizing the social structure and dynamics of the work place.

At this stage, about one year later, we have a running prototype (implemented in DirectX), for which we did perform a preliminary field study as well as a first user evaluation,  [Vyas et al. (2007)], and also we have experimented with a light-weight web-based variant, allowing access from the desktop. In this paper, our primary focus, however, will be to establish the relation between interaction aesthetics and game play, as inspired by our experiences in developing PANORAMA.


The structure of this paper is as follows. First we will give a brief sketch of the PANORAMA system, that is the ideas underlying it and the realization of a first prototype. Then we will present, in a more general fashion, possible guidelines for the design of PANORAMA, followed by a discussion of common issues in interaction aesthetics and game play. We will then introduce the notion of dialectics of awareness, and in conclusion identify the primary dimensions of aesthetic experience.


Fig 1. (a) context(b) self-reflection


The PANORAMA system is meant to support social awareness, in non-work related ways, using a large screen display in a public room in our faculty. To achieve social awareness, we ask the staff to contribute items of self-reflection, such as holiday postcards or birth announcements. In order to reflect the liveliness of the workplace, we monitor places where occasional encounters may take place, for example during a break at the coffee machine or in the printer room, waiting for the printer queue. Encounters in such places are often of an informal, personal nature, but may be mixed with work-related interests. As an experimental feature, we consider to allow for direct interaction using the system, for example, to play a game, possibly with a mobile phone as an input device. In summary, the PANORAMA system is determined by the following contributions of its users, contributions that are not necessarily direct or even do require explicit activity.


For a deeper understanding of what role the system would play in the (working) life of the staff, we engaged in several field studies and used cultural probes to determine what could be valuable contributions to ask for and how to display these on the PANORAMA screen,  [Vyas et al. (2007b)].

We have developed a first prototype implementation in DirectX 9, using ViP technology, based on the system described in  [Eliens (2006)]. In this realization, we deploy a moving virtual gallery, containing video and image feeds. The gallery acts like a moving scroll, displaying information in a continuous manner, in a panorama-like way. The images in the gallery are fed by channels, containing information that is either due to explicit contributions (self-reflections) or ongoing activity in the work place (casual encounters or occuring events), monitored by cameras or other sensors.

Obviously, as we will discuss later, the PANORAMA system is subject to a dialectic of awareness, that is it will be present, but the staff will only occasionally pay atention to it, dependent on their interests and also on what visual cues and effects the system presents to draw attention to ongoing activity. Although we would like the system to be autonomous in the decision how to present information, we cannot hope to do this by computational means only,  [Eliens (1988)], and hence we need to provide interaction markers to invite the users to contribute actively to the system, or influence the way information is displayed according to their preference.

For the display of information, we provide a rich context of material, including videos showing the faculty and its surroundings, fragments from video clips, and of course the material resulting from the occassional encounters and self reflections. In PANORAMA we use particle systems displaying the information in a pictorial way by images flowing according to the rules of the particle system chosen to represent that particular type of information. To organize this material we took conventions governing our interpretation of 2D displays as a guideline in designing the flow of particle systems. A more detailed discussion of these conventions will be given in the next section. Identifying bottom with plain, top with ideal, left with given and right with new, we arrived at the following identifications.

semiotic rules

  • self reflections: plain ==> ideal/new
  • casual encounters: plain ==> ideal/given
  • contextual stories: ideal/given ==> plain/new
  • personell faces: ideal/new ==> plain/given
  • occurring events: ideal ==> plain
For example one may remark that people's faces become more familar in time, and that in the process of getting to know them we see more of the plain reality of people. Naturally, different interpretations and different designs are possible.

Apart from the spatial characteristics of these flows of information we also used the speed with which the images move accross the screen as a parameter of design. For example events and occurrences move very fast, while both casual encounters and self reflections move slowly. Faces come across the screen with intermediate speed. To give self reflections more visual salience, the images are displayed in a non-transparent way, whereas all other flows of images merge with the background due to transparency. Although it is debatable whether the interpretations given above hold, we found the heuristics given by semiotic theory extremely helpful in deciding how to represent the information as flows of images in space/time.


Aesthetic awareness is common to us all. Having an understanding of aesthetic awareness, can we isolate the relevant design parameters and formulate rules of composition that may help us in developing interactive applications? According to our philosophical credo,  [Eliens (1979)], no! However, the history of art clearly shows the impact of discoveries, such as the discovery of perspective, as well as conventions in the interpretation of art, as for example in the iconic representation of narrative context in 17th century Dutch painting. Moreover, the analysis of the visual culture of mass media may also give us better understanding of the implied meaning of compositional structures.

The notion of perspective, described in  [Alberti (1435)], is an interesting notion in itself, since it describes both the organisation of the image as well as the optimal point of view of the viewer. The normal perspective as we know it is the central perspective. However, there are variants of perspective that force the viewer in an abnormal point of view, as for example with anamorphisms.

Perspective had an enormous impact on (western) art and visual culture. It defines our notion of naturalist realism, and allowed for the development of the panorama as a mass medium of the 19th century,  [Grau (2003)]. Art that deviated from central perspective, such as cubism or art from other cultures, was often considered naive. Photography and its pre-cursors had a great impact on the perfection of perspectivist naturalism, and what is called photorealism became the touchstone of perfection for early computer graphics,  [Bolter and Grusin (2000)].

Apart from perspective, other conventions regulate the composition of the 2D image, in particular, following  [Kress and van Leeuwen (1996)], the information value related to where an object is placed in the image, and the salience of the object, determined by its relative size, being foreground or background, and visual contrast. Also framing is used to emphasize meaning, as for example in the close-up in a movie shot. In analysing a large collection of image material,  [Kress and van Leeuwen (1996)], somewhat surprisingly found that lef/right positioning usually meant given versus new, top/bottom positioning ideal versus real, and centre/margin positioning important versus marginal. It is doubtful whether these meaning relationships hold in all cultures, but as a visual convention it is apparently well-rooted in western visual culture.

For 2D images,  [Kress and van Leeuwen (1996)] further identify narrative elements, that is relations between objects in the image that suggest a story, such as a diagonal line from a person to a door, or a relation of an object to the viewer, such as a gaze towards the viewer, a technique that has been used only since late renaissance painting.

More than paintings or 2D images, film is the medium for conveying narrative structures. The art of storytelling in film has been perfected in such a way that Hollywood films may seem more real than life. However, as emphasized in  [Bolter and Grusin (2000)], this is not due to any inherent form of naturalism, but to the fact that we have got accustomed to the conventions applied, that is the techniques of cutting, montage, camera movements, close-ups, etcetera. In a highly recommended book,  [Arnheim (1957)], Rudolf Arnheim gives an extensive analysis of the principles of montage and film technique, and he explains why film is such an effective medium:

frames of reference

It is one of the most important formal qualities of film that every object that is reproduced appears simultaneously in two entirely different frames of reference, namely the two-dimensional and the three-dimensional, and that as one identical object it fulfills two different functions in the two contexts.

Due to the subtle play between these two frames of reference film may be considered an art form, and as such perhaps the dominant art form of the 20th century. As a mass medium, film may be characterized by what Arnheim, following Benjamin, called the aesthetics of shock, replacing reflective distance with immersive thrill. As an art form, however, it is the dominant paradigm for aesthetic awareness, lacking however still one dimension, interactive dynamics.

As observed in  [Bolter and Grusin (2000)], interaction is what distinguishes video games from film. Current day technology allows for high-resolution photorealist graphics, that make video games or virtual applications almost indistinguishable from film. Virtual reality technology as applied in video games adds arbitrary choice of perspective, as exemplified in first-person shooters or fly-overs, as well as an arbitrary mix of the imaginary and real, as in CG movies, in an interactive fashion.

Now, should we take the aesthetics of interactive video games as the standard for interactive applications? Not necessarily, since the naturalism strived for in most games may at best be characterized as naive realism, mostly photorealism. As observed in  [Kress and van Leeuwen (1996)], realism is a social construct, and hence the program for developing an aesthetics for interactive applications should perhaps include the development of appropriate realisms. Again with an eye to the history of art, where we have for example impressionism, cubism, expressionism, as a guideline in the design of interactive systems, it might be even better to look for appropriate interaction-isms, styles of developing interactive systems and games from a particular perspective. Not excluding provocative perspectives! Cf.  [Burger (1981)].


Where an arbitrary interactive system may differ from a game played for entertainment is obviously the actual outcome, that is the value attributed to using the system in the real world, and probably the effort required and the possible consequences. You would not like to run the risk to die a virtual death when answering your email, would you? However, when interactive systems replace task-bound functionality with fun, the difference becomes less clear.

As we indicate in  [Eliens & Chang (2007)], one element not sufficiently captured by a classic game model, as introduced in  [Juul (2005)], is the narrative aspect of the game play. To quote  [Juul (2005)]:

rules vs fiction

Game fiction is ambiguous, optional and imagined by the player in uncontrollable and unpredictable ways, but the emphasis on fictional worlds may be the strongest innovation of the video game.

We may observe that many games already have a strong relation to reality in what narrative context they supply, or else in the realities of the media industry, in particular Hollywood. For serious interactive systems, we may assume an even stronger and in some sense more straightforward relation with reality, by the use of media content that is relevant for the life of the individual.

All these aspects of playing games are clearly relevant for the new interactive systems, which appeal more to play than task-oriented behavior. For example rules may be used to describe the visual characteristics of a system (e.g. the display of images as a flow in a particle system), outcome may be regarded as the benefits of the system (e.g. social awareness), value may include the risks of the system (e.g. a transgression of privacy), efforts is important when asking for contributions from the user (e.g. as image material to be displayed in the system), attachment may result when the system is installed (e.g. when people look forward to find new information), and finally consequences must be considered when a system is installed and used (e.g. interaction between people may actually change when they get to know eachother, for better or worse).

interaction markers

Where in game playing the variety of interaction modes seems to be well understood within each community of game players, for the development of more general interactive systems we will have to think seriously whether the target user will be able to learn the various modes of interaction, either by explicit instruction or during play. And as designers we must be concerned with the rules of interaction as well as issues of visualisation and interaction mappings, that is in other words which affordances the application offers for a particular group of users.


In the course of our field study for the PANORAMA system,  [Vyas et al. (2007)], we tried to establish what relation users would have to the system, not only in the way they interact with it, but also in terms of what role the system plays in their lives, and when and how they would be aware of the system.

Due to the intrinsic properties of the PANORAMA system, as a system meant to support social awareness in a work environment, we could not assume direct focussed attention. Instead, we must take the various forms of awareness or attention into account.

Our thoughts in this direction were triggered by a lecture of Linda Stone (former vice-president of Microsoft) at the Crossmedia Week September 2006 in Amsterdam, entitled Attention -- the Real Aphrodisiac. In that lecture Linda Stone made a distinction between applications popular before 1985, applications which were in general meant for self-improvement, for example language-learning, applications that were popular between 1985 and 2005, applications that she characterized as supporting continuous partial awareness, such as email and news-feeds, and applications of the period thereafter, from now into the future, which may be characterized as applications that allow the user to be creative, take part in a community, and are in other words more focussed and less dependent on the external environment.

Admittedly, it takes a few more steps to formulate a theory of the dialectics of awareness. However, with the function of the PANORAMA system in mind, we may make, following  [Benjamin (1936)], some interesting distinctions between the experience of art and architecture. Where art is usually experienced in a delimited time span, and is similarly delimited in space, that is the position of the observer, architecture is everywhere and always there. As a consequence, art receives focussed attention and may be appreciated with reflective distance, whereas architecture is often not perceived consciously, but merely present and subject to an almost sub-conscious sensibility, which is only brought to the focus of attention when it is either aesthetisized, for example when taking photographs, or when something surprising is sensed, for example in the change of skyline in New York.

As argued in  [Hallnäss and Redström (2002)], many of the new interactive systems, whether in the category of ambient media, ubiquitous computing or calm technology, will fall somewhere inbetween the spectrum spanned by art and architecture, or more likely even alternate between the forms of awareness associated with respectively art and architecture.

In designing the new interactive systems and games, we need to be explicitly concerned with the actual phases of awareness that occur, simply because it is not clear what role these systems play in our life. When introducing a new system or artefact, we may distinguish between the following phases:

phases of awareness

  • initiation -- appeal to curiosity
  • promotion -- raising interest
  • progression -- prolonged involvement
As designers we must ask ourselves the following questions. How do we appeal to the users' curiosity, so that our system is noticed? How do we get a more sustained interest? How de we get the user to interact with or contribute to the system? And, how do we obtain prolonged involvement, and avoid boredom? These questions are not simple to answer, and require also an understanding of the actual context in which the system is deployed as well as an understanding of the level of (aesthetic) literacy of the user(s).


In  [Hallnäss and Redström (2002)] the notion of expressional is introduced, to convey the expressive meaning of objects, that is the shift of attention from use to presence, subject to a dialectic process of appearance and gradual disappearance, when objects gradually become more familiar. For the design of presence, aesthetics is then considered as a logic of expressions, in which expressions act as the presentation of a structure in a given space of design variables. However appealing the notion of expressional, following idealist aesthetics,  [Kant (1781)], where a distinction is made between aesthetic awareness as a given, or a priori, sensibility and aesthetic judgement as being of a more empirical nature, we would prefer to consider aesthetics as a logic of sensibility, which includes a dimension of self-reflection in the sense of its being aware of its own history. Put differently, to characterize the contextual aspect of aesthetics, as it certainly applies to art, we may speak of aesthetic literacy, that is aesthetic awareness that is self-reflective by nature.

Assuming a notion of aesthetics as a logic of sensibility, we may distinguish between three dimensions of form, extending Kant's original proposal, as indicated below:

dimensions of aesthetic experience

  • spatial -- topological relations, layout of image
  • temporal -- order, rhythm, structure
  • dynamic -- interaction, reflection, involvement
The dimension of dynamics clearly is the great unknown, and more in particular it is the dimension we have to explore in the context of interactive systems, not in isolation but in relation to the other dimensions, not so much to establish definite criteria, but to understand the forces at work, or in other words the relevant parameters of design.  [Sartre (1936)] gives an existential foundation for the dimension of dynamics, by observing that the human body is instrumental in gaining awareness, as the centre of both obscurity and reflection from which consciousness emerges, through selection and action. It is in the existential dimension of aesthetic awareness that we come most close to the experience of the new digital artefacts, since it concerns both involvement and human action. Interestingly, and in apparent contradiction with  [Hallnäss and Redström (2002)], cited previously, to establish a foundation for the aesthetics of interactive systems  [Graves-Petersen et al. (2004)] seek refuge with pragmatist aesthetics as it promotes aesthetics of use rather than aesthetics of appearance. Again, although we agree with the gist of  [Graves-Petersen et al. (2004)], we wish to emphasize that the contribution of pragmatist aesthetics is not its focus on use, but the role of experience in understanding and appreciating aesthetic artefacts and for that matter games, that is the active role of the subject in becoming aware of its meaning.

Reflecting on the epistemological value of game playing in  [Climate], we observed following  [Magic], that the game player enters a magic circle akin to a complex social system, where actors, rules, and resources are combined in intricate (game) configurations:

game as social system

players eventsgame space
Leaving the interpretion of the elements of such a (game) system, indicated in the table above, to the reader, we may wonder what meaning games have, and looking at the fantasy items and visual effects of current day video games, we may wonder not only what is the meaning of meaningful elements, having a logical place in the narrative, but also what is the meaning or function of the apparently meaningless elements. The answer is simple, involvement and more in particular emotional involvement due to the in-born playfulness of humans.


The original version of PANORAMA was developed in DirectX, and was meant to be displayed on a large screen, with a static viewpoint on a dynamically changing scene, reflecting the activity in the workspace and the self-reflective contributions in an artful way. In  [VUSL], we speculated about using Second Life for the realization of PANORAMA. Moving from a central large screen in first life to a visualisation embedded in Second Life, which is moreover subject to first-person viewpoint dynamics, is quite a challenge. The most simple solution would be to project the large PANORAMA screen onto a display object in Second Life using live video streams but this would leave that challenge unanswered. Embedding PANORAMA in Second Life would allow us to observe, in more detail than in a previous user study, the behavior of users, that is, to be more precise, the proximity to particular objects of interest, the duration of their presence, and, using mechanisms of recommendation, their interest in related items.

We also explored the use of AJAX and web services in an X3D/VRML implementation of PANORAMA, using the google GWT toolkit to allow users to contribute their image material, and a PHP server for storage and retrieval of images,  [Si & Eliens (2007)].

In addition, we consider to retarget the promotional game we developed for our faculty, using the Half Life 2 Source SDK, into a (virtual) platform for social encounters and awareness,  [VULife].


In an earlier study on interaction aesthetics, we came along a report of how the Belgium curator Jan Hoet organized the Documenta IX, a famous yearly art event in Germany, and we were struck by the phrase art and the public sharing accomodation,  [Documenta], which in a unique way expresses the intuition we have with respect to the role the new interactive systems and games may play in our lives.

The PANORAMA system, as presented in this paper, may be regarded as one of the new interactive systems, with game playing -- in the form of occasional battle(s) -- as an intrinsic element. PANORAMA, and similar systems alike, presents us not only with a technical challenge, but more importantly also with a design challenge, which requires a new way of looking at the aesthetics of interaction, or perhaps we should say the meaning of such systems in our day to day experience, amplifying our awareness.

As our initial prototype was received with much interest, we see as important targets for future research, firstly the deployment of alternative platforms,  [Si & Eliens (2007)], and secondly the development of suitable games, that fit within the aesthetic framework determined by the primary raison d'etre for PANORAMA, to promote and support social awareness.


[Perspective] Alberti L.B. (1435),
On painting and on sculpture, Phaidon, edited by C. Grayson, 1972
[Film] Arnheim R. (1957),
Film as Art, University of California Press
[Reproduction] Benjamin W. (1936),
The Work of Art in the Age of Mechanical Reproduction, Online Archive --
[Remediation] Bolter J.D and Grusin R. (2000),
Remediation -- Understanding New Media, MIT Press
[Avantgarde] Burger P. (1981),
Theorie der Avantgarde, Edition Suhrkamp
[Creativity] Eliens A. (1979),
Creativity: reflection and involvement, Ninth Int Conf of Aesthetics, Dubrovnic, August 1979
[Computational] Eliens A. (1988),
Computational Art, First Int Symposium on Electronic Art, Leonardo Supplementary Issue, Pergamon Press 1988, pp. 21-26
[Odyssee] Eliens A. (2006),
Odyssee -- explorations in mixed reality theatre, In Proc. GAME'ON-NA 2006, P. McDowell ed., Eurosis-ETI, pp. 62-64, Sept 19-21, Monterey, USA
[Serious] Eliens A. and Chang T. (2007),
Let's be serious -- ICT is not a (simple) game, In Proc. FUBUTEC 2007, April 2007, Delft
[VirtualArt] Grau O. (2003),
Virtual Art -- From Illusion to Immersion, MIT Press
[Interaction] Graves-Petersen M., Iversen O.S., Gall Krogh P.,
Ludvigsen M. (2004), Aesthetic Interaction -- A Pragmatist's Aesthetics of Interactive Systems, Proc. DIS2004, Cambridge Mass. USA, pp. 269-276
[Presence] Hallnäss L. and Redström J. (2002),
From Use to Presence: On the Expressions and Aesthetics of Everyday Computational Things, ACM Trans. on Computer-Human Interactions, Vol. 9, No 2, pp. 106-124
[HalfReal] Juul J. (2005),
Half Real -- Video Games between Real Rules and Fictional Worlds, MIT Press
[Kritik] Kant E. (1781),
Kritik der reinen Vernunft, Felix Meiner Verlag, ed. 1976
[Semiotics] Kress G. and van Leeuwen T. (1996),
Reading Images: The Grammar of Visual Design, Routledge
[Imagination] Sartre J.P. (1936),
L'imagination, Presse Universitaire de France, ed. 8, 1971
[PanoramaWeb] Si Yin and Eliens A. (2007),
PANORAMA -- A Rich VRML Application Platform For Online Gaming, Workshop Web 3D Games, Web3D Symposium 07, Perugia Italy, April 15-18
[Panorama] Vyas D., van de Watering M., Eliens A.,
van der Veer G. (2007), Engineering Social Awareness in Work Environments, Proc. HCI International 2007, 22-27 July, Beijing, China
[Social] Vyas D. van de Watering M., Eliens A.,
van der Veer G. (2007b), Being Social @ Work: Designing for Playfully Mediated Social Awareness in Work, Proc. HOIT 2007, Chennai, India in August 2007

(C) Æliens 04/09/2009

You may not copy or print any of this material without explicit permission of the author or the publisher. In case of other copyright issues, contact the author.