preface -- topical media

topical media & game development

This book provides a concise and comprehensive introduction to multimedia. It arose out of the need for material with a strong academic component, that is material related to scientific research.

Indeed, studying multimedia is not only fun. Compare it with obtaining a driver license. Before you are allowed to drive on the highway, you have to take a theory exam. So why not take such an exam before entering the multimedia circus.

Don't complain, and take the exam. After all it makes you aware of the rules governing the (broadband) digital highway.

themes and variations


An underlying thought that motivated the writing of this book is that somehow the gap between authoring and retrieval should be bridged.

In other words, either by developing the technology for extracting features or attributes from multimedia objects, or by applying content annotation for such objects, multimedia information retrieval should be considered as a necessary asset to make a multimedia web an effective information repository.

what do we have to learn?

digital convergence

...

broadband communication

...

multimedia information retrieval

assignment

Annotated Tour in Amsterdam

multimedia

intended audience(s)


multimedia course


part(s)


This book is the result of developing the course notes for an introduction to multimedia for first year Computer Science and Information Science students. Hence, first of all, I like to thank the students that had to endure all the rough drafts of this material, and perhaps not to forget my experiment(s) with the presentation format of it.

references

  1.  [Convergence],  [Entertainment],  [Experience].
  2.  [Spaces],  [Hypermedia],  [Magic].
  3.  [Codecs],  [MPEG-4],  [Web].
  4.  [Game],  [Animation],  [Cg]
  5.  [MMDBMS],  [IR].
  6.  [MMDBMS],  [Meldex],  [ACOI].
  7.  [MMDBMS],  [Networked],
  8.  [Networked],  [Query],  [Community].
  9.  [Effects],  [Remediation],
  10.  [Chapman1],  [Chapman2],  [Magic],
  11.  [Ultimate],  [VirtualArt],
  12.  [HalfReal],  [Film],  [Cinema],  [Semiotics].

more than the art of turning base metals into gold, alchemy is a system of cosmic symbolism

perfect solutions

digital convergence


...

perspectives -- digital convergence


...

the artwork

  1. alchemy -- an illustration from a book about alchemy, from which also the quote is taken, the quote is explained in the afterthoughts.
  2. signs -- ancient chemical symbols,  [Signs], p. 171, 172.
  3. photographs -- Jaap Stahlie, from portrait series.

1

digital culture

life is becoming digital

digital convergence

learning objectives

After reading this chapter you should be able to define the notion of multimedia, recount the history of digital entertainment, explain the concept of digital convergence, discuss the future of cyberspace, and speculate about the commercial viability of mobile multimedia.

We live in the digital era,  [Negroponte (1995)]. We are surrounding ourselves with gadgets and we are consuming immense amounts of information, that is increasingly being delivered to us via the Internet. We play games, and we still watch (too much) television.

Some of us watch televion on our PCs, and may be even looking forward to watch television on their mobile phone. This is multimedia. For others, the PC is still a programmable machine. Being able to program it might earn you a living. Understanding multimedia, however, might even provide you with a better living.

In this chapter, we study what trends may currently be observed in the creation and delivery of multimedia information, and we explore what impact the digital revolution may have from a commercial perspective.

...

levels of meaning


cultural convergence


The cultural convergence of art, science, and technology provides ample opportunity for artists to challenge the very notion of how art is produced and to call into question its subject matter and its function in society.

standardization and uniformity


  1. Telematic media were incorporated very quickly in the globalization strategies of transnational corporations and their political administrators and they became increasingly dependent on existing power structures.
  2. At the other end of the scale, there were individuals, or comparatively small groups, who projected great hopes onto these networks as a testing ground for cultural, artistic and political models that would give greater prominence and weight to divergence and plurality.

Scientific American (november 2000)


The barriers between TV, movies, music, videogames and the Internet are crumbling.

Audiences are fetting new creative options.

Here is what entertainment could become if the technological and legal hurdles can be cleared ...

Underlying the importance of entertainment in the era of digital convergence is the premisse governing an entertainment economy, which may be stated as

there is no business without show business

evolution of digital entertainment


  • 1953: Winky Dink (CBS) -- interactive television, drawing exercise
  • 1972: Pong (Atari) -- ping-pong on computer screen
  • 1977: Adventure -- text-based interactive fiction
  • 1983: Dragon's Liar -- laser-disc technology 3D game
  • 1989: SimCity -- interactive simulation game
  • 1989: Back to the Future -- the Ride
  • 1993: Doom -- 3D action game
  • 1995: The Spot -- interactive web-based soap opera (Webisodic)
  • 1999: IMAX3D -- back to Atlantis (Las Vegas)
  • 2000: Big Brother -- TV + around the clock Web watch + voting
  • 2001: FE Sites -- fun enhanced web sites

experience is fundamental to human life


The desire to share experiences will be the motivating factor in the development of exciting multimedia technology in the foreseeable future.

communication technology


  • oral -- communicate symbolic experiences
  • writing -- record symbolic experiences
  • paper -- portability
  • print -- mass distribution
  • telegraph -- remote narrow communication
  • telephone -- remote analog communication
  • radio -- analog broadcasting of sound
  • television -- analog A/V broadcasting
  • recording media -- analog recording
  • digital processing -- machine enhancement
  • internet -- multimedia communication

the medium was the message when only one medium could be used to communicate messages.

...

...
VR for pain reliefimage delivery system

... cyberspace is a construct in terms of an electronic system.

cyberspace


television, video cassettes, video tape-recorder/players, video games, and personal computers all form an encompassing electronic system whose various forms interface to constitute an alternative and absolute world that uniquely incorporates the spectator/user in a spatially decentered, weakly temporalized and quasi-disembodied state.

virtual reality


virtual reality (is) when and where the computer disappears and you become the 'ghost in the machine' ...

history

the receiver at the RCA Pavillon was way ahead of its time, it was a combination of television - radio - recorder - playback - facsimile - projector ...

digital convergence

the union of audio, video and data communication into a single source, received on a single device, delivered by a single connection

subsidiary convergences


  • content -- audio, video, data
  • platform -- PC, TV, internet, game machine
  • distribution -- how it gets to your platform

...

convergence


  • content -- 2D/3D graphics, data, video, audio
  • distribution -- broadcast, wireless, DVD, internet, satelite, cable
  • platform -- PC, television, game machine, wireless data pad, mobile phone

...
exposition on the history of TV in Institute for Time-based Arts/Montevideo

acronyms


  • HDTV -- high definition television
  • SDTV -- standard definition television
  • ITV -- interactive television

a killer d-TV appliance ...

  • personal television -- TiVo, Replay-TV (MPEG-2 cache)
  • game machine -- Sony PS 2/3, X-Box

TV or PC


The roadblock to the Entertainment PC could be the PC itself. Even a cheap TV doesn't crash or freeze. The best computers still do.

distribution


  • telephone network -- from 0.5 - 2 Mbps to 60 Mpbs (2.5km)
  • broadcast TV -- 6 MHz / 19 Mbps (4 channels MPEG HDTV)
  • cable TV -- hybrid fiber-optic coaxial cable 6 Mbps
  • fixed wireless -- 2 Mbps (radiotowers + rooftop antenna), phones/handhelds
  • satellite -- downloads to 100kbps, modem for uploads ...

digital convergence


what will we do with convergence once we have it?

emergence

we will watch

...
Berkeley meshSan Francisco viewaugmented terrain map

Google Earth


media as materials


each medium of communication tended to create a dangerous monopoly of knowledge

technological determinism


technological determinism was not the answer, ... more attempts were to be made to provide answers about the social consequences of television than had ever been asked about radio.

information


Information became a major concern anywhere during the late 1960 and 1970s where there was simultaneous talk both of 'lack of information' and 'information saturation'.

 [Briggs and Burke (2001)], p. 555

...

Peter Greven 23/3/2001 (Volkskrant)


new media sucks

--

people like new technology.

they don't like new media.

www.tvmeetstheweb.com


streaming media (audio and video), interactive gaming, virtual reality and 3D animation, interactive TV programming, interactive advertising, video on-demand, webcasting and multimedia

strategic questions


  • how quickly will wireless connectivity speeds improve?
  • what is the demand for services that deliver music and video to wireless devices?
  • how can suppliers of multimedia services monetize demand for wireless access?
  • how much will it cost to stream multimedia content to wireless devices now and in 2006?
  • are consumers willing to compromise quality for lower cost?

the players


Alltel, AT&T Wireless, AtomShockwave, Cingular Wireless, Clear Channel, HitHive, Ifilm, Infinity, KDDI, Liquid Audio, LMIV, Mannesmann, MP3.com, MTV, NetCom, Myplay, Nortel Networks, NTT DoCoMo, Omnitel, Sprint, Telefonica, Telstra, Vitaminic, Verizon Wireless, Virgin Megastores, Vodafone, Voicestream.

...
Vasulka Objects (1)Vasulka Objects (2)

...
(a) personal events(b) gadgets

functions of media


information, education, entertainment

medium


television is a medium 'because it is neither rare nor well done'

information society


the new term 'information society' gave form to a cluster of hitherto more loosely related aspects of communication -- knowledge, news, literature, entertainment, all exchanged through different media and different media materials -- paper, ink, canvas, paint, celluloid, cinema, radio, television and computers.

From the 1960s onwards, all messages, public and private, verbal and visual, began to be considered as 'data', information that could be transmitted, collected, recorded, whatever their point of origin, most effective through electronic technology.

communication


  • what -- content
  • who -- control
  • whom -- audience (how many)

第二生命


Second Life seems to be overtaking the world. In the whole range of cummunity-building platforms, Second Life stands out as an immersive 3D world with an almost stunning adoption, by both individuals, companies and institutions, followed attentively by the Press. Not entirely without an understanding of the value of press coverage, the VU University Amsterdam decided to create presence in Second Life, by creating a virtual campus, to realize a (virtual) community of learners,  [Eliens et al. (2007)]. And, indeed, we succeeded in being the first university in The Netherlands with presence in Second Life and, as hoped, this was covered in the 8 o'clock nation-wide TV news.

...
(a) visitors outside(b) visitors inside

success factors (1/2)


  • convergence of social networking and content creation
  • immersive networked 3D environment
  • inclusion of elementary economic principles

success factors (2/2)


  • don't miss the boat effect
  • free and easy accessible 3D design tool set
  • adoption by big companies like IBM, Reebok, ...
  • marketing of Second Life by Linden Lab (?)
  • the promise to make (real) money (?)

reference model


  • rules -- construct and communicate!
  • outcome -- a second world
  • value -- virtual and real (monetary) rewards
  • effort -- requires elementary skills
  • attachment -- a virtual identity
  • consequences -- transfer to first life

critical theory


attempt(s) to link the arts, literature, media studies, politics, sociology, antropology, philosophy and technology in an interdisciplinary search for relevant concepts and frameworks with which to understand the current world.

contemporary perception(s)


... in the contemporary world, the perceptual task has changed, in both leisure and work, to monitor data displays, ready for events.

californian dream(s)


... the new faith has emerged from a bizarre fusion of the cultural bohemianism of San Francisco with the high-tech industries of Silocon Valley...

and, to further de-construct the digital utopianism:

... the californian ideology promiscuously combines the freewheeling spirit of the hippies and the entrepeneurial zeal of of the yuppies

digital class


... the shadow side of the digital class's freedom and individuality is a lack of connection ... and an unrealized acceptance of work as the main life value.

cyberfeminism(s)


... empowerment of women in the field of new electronic media can only result from the demystification of technology and the appropriation of access to (these) tools.

...

1. digital convergence

concepts


technology


projects & further reading

As a project, consider the development of a Java-based mobile game using J2ME, see  [Morrison (2005)], or a web-based game using Visual Basic .NET, see  [Santos Lobao and Hatton (2003)].

You may further explore multiplatform game development, and find arguments to choose for either Java-based or managed code based implementations.

For further reading, I advice to have a look at the special issues of the Scientific American,  [American], and the CACM on the next 1000 years of computing,  [CACM (2001)], and, for getting an idea where this all leads to, Schneidermann's Leonardo's laptop,  [Shneiderman (2003)]. For Second Life, see  [Rymaszweski et al. (2007)].

the artwork

  1. photographs of art works by Marina Abramovic, Art must be beautiful, Blue period, Dissolution, Dozing consciousness, In between, with (pending) permission from Montevideo. See also section 10.2.
  2. medium, according to the Visual Thesaurus.
  3. fMRI Research on Virtual Reality Analgesia, see section 1.1.
  4. television and communication, according to the Visual Thesaurus.
  5. TV Today, exhibition at Montevideo, februari 2005.
  6. visible world -- taken from  [Rosenblum and Macedonia (2002)], see section 1.2.
  7. personal event database and personal gadgets, from Freeband project.
  8. Thomas Lips 1975, Thomas Lips 1993, from Marina Abramovic, with permission from Montevideo.
  9. scanlines from Woody Vasulka, 197x, with permission from the artist.
  10. VU @ SecondLife, taken from  [Eliens et al. (2007)].
  11. signs -- people,  [ van Rooijen (2003)], p. 254, 256.

2

hypermedia information spaces

everything must be intertwinkled

hypermedia information spaces

learning objectives

After reading this chapter you should be able to define information spaces in a precise manner, position the hypertextual capabilities of the web in a historical perspective, explain the difference between multimedia and hypermedia, and argue why computational support for narrative structure in multimedia applications is desirable.

However entertaining it might be presented to you, underlying every multimedia presentation there is an information space. That is to say, irrespective of the medium, there is a message. And being confronted with a message, we might want to inquire for more information.

In this chapter, we will define the notion of information space more precisely.

We will extend this definition to include information hyperspaces, by looking at the history of hypertext and hypermedia.

Finally, we will discuss visualisation as a means to present (abstract) information in a more intuitive way, and we will reflect on what is involved in creating compelling multimedia.

...

Current day multimedia information systems distinguish themselves from older day information systems not only by what information they contain, that includes multimedia objects such as images and sounds, but also by a much more extensive repertoire of query mechanisms, visual interfaces and rich presentation facilities. See  [Chang and Costabile (1997)].

S.K. Chang and M.F. Costabile -- Visual Interfaces to Multimedia Databases


The Handbook of Multimedia Information Management


multimedia information systems

  • storage technology -- multimedia databases
  • wideband communication -- distribution accross networks
  • parallel computing -- voice, image and video processing
  • graphic co-processors -- visual information with high image quality

multimedia applications

geographical information systems, office automation, distance learning, health care, computer aided design, scientific visualization, and information visualization.

multimedia databases

  • the size of data,
  • synchronization issues,
  • query mechanisms, and
  • real time processing.

...

an information space is a representation of the information stored in a system or database that is used to present that information to a user.

we must distinguish between a visual information space (for presentation), a logical information space (in which we can reason about abstract information objects) and a physical information space (where our concrete multimedia objects are stored).

  • physical information space -- images, animations, video, voice, ...
  • logical information space -- abstract database objects
  • presentational information space -- to present information to the user

a logical information space is a multidimensional space where each point represents an object from the physical information space (read database).

  • information object -- a point in the (logical) information space
  • query -- an arbitrary region in this information space
  • clue -- a region with directional information, to facilitate browsing

...

www.w3.org/XML/1999/XML-in-10-points


XML is a set of rules (you may also think of them as guidelines or conventions) for designing text formats that let you structure your data.

XML in 10 points


  1. XML is for structuring data
  2. XML looks a bit like HTML
  3. XML is text, but isn't meant to be read
  4. XML is verbose by design
  5. XML is a family of technologies
  6. XML is new, but not that new
  7. XML leads HTML to XHTML
  8. XML is the basis for RDF and the Semantic Web
  9. XML is license-free, platform-independent and well-supported

related technologies


  • Xlink -- hyperlinks
  • XPointer -- anchors and fragments
  • XSL -- advanced stylesheets
  • XSLT -- transformation language
  • DOM -- object model for application programmer interface
  • schemas -- to specify the structure of XML documents

XML


  • separate data from presentation
  • transmit data between applications

...

information hyperspace


the logical information space may further be structured in a logical information hyperspace, where the clues become hyperlinks that provide directional information, and the information space can be navigated by the user following directional clues.

history


  • 1945 -- Vannevar Bush (Memex) -- as we may think,  [ Bush (1995)]
  • 1963 -- Douglas Engelbart (Augment) -- boosting the human intellect  [Engelbart (1963)]
  • 1980 -- Ted Nelson (Xanadu) -- everything is intertwinkled,  [Nelson (1980)]

  • flash 1: we are in trouble (human mankind)
  • flash 2: we need to boost mankind's ability to deal with complex urgent problems
  • flash 3: aha, graphic vision surges forth of me ...
  • flash 4: hypermedia -- to augment the human intellect
  • flash 5: augment (multimedia) workstation -- portal into an information space

hypermedia systems

  • components -- text, graphics, audio, video
  • links -- relations between components
  • presentation -- structured display

A curriculum promotes a false simplification of any subject, cutting the subject's many interconnections and leaving a skeleton of sequence which is only a charicature of its richness and intrinsic fascination.

classification of hypermedia systems

  • macro-literary systems -- publishing, reading, criticism
  • problem exploration tools -- authoring, outlining, programming
  • browsing systems -- teaching, references, information
  • general hypermedia technology -- authoring, browsing, collaboration
  • embedded hypermedia -- CASE, decision support, catalogs

...

component

  • content -- text, graphics, video, program
  • attributes -- semantic description
  • anchors -- (bi-directional) links to other documents
  • presentation -- display characteristics

(CMIF) multimedia model

  • data block -- atomic component
  • channel -- abstract output device
  • synchronization arc -- specifying timing constraints
  • event -- actual presentation

Amsterdam Hypermedia Model

  • contents -- data block
  • attributes -- semantic information
  • anchors -- (id, value)
  • presentation -- channel, duration, ...

...

research issues


  • search and query -- for better access
  • composition -- for imposing structure
  • virtual structures -- on top of existing structures
  • computation -- for flexibility and interaction
  • versioning -- to store modification histories
  • collaborative work -- sharing objects with multiple users
  • extensibility and tailorability -- to adapt to individual preferences

...

visualization


Grasping the whole is a gigantic theme, intellectual history's most important.

Ant vision is humanity's usual fate; but seeing the whole is every thinking person's aspiration.

David Gelernter, Mirror Worlds 1992


data types


  • 1-D linear data -- text, source code, word index
  • 2-D map data -- floor plan, office layout
  • 3-D world -- molecules, schematics, ...
  • temporal data -- 1 D (start, finish)
  • multi-dimensional data -- n-dimensional (information) space
  • tree data -- hierarchical
  • network data -- graph structure

  • interactive -- overview first, zoom and filter, then details on demand
  • storytelling -- as a paradigm for information presentation

Whatever your target audience, whatever your medium, whatever your message, you have to be convincing if not compelling.

persuasion

  • a communication process in which the communicator seeks to elicit a desired response from his receiver
  • a conscious attempt by one individual to change the attitudes, beliefs or behaviours of another individual or group individual through the transmission of some messages.

(re)mediation

  • immediacy -- a tendency towards transparent immersion, and
  • hypermediacy -- the presence of referential context

Virtual Reality won't merely replace TV. It will eat it alive.

immediacy

  • epistemological: transparency, the absence of mediation
  • psychological: the medium has disappeared, presence, immersion

hypermediacy

  • epistemological: opacity, presence of the medium and mediation
  • psychological: experience of the medium is an experience of the real

Convergence is the mutual remediation of at least three important technologies -- telephone, televison and computer -- each of which is a hybrid of technical, social and economic practice, and each of which offers its own path to immediacy.

The telephone offers the immediacy of voice or the interchange of voices in real-time.

Television is a point-of-view technology that promises immediacy through its insistent real-time monitoring of the world.

The computer's promise of immediacy comes through the combination of three-dimensional graphics, automatic (programmed) action, and an interactivity that television can not match.

As they come together, each of these is trying to absorb the others and promote its own version of immediacy.

...

convergence

(p. 27) ... merging previously disparate technologies of communication and representation into a single medium.

The networked computer acts like a telephone in offering one-to-one real-time communication, like a television in broadcasting moving pictures, like an auditorium in bringing groups together for lectures and discussion, like a library in offering vast amounts of textual information for reference, like a museum in its ordered presentation of visual information, like a billboard, a radio, a gameboard and even like a manuscript in its revival of scrolling text.

interactive


  • procedural -- 'programmed media' ...
  • participatory -- offering agency

...

immersive


  • spatial -- explorable in (state) space
  • encyclopedic -- with (partial) information closure

multimedia authoring


  • narrative format
  • procedural authorship

web 2.0


video sharing / online gaming / social networking

daft punk -- technologic (cn / jp) 



  Buy it, use it, break it, fix it.
  Trash it, change it, melt -- upgrade it.
  Change it, point it, zoom it, press it.
  Snap it, work it, quick -- erase it.
  Write it, out it, paste it, save it.
  Load it, check it, quick -- rewrite it.
  Plug it, play it, burn it, rip it.
  Drag and drop it, zip -- unzip it.
  Look it, fill it, curl it, find it.
  View it, coat it, jam -- unlock it.
  Surf it, scroll it, pose it, click it.
  Cross it, crack it, twitch -- update it.
  Name it, rate it, tune it, print it.
  Scan it, send it, fax -- rename it. 
  Touch it, bring it. Pay it, watch it.
  Turn it, leave it, stop -- format it.

mashup(s)


  • substituting a single pragmatism for ideal design
  • using light weight programming models

web 2.0 design pattern(s)


  • web 1.0 -- the web as platform
  • web 2.0 -- architecture of participation
  • web 3.0 -- data is the (intel) inside

Learnlog: XML Is The Fabric Of Web 2.0 Applications

  • the client side is AJAX (Asynchronous Javascript and XML)
  • the server application typically exposes data through XML
  • the interaction model is web services
  • mashups combine multiple webservices to create new types of applications

service(s)


REST


  • representation -- encoding in a particular format
  • state -- data encapsulated in an object
  • transfer -- using HTTP methods

the most powerful mashups don't just mix code and data, they mix cultures.

which provides a challenge that trancends all issues of mere technological correctness.

built-in(s)


resource(s)


  • /seen?user=SomeAvatar -- records the presence of SomeAvatar
  • /touched?user=SomeAvatar -- invokes flickr API with users tag
  • /set_tag?user=SomeAvatar&tag=FavoriteTag -- records SomeAvatar's favourite tag

flash/quicktime in SL


Quicktime supports Flash, but only up to Flash version 5. We're up to version 9 on that now! Luckily, I have been dabbling with Flash since the early days, so already knew how to do this 'the old way'... So, Flash is doing all the work. No LSL at all... I heart feeds. Did I say 'I heart feeds?

Referring to section 7.4 for a more detailed discussion, we may observe that there is no meaning in merely putting things together. Without mechanisms of personalization and recommendation we would simply be flooded by data and information, in a way that even search would not be able to cope with. Context, narratives and personalized presentation(s), notions from the past, reappear as keywords for the future of the web 2.0 and beyond.

...

2. information spaces

concepts


technology


projects & further reading

As a project, I suggest the development of a virtual tour in a city, museum or other interesting locatoion.

You may further explore the implementation of traversal within a context, taking into account the history of navigation when backtracking to a particular point, issues in hyperlinking and interaction in multimedia applications, and computational support for narratives.

For further reading I advice you to take a look at the history of hypermedia and the web, using online material from the W3C, or the history of media as accounted for in  [Briggs and Burke (2001)] and  [Bolter and Grusin (2000)].

the artwork

  1. book covers --  [Weishar (1998)],  [Eco (1994)],  [Burger (1981)],  [Kunst],  [Betsky (2004)]
  2. Federico Campanale -- Oxygen, fragments from video installation, 2004
  3. Vasarely --  [Diehl 1973].
  4. Vasarely --  [Diehl 1973].
  5. Vasarely --  [Diehl 1973].
  6. Federico Campanale -- Oxygen, more fragments.
  7. student work -- from introduction multimedia 2000.
  8. Rutger van Dijk -- mobius, interactive story, opening screen, see section 2.3.
  9. edgecodes -- screenshots, see section 2.3
  10. signs -- people,  [ van Rooijen (2003)], p. 244, 245.

if you linger for a long time in one place you'd almost think there must be something there

wittgenstein

delivery & presentation


...

perspectives -- delivery & presentation


  • technical -- codec selection
  • political -- market vs. consortium
  • sociological -- digital services
  • legal -- copyright protection
  • scientific -- experience design
  • computer science -- computational support
  • futuristic -- global & personal information
  • commercial -- WMV, Quicktime, RealONE

...

the artwork

  1. logo -- a drawing by Soutine, it is (almost) my personal logo, and also decorates the cover of  [Eliens (2000)].
  2. signs -- property marks,  [ van Rooijen (2003)], p. 76, 77.
  3. photographs -- Jaap Stahlie, commissioned work.

3

codecs and standards

without compression delivery is virtually impossible

codecs and standards

learning objectives

After reading this chapter you should be able to demonstrate the necessity of compression, to discuss criteria for the selection of codecs and mention some of the alternatives, to characterize the MPEG-4 and SMIL standards, to explain the difference between MPEG-4 and MPEG-2, and to speculate about the feasibility of a semantic multimedia web.

Without compression and decompression, digital information delivery would be virtually impossible. In this chapter we will take a more detailed look at compression and decompression. It contains the information that you may possibly need to decide on a suitable compression and decompression scheme (codec) for your future multimedia productions.

We will also discuss the standards that may govern the future (multimedia) Web, including MPEG-4, SMIL and RM3D. We will explore to what extent these standards allow us to realize the optimal multimedia platform, that is one that embodies digital convergence in its full potential. Finally, we will investigate how these ideas may ultimately lead to a (multimedia) semantic web.

...

compression is the key to effective delivery

mediauncompressedcompressed
voice 8k samples/sec, 8 bits/sample64 kbps2-4 kbps
slow motion video 10fps 176x120 8 bits5.07 Mbps8-16 kbps
audio conference 8k samples/sec 8bits64 kbps16-64 kbps
video conference 15 fps 352x240 8bits30.4 Mbps64-768 kbps
audio (stereo) 44.1 k samples/s 16 bits1.5 Mbps128k-1.5Mbps
video 15 fps 352x240 15 fps 8 bits30.4 Mbps384 kbps
video (CDROM) 30 fps 352x240 8 bits60.8 Mbps1.5-4 Mbps
video (broadcast) 30 fps 720x480 8 bits248.8 Mbps3-8 Mbps
HDTV 59.9 fps 1280x720 8 bits1.3 Gbps20 Mbps

(phone: 56 Kb/s, ISDN: 64-128 Kb/s, cable: 0.5-1 Mb/s, DSL: 0.5-2 Mb/s)

images, video and audio are amenable to compression

statistical redundancy in signal


  • spatial correlation -- neighbour samples in single frame
  • temporal correlation -- between segments (frames)

irrelevant information


  • from perceptual point of view

B. Vasudev & W. Li, Memory management: Codecs


codec = (en)coder + decoder



  signal  -> source coder   ->  channel coder    (encoding)
  
  signal  <- source decoder <-  channel decoder  (decoding)

codec design problem


From a systems design viewpoint, one can restate the codec design problem as a bit rate minimization problem, meeting (among others) constraints concerning:

  • specified levels of signal quality,
  • implementation complexity, and
  • communication delay (start coding -- end decoding).

...

tradeoffs

  • resilience to transmission errors
  • degradations in decoder output -- lossless or lossy
  • data representation -- browsing & inspection
  • data modalities -- audio & video.
  • transcoding to other formats -- interoperability
  • coding efficiency -- compression ratio
  • coder complexity -- processor and memory requirements
  • signal quality -- bit error probability, signal/noise ratio

  • pixel-based -- MPEG-1, MPEG-2, H3.20, H3.24
  • object-based -- MPEG-4

MPEG-1 video compression uses both intra-frame analysis, for the compression of individual frames (which are like images), as well as. inter-frame analysis, to detect redundant blocks or invariants between frames.

frames


  • I: intra-frames -- independent images
  • P: computed from closest frame using DCT (or from P frame)
  • B: computed from two closest P or I frames

GigaPort


  • optical network technologies - models for network architecture, optical network components and light path provisioning.
  • high performance routing and switching - new routing technologies and transport protocols, with a focus on scalability and stability robustness when using data-intensive applications with a high bandwidth demand.
  • management and monitoring - incident response in hybrid networks (IP and optical combined) and technologies for network performance monitoring, measuring and reporting.
  • grids and access - models, interfaces and protocols for user access to network and grid facilities.
  • test methodology - effective testing methods and designing tests for new technologies and network components.

system spatial resolution frame rate mbps
NTSC704 x 480 30 243 mbps
PAL/SECAM 720 x 576 25 249 mbps

item streaming downloaded
bandwidth equal to the display rate may be arbitrarily small
disk storage none the entire file must be stored
startup delay almost none equal to the download time
resolution depends on available bandwidth depends on available disk storage

formats


Quicktime, introduced by Apple, early 1990s, for local viewing; RealVideo, streaming video from RealNetworks; and Windows Media, a proprietary encoding scheme fromMicrosoft.

Examples of these formats, encoded for various bitrates are available at Video at VU.

...

standards


  • XML -- eXtensible Markup Language (SGML)
  • MPEG-4 -- coding audio-visual information
  • SMIL -- Synchronized Multimedia Integration Language
  • RM3D -- (Web3D) Rich Media 3D (extensions of X3D/VRML)

...

"Perhaps the most immediate need for MPEG-4 is defensive. It supplies tools with which to create uniform (and top-quality) audio and video encoders on the Internet, preempting what may become an unmanageable tangle of proprietary formats."

MPEG-4


a toolbox of advanced compression algorithms for audiovisual information

scalability

  • bitrate -- switching to lower bitrates
  • bandwidth -- dynamically discard data
  • encoder and decoder complexity -- signal quality

...

audiovisual information


  • still images, video, audio, text
  • (synthetic) talking heads and synthesized speech
  • synthetic graphics and 3D scenes
  • streamed data applied to media objects
  • user interaction -- e.g. changes of viewpoint

example


Imagine, a talking figure standing next to a desk and a projection screen, explaining the contents of a video that is being projected on the screen, pointing at a globe that stands on the desk. The user that is watching that scene decides to change from viewpoint to get a better look at the globe ...

media objects


  • media objects -- units of aural, visual or audiovisual content
  • composition -- to create compound media objects (audiovisual scene)
  • transport -- multiplex and synchronize data associated with media objects
  • interaction -- feedback from users' interaction with audiovisual scene

composition


  • placing media objects anywhere in a given coordinate system
  • applying transforms to change the appearance of a media object
  • applying streamed data to media objects
  • modifying the users viewpoint

transport


The data stream (Elementary Streams) that result from the coding process can be transmitted or stored separately and need to be composed so as to create the actual multimedia presentation at the receivers side.

scenegraph


  • BIFS (Binary Format for Scenes) -- describes spatio-temporal arrangements of (media) objects in the scene
  • OD (Object Descriptor) -- defines the relationship between the elementary streams associated with an object
  • event routing -- to handle user interaction

...

DMIF


Delivery Multimedia Integration Framework

...
(a) scene graph (b) sprites

benefits


  • end-users -- interactive media accross all platforms and networks
  • providers -- transparent information for transport optimization
  • authors -- reusable content, protection and flexibility

managing intellectual property

...

XMT


  • XMT contains a subset of X3D
  • SMIL is mapped (incompletely) to XMT

...

SMIL


TV-like multimedia presentations

parallel and sequential


Authoring a SMIL presentation comes down, basically, to

name media components for text, images,audio and video with URLs, and to schedule their presentation either in parallel or in sequence.

presentation characteristics


  • The presentation is composed from several components that are accessible via URL's, e.g. files stored on a Web server.
  • The components have different media types, such as audio, video, image or text. The begin and end times of different components are specified relative to events in other media components. For example, in a slide show, a particular slide is displayed when the narrator in the audio starts talking about it.
  • Familiar looking control buttons such as stop, fast-forward and rewind allow the user to interrupt the presentation and to move forwards or backwards to another point in the presentation.
  • Additional functions are "random access", i.e. the presentation can be started anywhere, and "slow motion", i.e. the presentation is played slower than at its original speed.
  • The user can follow hyperlinks embedded in the presentation.

applications


  • Photos taken with a digital camera can be coordinated with a commentary
  • Training courses can be devised integrating voice and images.
  • A Web site showing the items for sale, might show photos of the product range in turn on the screen, coupled with a voice talking about each as it appears.
  • Slide presentations on the Web written in HTML might be timed so that bullet points come up in sequence at specified time intervals, changing color as they become the focus of attention.
  • On-screen controls might be used to stop and start music.

example



   <par>
      <a href="#Story"> <img src="button1.jpg"/> </a>
      <a href="#Weather"> <img src="button2.jpg"/></a>
       <excl>
           <par id="Story" begin="0s">
             <video src="video1.mpg"/>
             <text src="captions.html"/>
           </par>
  
           <par id="Weather">
             <img src="weather.jpg"/>
             <audio src="weather-rpt.mp3"/>
           </par>
       </excl>
   </par>

history


Experience from both the CD-ROM community and from the Web multimedia community suggested that it would be beneficial to adopt a declarative format for expressing media synchronization on the Web as an alternative and complementary approach to scripting languages.

Following a workshop in October 1996, W3C established a first working group on synchronized multimedia in March 1997. This group focused on the design of a declarative language and the work gave rise to SMIL 1.0 becoming a W3C Recommendation in June 1998.

SMIL 2.0 Modules


  • The Animation Modules
  • The Content Control Modules
  • The Layout Modules
  • The Linking Modules
  • The Media Object Modules
  • The Metainformation Module
  • The Structure Module
  • The Timing and Synchronization Module
  • The Time Manipulations Module
  • The Transition Effects Module

module-based reuse


  • SMIL modules could be used to provide lightweight multimedia functionality on mobile phones, and to integrate timing into profiles such as the WAP forum's WML language, or XHTML Basic.
  • SMIL timing, content control, and media objects could be used to coordinate broadcast and Web content in an enhanced-TV application.
  • SMIL Animation is being used to integrate animation into W3C's Scalable Vector Graphics language (SVG).
  • Several SMIL modules are being considered as part of a textual representation for MPEG4.

...

www.web3d.org


  • VRML 1.0 -- static 3D worlds
  • VRML 2.0 or VRML97 -- dynamic behaviors
  • VRML200x -- extensions
  • X3D -- XML syntax
  • RM3D -- Rich Media in 3D

groups.yahoo.com/group/rm3d/


The Web3D Rich Media Working Group was formed to develop a Rich Media standard format (RM3D) for use in next-generation media devices. It is a highly active group with participants from a broad range of companies including 3Dlabs, ATI, Eyematic, OpenWorlds, Out of the Blue Design, Shout Interactive, Sony, Uma, and others.

RM3D


The Web3D Consortium initiative is fueled by a clear need for a standard high performance Rich Media format. Bringing together content creators with successful graphics hardware and software experts to define RM3D will ensure that the new standard addresses authoring and delivery of a new breed of interactive applications.

...

requirements


  • rich media -- audio, video, images, 2D & 3D graphics (with support for temporal behavior, streaming and synchronisation)
  • applicability -- specific application areas, as determined by commercial needs and experience of working group members

  • interoperability -- VRML97, X3D, MPEG-4, XML (DOM access)

  • object model -- common model for representation of objects and capabilities
  • extensibility -- integration of new objects (defined in Java or C++), scripting capabilities and declarative content

  • high-quality realtime rendering -- realtime interactive media experiences
  • platform adaptability -- query function for programmatic behavior selection
  • predictable behavior -- well-defined order of execution
  • high precision number systems -- greater than single-precision IEEE floating point numbers
  • minimal size -- download and memory footprint

SMIL is closer to the author and RM3D is closer to the implementer.

...

working draft


Since there are three vastly different proposals for this section (time model), the original <RM3D> 97 text is kept. Once the issues concerning time-dependent nodes are resolved, this section can be modified appropriately.

time model


  • MPEG-4 -- spring metaphor
  • SMIL -- cascading time
  • RM3D/VRML -- event routing

MPEG-4 -- spring metaphor


  • duration -- minimal, maximal, optimal

SMIL -- cascading time


  • time container -- speed, accelerate, decelerate, reverse, synchronize


  <seq speed="2.0">
     <video src="movie1.mpg" dur="10s"/>
     <video src="movie2.mpg" dur="10s"/>
     <img src="img1.jpg" begin="2s" dur="10s">
                 <animateMotion from="-100,0" to="0,0" dur="10s"/>
     </img>
     <video src="movie4.mpg" dur="10s"/>
  </seq>

RM3D/VRML -- event routing


  • TimeSensor -- isActive, start, end, cycleTime, fraction, loop

...

...

web content

  • 1st generation -- hand-coded HTML pages
  • 2nd generation -- templates with content and style
  • 3rd generation -- rich markup with metadata (XML)

structure to the meaningful content of web pages,

...

meta data


Metadata is data about data. Specifically, the term refers to data used to identify, describe, or locate information resources, whether these resources are physical or electronic. While structured metadata processed by computers is relatively new, the basic concept of metadata has been used for many years in helping manage and use large collections of information. Library card catalogs are a familiar example of such metadata.

Dublin Core example



  <rdf:RDF
      xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
      xmlns:dc="http://purl.org/dc/elements/1.1/"
      xmlns:dcterms="http://purl.org/dc/terms/">
      <rdf:Description rdf:about="http://www.dlib.org/dlib/may98/miller/05miller.html">
        <dc:title>An Introduction to the Resource Description Framework</dc:title>
        <dc:creator>Eric J. Miller</dc:creator>
        <dc:description>The Resource Description Framework (RDF) is an
         infrastructure that enables the encoding, exchange and reuse of
         structured metadata. rdf is an application of xml that imposes needed
         structural constraints to provide unambiguous methods of expressing
         semantics. rdf additionally provides a means for publishing both
         human-readable and machine-processable vocabularies designed to
         encourage the reuse and extension of metadata semantics among
         disparate information communities. the structural constraints rdf
         imposes to support the consistent encoding and exchange of
         standardized metadata provides for the interchangeability of separate
         packages of metadata defined by different resource description
         communities. </dc:description>
        <dc:publisher>Corporation for National Research Initiatives</dc:publisher>
        <dc:subject>
          <rdf:Bag>
            <rdf:li>machine-readable catalog record formats</rdf:li>
            <rdf:li>applications of computer file organization and
             access methods</rdf:li>
          </rdf:Bag>
        </dc:subject>
        <dc:rights>Copyright © 1998 Eric Miller</dc:rights>
        <dc:type>Electronic Document</dc:type>
        <dc:format>text/html</dc:format>
        <dc:language>en</dc:language>
        <dcterms:isPartOf rdf:resource="http://www.dlib.org/dlib/may98/05contents.html"/>
      </rdf:Description>
  </rdf:RDF>

Dublin Core


  • title -- name given to the resource
  • creator -- entity primarily responsible for making the content of the resource
  • subject -- topic of the content of the resource
  • description -- an account of the content of the resource
  • publisher -- entity responsible for making the resource available
  • contributor -- entity responsible for making contributions to the content of the resource
  • date -- date of an event in the lifecycle of the resource
  • type -- nature or genre of the content of the resource
  • format -- physical or digital manifestation of the resource
  • identifier -- unambiguous reference to the resource within a given context
  • source -- reference to a resource from which the present resource is derived
  • language -- language of the intellectual content of the resource
  • relation -- reference to a related resource
  • coverage -- extent or scope of the content of the resource
  • rights -- information about rights held in and over the resource

...

information repository


The Web is becoming a universal repository of human knowledge and culture, which has allowed unprecedented sharing of ideas and information in a scale never seen before.

browsing & navigation


To satisfy his information need, the user might navigate the hyperspace of web links searching for information of interest. However, since the hyperspace is vast and almost unknown, such a navigation task is usually inefficient.

information agent


  • gather information
  • filter and select

presentation agent


  • access information
  • find suitable mode of presentation

...

PERsonal and SOcial NAvigation through information spaceS

PERSONAS


investigating a new approach to navigation through information spaces, based on a personalised and social navigational paradigm.

Agneta & Frieda


The AGNETA & FRIDA system seeks to integrate web-browsing and narrative into a joint mode. Below the browser window (on the desktop) are placed two female characters, sitting in their livingroom chairs, watching the browser during the session (more or less like watching television). Agneta and Frida (mother and daughter) physically react, comment, make ironic remarks about and develop stories around the information presented in the browser (primarily to each other), but are also sensitive to what the navigator is doing and possible malfunctions of the browser or server.

Agneta & Frieda


In this way they seek to attach emotional, comical or anecdotal connotations to the information and happenings in the browsing session. Through an activity slider, the navigator can decide on how active she wants the characters to be, depending on the purpose of the browsing session (serious information seeking, wayfinding, exploration or entertainment browsing).

game as social system


actorsrule(s)resource(s)
players eventsgame space
rolesevaluationsituation
goalsfacilitator(s)context

criteria


  • relevance -- what is our message?
  • identity -- who are we?
  • impact -- why would anybody be interested?

climate star


  • climate strategies -- (1) emission reduction, (2) adaptation
  • climate systems -- (3) feedback monitoring, (4) investment in research, (5) climate response
  • energy and CO2 -- (6) investment in efficiency, (7) investment in green technology, (8) governement rules
  • regional development -- (9) campain for awareness, (10) securing food and water
  • adaptation measures -- (11) public space, (12) water management, (13) use of natural resources
  • international relations -- (14) CO2 emission trade, (15) European negotiations, (16) international convenants

simulation parameters


  • people -- how is the policy judged by the people?
  • profit -- what is the influence on the (national) economy?
  • planet -- what are the effects for the environment?

...
game play, model-based simulation, exploration

game elements


  1. game cycle -- turns in subsequent rounds (G)
  2. simulation(s) -- based on (world) climate model (W)
  3. exploration -- by means of interactive video (E)

argument(s)


  • topic-centered -- common beliefs, use of logic, examples
  • viewer-centered -- patriotisms, religious or romantic sentimentality
  • speaker-centered -- the makers are well-informed, sincere and trusthworthy

...

3. codecs and standards

concepts


technology


projects & further reading

As a project, you may think of implementing for example JPEG compression, following  [Li and Drew (2004)], or a SMIL-based application for cultural heritage.

You may further explore the technical issues on authoring DV material, using any of the Adobe, mentioned in appendix E. or compare

For further reading I advice you to take a look at the respective specifications of MPEG-4 and SMIL, and compare the functionality of MPEG-4 and SMIL-based presentation environments. An invaluable book dealing with the many technical aspects of compression and standards in  [Li and Drew (2004)].

the artwork

  1. costume designs -- photographed from Die Russchische Avantgarde und die Buhne 1890-1930
  2. theatre scene design, also from (above)
  3. dance Erica Russel,  [Wiedermann (2004)]
  4. MPEG-4 -- bits rates, from  [Koenen (2000)].
  5. MPEG-4 -- scene positioning, from  [Koenen (2000)].
  6. MPEG-4 -- up and downstream data, from  [Koenen (2000)].
  7. MPEG-4 -- left: scene graph; right: sprites, from  [Koenen (2000)].
  8. MPEG-4 -- syntax, from  [Koenen (2000)].
  9. MIT Media Lab web site.
  10. student work -- multimedia authoring I, dutch windmill.
  11. student work -- multimedia authoring I, Schröder house.
  12. student work -- multimedia authoring I, train station.
  13. animation -- Joan Gratch, from  [Wiedermann (2004)].
  14. animation -- Joan Gratch, from  [Wiedermann (2004)].
  15. animation -- Joan Gratch, from  [Wiedermann (2004)].
  16. animation -- Joan Gratch, from  [Wiedermann (2004)].
  17. Agneta and Frieda example.
  18. diagram (Clima Futura) game elements
  19. signs -- people,  [ van Rooijen (2003)], p. 246, 247.

4

multimedia platforms

with DirectX 9 digital convergence becomes a reality

multimedia platforms

learning objectives

After reading this chapter you should be able to characterize the functionality of current multimedia platforms, to describe the capabilities of GPUs, to mention the components of the Microsoft DirectX 9 SDK, and to discuss how to integrate 3D and video.

Almost 15 years ago I bought my first multimedia PC, with Windows 3.1 Media Edition. This setup included a video capture card and a 4K baud modem. It was, if I remember well, a 100 Mhz machine, with 16 Mb memory and a 100 Mb disk. At that time, expensive as it was, the best I could afford. Some 4 years later, I acquired a Sun Sparc 1 multimedia workstation, with a video capture card and 3D hardware accelerator. It allowed for programming OpenGL in C++ with the GNU gcc compiler, and I could do live video texture mapping at a frame rate of about one per second. If you consider what is common nowadays, a 3Ghz machine with powerful GPU, 1 Gb of memory, a 1.5Mb cable or ADSL connection and over 100 Gb of disk space, you realize what progress has been made over the last 10 years.

In this chapter, we will look in more detail at the capability of current multimedia platforms, and we will explore the functionality of the Microsoft DirectX 0 platform. In the final section of this chapter, I will then report about the work I did with the DirectX 9 SDK to implement the ViP system, a presentation system that merges video and 3D.

...

4 generations of GPU


  • Before the introduction of the GPU, there only existed very expensive specialized hardware such as the machines from SGI.
  • The first generation of GPU, including NVIDIA TNT2, ATI Rage and 3dfx Voodoo3, only supported rasterizing pre-transformed triangles and some limited texture operations.
  • The second generation of GPUs, which were introduced around 1999, included the NVIDIA GeForce 2 and ATI Radeon 7500. They allowed for both 3D vertex transformations and some lighting, conformant with OpenGL and DirectX 7.
  • The tird generation GPUs, including NVIDIA GeForce 3, Microsoft Xbox and ATI Radeon 8500, included both powerful vertex processing capabilities and some pixel-based configuration operations, exceeding those of OpenGL and DirectX 7.
  • Finally, the fourth generation of GPUs, such as the NVIDIA GeForce FX and ATI Radeon 9700, allow for both complex vertex and pixel operations.

graphics pipeline


  1. vertex transformation -- apply world, view, projection transforms
  2. assembly and rasterization -- combine, clip and determine pixel locations
  3. fragment texturing and coloring -- determine pixel colors
  4. raster operations -- update pixel values

...
A simple morphing shader in ViP, see section 4.3.

HLSL declarations



  texture tex;
  float4x4 wvp;	// World * View * Projection matrix
  
  sampler tex_sampler = sampler_state
  {
      texture = /;    
  };

vertex shader data flow



  struct vsinput {
      float4 position : POSITION; 
      float3 normal : NORMAL; 
      float2 uv : TEXCOORD0;
  };
  struct vsoutput {
      float4 position   : POSITION;   // vertex position 
      float4 color    : COLOR0;     // vertex diffuse color
      float2 uv  : TEXCOORD0;  // vertex texture coords 
  };

vertex shader



  vsoutput vs_id( vsinput vx ) {
      vsoutput vs;
    
      vs.position = mul(vx.position, wvp);
      vs.color = color;
      vs.uv = vx.uv; 
      
      return vs;    
  }

pixel shader



  struct psoutput
  {
      float4 color : COLOR0;  
  };
  
  
  psoutput ps_id( vsoutput vs ) 
  { 
      psoutput ps;
  
      ps.color = tex2D(tex_sampler, vs.uv) * vs.color;
  
      return ps;
  }

technique selection



  technique render_id
  {
      pass P0
      {          
          VertexShader = compile vs_1_1 vs_id();
          PixelShader  = compile ps_2_0 ps_id(); 
      }
  }

...
Examples of Impasto, see examples -- impasto

morphing (vertex) shader



     float3 spherePos = normalize(vx.position.xyz);
     float3 cubePos = 0.9 * vx.position.xyz;
  
     float t = frac(speed * time);
     t = smoothstep(0, 0.5, t) - smoothstep(0.5, 1, t);
  
     // find the interpolation factor
     float lrp = lerpMin + (lerpMax - lerpMin) * t;
  
     // linearly interpolate the position and normal
     vx.position.xyz = lerp(spherePos, cubePos, lrp);
     vx.normal = lerp(sphereNormal, cubeNormal, lrp);
  
     // apply the transformations
     vs.position = mul(wvp, vx.position);

coloring (pixel) shader



      float4 x = tex2D(tex_sampler, vs.uv);
      if (x.r > x.g && x.r > x.b) { x.r *= xi; x.g *= xd; x.b *= xd; }
      else  if (x.g > x.r && x.g > x.b) { x.g *= xi; x.r *= xd; x.b *= xd; }
      else  if (x.b > x.r && x.b > x.g) { x.b *= xi; x.r *= xd; x.g *= xd; }
      ps.color = x;

...
rendering of van Gogh painting with Impasto

...
Stacks and stacks of books on DirectX

DirectX


Microsoft DirectX is an advanced suite of multimedia application programming interfaces (APIs) built into Microsoft Windows; operating systems. DirectX provides a standard development platform for Windows-based PCs by enabling software developers to access specialized hardware features without having to write hardware-specific code. This technology was first introduced in 1995 and is a recognized standard for multimedia application development on the Windows platform.

DirectX 9 components


  • Direct3D -- for graphics, both 2D and 3D
  • DirectInput -- supporting a variety of input devices
  • DirectPlay -- for multiplayer networked games
  • DirectSound -- for high performance audio
  • DirectMusic -- to manipulate (non-linear) musical tracks
  • DirectShow -- for capture and playback of multimedia (video) streams

Direct3D tutorials


  • tutorial 1: creating a device
  • tutorial 2: rendering vertices
  • tutorial 3: using matrices
  • tutorial 4: creating and using lights
  • tutorial 5: using texture maps
  • tutorial 6: using meshes 


  class  drumpad {
  public:
      drumpad()
      ~drumpad();
      bool    initialize( DWORD dwNumElements, HWND hwnd );
      bool    load( DWORD dwID, const TCHAR* tcszFilename );
      bool    play( DWORD dwID );
  protected:
      void    CleanUp();
      CSoundManager* m_lpSoundManager;
      CSound **      m_lpSamples;
  };
  


  bool drumpad::play( DWORD id ) {
      m_lpSamples[id] -> Stop();
      m_lpSamples[id] -> Reset();
      m_lpSamples[id] -> Play( 0, 0 );
      return true;
  }

multimedia challenges


  • volume -- multimedia streams contain large amounts of data, which must be processed very quickly.
  • synchronization -- audio and video must be synchronized so that it starts and stops at the same time, and plays at the same rate.
  • delivery -- data can come from many sources, including local files, computer networks, television broadcasts, and video cameras.
  • formats -- data comes in a variety of formats, such as Audio-Video Interleaved (AVI), Advanced Streaming Format (ASF), Motion Picture Experts Group (MPEG), and Digital Video (DV).
  • devices -- the programmer does not know in advance what hardware devices will be present on the end-user's system.

...

...
DepthCube, see example(s) -- 3D vision

...
3D visionPerspectraDepthCube

  • platform-dependence -- both hardware and OS
  • programming language -- C/C++, Java, .NET languages
  • functionality -- graphics, streaming media
  • deployment -- PC/PDA, local or networked, web deployment

Software will be the single most important force in digital entertainment over the next decade.

...
Animation in front of television news in ViP

www.virtualpoetry.tv


The ViP system enhances your party with innovative multimedia presentations.

It supports multiple webcams and digital video cameras, mixed with video and images, enhanced by 3D animations and text, in an integrated fashion.

For your party, we create a ViP presentation, with your content and special effects, to entertain your audience.

...

...
(a) VMR filter(b) multiple VMRs

...
Lecture on digital dossier for Abramovic, in ViP 


  class  scene  {
  public:
  	virtual int init(IDirect3DDevice9*);   // initialize scene (once)
  	virtual int compose(float time);  // compose (in the case of an animation)
  	virtual int restore(IDirect3DDevice9*);  // restore device settings
  	virtual int render(IDirect3DDevice9*  device,  IDirect3DTexture9* texture); 
  protected:
  ...
  };
  


  class  scene  {
  public:
  	virtual int load();   // initialize scene (once)
  	virtual int compose();  // compose (in the case of an animation)
  	virtual int restore();  // restore device settings
  	virtual int render();  // display the (sub) scene
  protected:
  ...
  };

...
installationreality of TV news

comparative overview


BlC AW D3D HL2 SL
in-game building -++/--++
avatar manipulation ++++/-+++
artifical intelligence +-+/-+-
server-side scripts +-+/-+++
client-side scripts ++-+/-+-
extensibility +-++++/-
open source --++-+/-
open standards --+/--+/-
interaction +/-+/-+++++/-
graphics quality +/-+/-+++++
built-in physics --++++
object collision --+++++
content tool support +/--+++-

...
Clima Futura architecture

module(s)


  1. climate model(s) - action script module(s)
  2. game play interaction - event-handler per game event
  3. video content module - video fragment(s) and interaction overlays
  4. minigame(s) - flash module(s) with actionscript interface
  5. Clima Futura - integration of modules 1-4, plus server-side ranking
  6. adapted versions -- educational, commercial
  7. multi-user version --with server-side support

interactive application(s)


  • realtime interaction -- providing information
  • content navigation -- providing view(s)

collada


  • document(s) -- schema, asset, library, technique, ...
  • geometry -- array, accessor, meshes, vertices, polygons, ...
  • scene(s) -- material, light, optics, camera, imager, ...
  • effect(s) -- shader, profiles, techniques, pass, ...
  • animation(s) -- sampler, channel, controller, skin, morphing, ...
  • physics -- collision, equation, rigid body, constraints, force, ...

...

4. multimedia platforms

concepts


technology


projects & further reading

As a project, I suggest the development of shader programs using Rendermonkey or the Cg Toolkit, or a simple game in DirectX.

You may further explore the possibilities of platform independent integration of 3D and media, by studying for example OpenML. For further reading, among the many books about DirectX, I advice  [Luna (2003)],  [Adams (2003)] and  [Fay et al. (2004)].

the artwork

  1. dutch light -- photographs from documentary film Dutch Light.
  2. ViP -- screenshot, with morphing shader, see section 4.3.
  3. impasto -- examples, see section 4.1
  4. impasto -- after a painting of van Gogh, using Cg shaders,
  5. 3D vision, from  [Sullivan (2005)], see example(s) section 4.2.
  6. idem.
  7. photographs of DirectX and multimedia books, by the author.
  8. DirectX -- diagram from online documentation.
  9. ViP -- screenshot, with the news and animations.
  10. DirectX -- diagram from online documentation.
  11. DirectX -- diagram from online documentation.
  12. ViP -- screenshot, featuring Abramovic.
  13. Peter Frucht -- Reality of TV news, see section 4.3.
  14. Clima Futura -- architecture diagram.
  15. signs -- people,  [ van Rooijen (2003)], p. 248, 249.

.. my history might well be your future ...

ted nelson

information retrieval


...

perspectives -- multimedia information retrieval


  • application(s) -- digital dossier
  • psychological -- focus
  • experimental -- user interaction
  • algorithmic -- (information) access
  • system -- unified presentation space
  • presentation -- embodied agents
  • search -- semantic annotation
  • commercial -- future systems

...

the artwork

  1. kata -- japanese martial arts picture.
  2. signs -- japanese coats of arms,  [ van Rooijen (2003)], p. 140, 141.
  3. photographs -- Jaap Stahlie, two early experiments (left, and right)

5

information retrieval

information retrieval is usually an afterthought

information retrieval

learning objectives

After reading this chapter you should be able to describe scenarios for information retrieval, to explain how content analysis for images can be done, to characterize similarity metrics, to define the notions of recall and precision, and to give an example of frequence tables, as used in text search.

Searching for information on the web is cumbersome. Given our experiences today, we may not even want to think about searching for multimedia information on the (multimedia) web.

Nevertheless, in this chapter we will briefly sketch one of the possible scenarios indicating the need for multimedia search. In fact, once we have the ability to search for multimedia information, many scenarios could be thought of.

As a start, we will look at two media types, images and documents. We will study search for images, because it teaches us important lessons about content analysis of media objects and what we may consider as being similar. Perhaps surprisingly, we will study text documents because, due to our familiarity with this media type, text documents allow us to determine what we may understand by effective search.

...

Amsterdam Drugport


Amsterdam is an international centre of traffic and trade. It is renowned for its culture and liberal attitude, and attracts tourists from various ages, including young tourists that are attracted by the availability of soft drugs. Soft drugs may be obtained at so-called coffeeshops, and the possession of limited amounts of soft drugs is being tolerated by the authories.

The European Community, however, has expressed their concern that Amsterdam is the centre of an international criminal drug operation. Combining national and international police units, a team is formed to start an exhaustive investigation, under the code name Amsterdam Drugport.

information

  • video surveillance -- monitoring
  • telephone wiretaps -- audio recording
  • photography -- archive
  • documents -- investigations
  • transactions -- structured data
  • geographic information -- locations, routes

media types


  • images -- photos
  • video -- surveillance
  • audio -- interviews, phone tracks
  • documents -- forensic, reports
  • handwriting -- notes
  • structured data -- transactions

retrieval


  • image query -- all images with this person
  • audio query -- identity of speaker
  • text query -- all transactions with BANK Inc.
  • video query -- all segments with victim
  • complex queries -- convicted murderers with BANK transactions
  • heterogeneous queries -- photograph + murderer + transaction
  • complex heterogeneous queries -- in contact with + murderer + transaction

...

information retrieval


Information retrieval, according to  [Baeza-Yates and Ribeiro-Neto (1999)], deals with the representation, storage, organisation of, and access to information items.

To see what is involved, imagine that we have a (user) query like:

find me the pages containg information on ...

information retrieval models


  • boolean or set-theoretic models
  • vector or algebraic models
  • probabilistic models

vector models


  • attribute term weighting scheme improves performance
  • partial matching strategy allows retrieval of approximate material
  • metric distance allows for sorting according to degree of similarity

image query


  • obtaining descriptive information
  • establishing similarity

content-based description


  • objects in image
  • shape descriptor -- shape/region of object
  • property description -- cells in image

shape


  • bounding box -- (XLB,XUB,YLB,YUB)

property


  • property -- name=value

example



  shape descriptor: XLB=10; XUB=60; YLB=3; YUB=50   (rectangle)  
  property descriptor: pixel(14,7): R=5; G=1; B=3

definitions


  • image grid: (m * n) cells of equal size
  • cell property: (Name, Value, Method)

example



  property: (bwcolor,{b,w},bwalgo)

...

similarity-based retrieval


How do we determine whether the content of a segment (of a segmented image) is similar to another image (or set of images)?

solutions

  • metric approach -- distance between two image objects
  • transformation approach -- relative to specification

metric approach


distance d:X->[0,1] is distance measure if: 


           d(x,y) = d(y,x)
  	 d(x,y) <= d(x,z) + d(z,y)
  	 d(x,x) = 0

pixel properties


  • objects with pixel properties p_1,...,p_n
  • pixels: (x,y,v1,...,v_n)
  • object contains w x h (n+2)-tuples

complexity


a set of points in k-dimensional space for k = n + 2

feature extraction


  • maps object into s-dimensional space

...

transformation approach


Given two objects o1 and o2, the level of dissimilarity is proportional to the (minimum) cost of transforming object o1 into object o2 or vice versa

transformation operators



    to_1,...,to_r  -- translation, rotation, scaling

cost


  • cost(TS) = %S_{i=1}^{r} cost(to_{i})

distance


  • d(o,o') = min { cost(TS) | TS in TSeq(o,o') }

advantages


  • user-defined similarity -- choice of transformation operators
  • user-defined cost-function

operations



   rotate(image-id,dir,angle)
   segment(image-id, predicate)
   edit(image-id, edit-op)

...

image repository


  • storage -- unsegmented images
  • description -- limited set of features
  • index -- feature-based index
  • retrieval -- distance between feature vectors

mission


Our goal is to study aspects of the deployment and architecture of virtual environments as an interface to (intelligent) multimedia information systems ...

...

query


  • document database + string matching

problems


  • synonymy -- topic T does not occur literally in document D
  • polysemy -- some words may have many meanings

...

effective search


  • precision -- how many answers are correct
  • recall -- how many of the right documents are returned

precision and recall



  precision = ( returned and relevant ) / returned  
  recall = ( returned and relevant ) / relevant

anomalies


  • return all documents: perfect recall, low precision
  • return 'nothing': 'perfect' precision, low recall

example


term/documentd0d1d2
snacks100
drinks103
rock-roll011

complextity


compare term frequencies per document -- O(M*N)

reduction


  • stop list -- irrelevant words
  • word stems -- reduce different words to relevant part

...

...

user-oriented measures


  • coverage ratio -- fraction of known documents
  • novelty ratio -- fraction of new (relevant) documents
  • relative recall -- fraction of expected documents
  • recall effort -- fraction of examined documents

...
(a) context(b) self-reflection

Aesthetics


  • intentions -- motives of the artist
  • expression -- where form takes over
  • representation -- the relation of art to reality

...

5. information retrieval

concepts


technology


projects & further reading

As a project, you may implement simple image analysis algorithms that, for example, extract a color histogram, or detect the presence of a horizon-like edge.

You may further explore scenarios for information retrieval in the cultural heritage domain. and compare this with other applications of multimedia information retrieval, for example monitoring in hospitals.

For further reading I suggest to make yourself familiar with common techniques in information retrieval as described in  [Baeza-Yates and Ribeiro-Neto (1999)], and perhaps devote some time to studying image analisis,  [Gonzales and Wintz (1987)].

the artwork

  1. artworks -- ..., Miro, Dali, photographed from Kunstsammlung Nordrhein-Westfalen, see artwork 2.
  2. left Miro from  [Kunst], right: Karel Appel
  3. match of the day (1) -- Geert Mul
  4. match of the day (2) -- Geert Mul
  5. match of the day (3) -- Geert Mul
  6. mario ware -- taken from gammo/veronica.
  7. baten kaitos -- eternal ways and the lost ocean, taken from gammo/veronica.
  8. idem.
  9. PANORAMA -- screenshots from field test.
  10. signs -- people,  [ van Rooijen (2003)], p. 252, 253.

6

content annotation

video annotation requires a logical approach to story telling

content annotation

learning objectives

After reading this chapter you should be able to explain the difference between content and meta information, to mention relevant content parameters for audio, to characterize the requirements for video libraries, to define an annotation logic for video, and to discuss feature extraction in samples of musical material.

Current technology does not allow us to extract information automatically from arbitrary media objects. In these cases, at least for the time being, we need to assist search by annotating content with what is commonly referred to as meta-information.

In this chapter, we will look at two more media types, in particular audio and video. Studying audio, we will learn how we may combine feature extraction and meta-information to define a data model that allows for search. Studying video, on the other hand, will indicate the complexity of devising a knowledge representation scheme that captures the content of video fragments.

Concluding this chapter, we will discuss an architecture for feature extraction for arbitrary media objects.

...

audio databases


  • audio signals -- compression, discrete representation
  • musical patterns -- similarity-based retrieval

audio data model


  • meta-data -- describing content
  • features -- using feature extraction

example



   singers -- (Opera,Role,Person)
   score -- ...
   transcript -- ...

signal-based content


  • audio data -- %F(x) over time x
  • wave -- period T, frequency f = 1/T
  • velocity -- v = w/T = w * f , with w wavelength
  • amplitude -- a

windowing


  • break signal up in small windows of time

feature extraction


  • intensity -- watts/m^2
  • loudness -- in decibels
  • pitch -- from frequency and amplitude
  • brightness -- amount of distortion

...

...

...

video annotation


  • what are the interesting aspects?
  • how do we represent this information?

video content



  video v, frame f 
  f has associated objects and activities 
  objects and activities have properties

property


  property: name = value

object schema


   (fd,fi) -- frame-dependent and frame-independent properties

object instance: (oid,os,ip)

example


frameobjectsframe-dependent properties
1Janehas(briefcase), at(path)
-housedoor(closed)
-briefcase
2Janehas(briefcase), at(door)
-Dennisat(door)
-housedoor(open)
-briefcase

frame-independent properties


objectframe-independent propertiesvalue
Janeage35
height170cm
houseaddress...
colorbrown
briefcasecolorblack
size40 x 31

activity

  • activity name -- id
  • statements -- role = v

example


   { giver : Person, receiver : Person, item : Object } 
   giver = Jane, receiver = Dennis, object = briefcase

...

video libraries



  which videos are in the library 
  what constitutes the content of each video
  what is the location of a particular video

query language for video libraries


  • segment retrievals -- exchange of briefcase
  • object retrievals -- all people in v:[s,e]
  • activity retrieval -- all activities in v:[s,e]
  • property-based -- find all videos with object oid

VideoSQL



  SELECT -- v:[s,e] 
  FROM -- video:<source><V> 
  WHERE -- term IN funcall

example



  SELECT  vid:[s,e]
  FROM video:VidLib
  WHERE (vid,s,e) IN VideoWithObject(Dennis) AND
  	object IN ObjectsInVideo(vid,s,e) AND
  	object != Dennis AND
  	typeof(object) = Person

...

...

To improve library access, the Informedia Digital Video Library uses automatic processing to derive descriptors for video. A new extension to the video processing extracts geographic references from these descriptors.

The operational library interface shows the geographic entities addressed in a story, highlighting the regions discussed in the video through a map display synchronized with the video display.

The map can also serve as a query mechanism, allowing users to search the terabyte library for stories taking place in a selected area of interest.

questions


  • what -- content-related
  • when -- position on time-continuum
  • where -- geographic location

More recently, it has been recognized that the process of spatialization -- where a spatial map-like structure is applied to data where no inherent or obvious one does exist -- can provide an interpretable structure to other types of data.

atlas of cyberspace


We present a wide range of spatializations that have employed a variety of graphical techniques and visual metaphors so as to provide striking and powerful images that extend from two dimension 'maps' to three-dimensional immersive landscapes.

...

...

feature grammar



  
  detector song; ## to get the filename
  detector lyrics; ## extracts lyrics
  detector melody; ## extracts melody
  detector check;  ## to walk the tree
  
  atom str name;
  atom str text;
  atom str note;  
  
  midi: song;
  
  song: file lyrics melody check;
  
  file: name;
  
  lyrics: text*;
  melody: note*;
  
  


  event('twinkle',2,time=384, note_on:[chan=2,pitch=72,vol=111]).
  event('twinkle',2,time=768, note_off:[chan=2,pitch=72,vol=100]).

melody detector



  int melodyDetector(tree *pt, list *tks ){
  char buf[1024]; char* _result;
  void* q = _query;
  int idq = 0; 
  
    idq = query_eval(q,"X:melody(X)");
    while ((_result = query_result(q,idq)) ) {
           putAtom(tks,"note",_result);
           }
    return SUCCESS;
  }

...

...

prediction techniques


  • social-based -- dependent on (group) rating of item(s)
  • information-based -- dependent on features of item(s)
  • hybrid methods -- combining predictors

definition(s)


  • rating -- a value representing a user's interest
  • recommendation -- item(s) that might be of interest to the user
  • regret -- a function to measure the accuracy of recommendations

guided tour(s)


  • automated (viewpoint) navigation in virtual space,
  • an animation explaining, for example, the construction of an artwork, or
  • the (narrative) presentation of a sequence of concept nodes.

...

6. content annotation

concepts


technology


projects & further reading

As a project, think of implementing musical similarity matching, or developing an application retrieving video fragments using a simple annotation logic.

You may further explore the construction of media repositories, and finding a balance between automatic indexing, content search and meta information.

For further reading I advice you to google recent research on video analysis, and the online material on search engines.

the artwork

  1. works from  [Weishar (1998)]
  2. faces -- from www.alterfin.org, an interesting site with many surprising interactive toys in flash, javascript and html.
  3. mouth -- Annika Karlson Rixon, entitled A slight Acquaintance, taken from a theme article about the body in art and science, the Volkskrant, 24/03/05.
  4. story -- page from the comic book version of City of Glass,  [Auster (2004)], drawn in an almost tradional style.
  5. story -- frame from  [Auster (2004)].
  6. story -- frame from  [Auster (2004)].
  7. story -- frame from  [Auster (2004)].
  8. white on white -- typographical joke.
  9. modern art -- city of light (1968-69), Mario Merz, taken from  [Hummelen and Sill\'e (1999)].
  10. modern art -- Marocco (1972), Krijn Griezen, taken from  [Hummelen and Sill\'e (1999)].
  11. modern art -- Indestructable Object (1958), Man Ray, Blue, Green, Red I (1964-65), Ellsworth Kelly, Great American Nude (1960), T. Wesselman, taken from  [Hummelen and Sill\'e (1999)].
  12. signs -- sports,  [ van Rooijen (2003)], p. 272, 273.

7

information system architecture

effective retrieval requires visual interfaces

information system architecture

learning objectives

After reading this chapter you should be able to dicuss the considerations that play a role in developing a multimedia information system, characterize an abstract multimedia data format, give examples of multimedia content queries, define the notion of virtual resources, and discuss the requirements for networked virtual environments.

From a system development perspective, a multimedia information system may be considered as a multimedia database, providing storage and retrieval facilities for media objects. Yet, rather than a solution this presents us with a problem, since there are many options to provide such storage facilities and equally many to support retrieval.

In this chapter, we will study the architectural issues involved in developing multimedia information systems, and we will introduce the notion of media abstraction to provide for a uniform approach to arbitrary media objects.

Finally, we will discuss the additional problems that networked multimedia confront us with.

...

issues


  • multimedia storage and retrieval -- homegrown, third-party and legacy sources
  • information architecture -- common format, native format, hybrid
  • media abstraction -- unified indexes, query relaxation

content organisation


  • autonomy -- index per media type
  • uniformity -- unified index
  • hybrid -- media indexes + unified index

Principle of Uniformity


... from a semantical point of view the content of a multimedia source is independent of the source itself, so we may use statements as meta data to provide a description of media objects.

  • from a semantical point of view the content of a multimedia source is independent of the source itself.
  • use statements as meta data
  • md(o) -- metadata associated with media object o

tradeoffs


  • metadata can be stored using standard relational and OO structures
  • manipulating metadata is easy
  • feature extraction is (!) straightforward

    is it?


software architecture


  • a database of media object, supporting
  • operations on media objects, and offering
  • logical views on media objects

information retrieval cycle


  1. specification of the user's information need
  2. translation into query operations
  3. search and retrieval of media objects
  4. ranking according to likelihood or relevance
  5. presentation of results and user feedback
  6. resulting in a possibly modified query

  • despite high interactivity, access is difficult;
  • quick response is and will remain important!

...

media abstraction


  • state -- smallest chunk of media data
  • feature -- any object in a state
  • attributes -- characteristics of objects
  • feature extraction map -- to identify content
  • relations -- to capture state-dependent information
  • (inter)relations between 'states' or chunks

example -- image database



  states: { pic1.gif,...,picn.gif } 
  features: names of people 
  extraction: find people in pictures 
  relations: left-of, ...

example -- video database



  states:  set of frames 
  features:  persons and objects
  extraction:  gives features per frame 
  relations:  frame-dependent and frame-independent information
  inter-state relation:  specifies sequences of frames

simple multimedia database


  • a finite set M of media abstractions

structured multimedia database


  • equivalence relations --to deal with synonymy
  • partial ordering -- to deal with inheritance
  • query relaxation -- to please the user

...

SMDS -- functions



  Type: object  |->  type 
  ObjectWithFeatures:  f |-> { o |  object o contains  f }  
  ObjectWithFeaturesAndAttributes:  (f,a,v) |-> { o |  o contains f with  a=v }  
  FeaturesInObject:  o |-> { f | o  contains  f }  
  FeaturesAndAttributesInObject:  o |-> { (f,a,v) | o  contains  f  with  a=v }

SMDS-SQL



SELECT -- media entities
  • m -- if m is not a continuous media object
  • m:[i,j] -- m is continuous, i,j integers (segments)
  • m.a -- m is media entity, a is attribute

FROM

  • <media><source><M>

WHERE

  • term IN funcall

example



    SELECT M
    FROM   smds source1 M
    WHERE  Type(M) = Image AND
  	 M IN ObjectWithFeature("Dennis") AND
  	 M IN ObjectWithFeature("Jane") AND
  	 left("Jane","Dennis",M)

hybrid representations: HM-SQL


  • express queries in specialized language
  • perform operations (joins) between SMDS and non-SMDS data

differences


  • function calls are annotated with media source
  • queries to non-SMDS data may be embedded

example HM-SQL



   SELECT M
   FROM smds video1, videodb video2
   WHERE M IN smds:ObjectWithFeature("Dennis") AND
         M IN videodb:VideoWithObject("Dennis")

digital libraries


Digital libraries are constructed -- collected and organized -- by a community of users. Their functional capabilities support the information needs and users of this community. Digital libraries are an extension, enhancement and integration of a variety of information institutions as physicalplaces where resources are selected, collected, organized, preserved and accessed in support of a user community.

... federated structures that provide humans both intellectual and physical access to the huge and growing worldwide networks of information encoded in multimedia digital formats.

digital libraries (5S)


  • streams: (content) -- from text to multimedia content
  • structures: (data) -- from database to hypertext networks
  • spaces: (information) -- from vector space to virtual reality
  • scenarios: (procedures) -- from service to stories
  • societies: (stakeholders) -- from authors to libraries


   D-Lib Forum -- www.dlib.org
   Informedia -- www.informedia.cs.cmu.edu

...

networked multimedia


  • real-time transmission of continuous media information (audio, video)
  • substantial volumes of data (despite compression)
  • distribution-oriented -- e.g. audio/video broadcast

network criteria


  • throughput -- bitrates, burstiness
  • transmission delay -- including signal propagation time
  • delay variation -- jitter
  • error rate -- data alteration, loss

  • multicasting and broadcasting capabilities
  • document caching

Quality of Service


Quality of Service is a concept based on the statement that not all applications need the same performance from the network over which they run. Thus, applications may indicate their specific requirements to the network, before they actually start transmitting information data.

QoS requirements


  • hard requirements
  • guidance for optimizing internal resources
  • criteria for acceptance

...

virtual objects

  • VO = { (O_i,Q_i,C_i) | 1 <= i <= k }

where

  • C_1,...,C_k -- mutually exclusive conditions
  • Q_1,...,Q_k -- queries
  • O_1,...,O_k -- objects

...

networked virtual environments


  • shared sense of space -- room, building, terrain
  • shared sense of presence -- avatar (body and motion)
  • shared sense of time -- real-time interaction and behavior

  • a way to communicate -- by gesture, voice or text
  • a way to share ... -- interaction through objects

challenges


  • network bandwidth -- limited resource
  • heterogeneity -- multiple platforms
  • distributed interaction -- network delays
  • resource management -- real-time interaction and shared objects
  • failure management -- stop, ..., degradation
  • scalability -- wrt. number of participants

manage dynamic shared state

...

...

  • the Java Media Framework, and
  • the DLP+X3D platform

java Media Framework


The JavaTM Media APIs meet the increasing demand for multimedia in the enterprise by providing a unified, non-proprietary, platform-neutral solution. This set of APIs supports the integration of audio and video clips, animated presentations, 2D fonts, graphics, and images, as well as speech input/output and 3D models. By providing standard players and integrating these supporting technologies, the Java Media APIs enable developers to produce and distribute compelling, media-rich content.

recommender economy


  • cross sale -- users who bought A also bought B
  • up sale -- if you buy A and B together ...

recommender model



  U = user
  I = item
  B = behavior
  R = recommendation
  F = feature
  • observations -- U \* I \* B
  • recommendations -- U \* I 

  B = [ time = 20sec, rating = r ]
  F = [ artist = rembrandt, topic = portrait ]
  R = [ artist(rembrandt) = r, topic(portrait) = r ]
  

  A = [  p_{1}, p_2 , ... ]
  where p_{k} = [ f_1 = v_1, f_2 = v_2, ... ]
with as an example 

   A_{nightwatch} = [ artist=rembrandt, topic=group ]
   A_{guernica} = [ artist=picasso, topic=group ]

...
users, artworks and properties

distance metric



       d(x,y) = d(y,x)
       d(x,y) <= d(x,z) + d(z,y)
       d(x,x) = 0

dimension(s)


  • positive vs negative
  • individual vs community/collaborative
  • feature-based vs item-based

interpretation(s)


  • neutral interpretation -- use d(s_{n}, a_{k}) < d(s_{n}, s_{n+1} )
  • positive interpretation -- increase w(feature(a_{k}))
  • negative interpretation -- decrease w(feature(s_{n+1}))

...

7. information system architecture

concepts


technology


projects & further reading

As a project, you may implement a multi-player game in which you may exchange pictures and videos, for example pictures and videos of celebrities.

Further you may explore the development of a data format for text, images and video with appropriate presentation parameters, including postioning on the screen and intermediate transitions.

For further reading you may study information system architecture patterns, nd explore the technical issues of constructing server based advanced multimedia applications in  [Li and Drew (2004)].

the artwork

  1. examples of dutch design, from  [Betsky (2004)].
  2. idem.
  3. screenshots -- from splinter cell: chaos theory, taken from Veronica/Gammo, a television program about games.
  4. screenshots -- respectively Sekken 5, Sims 2, and Super Monkey Ball, taken from insidegamer.nl.
  5. screenshots -- from Unreal Tournament, see section 7.3.
  6. idem.
  7. idem.
  8. resonance -- exhibition and performances, Montevideo, april 2005.
  9. CHIP -- property diagram connecting users.
  10. signs -- sports,  [ van Rooijen (2003)], p. 274, 275.

a journey of a thousand miles begins with the first step

chinese/japanese proverb

applications & tools


...

perspectives -- multimedia applications


  • technical -- algorithmic effects
  • sociological -- stakeholders and teamwork
  • tool selection -- Maya vd 3DSMAX
  • political -- negotiating support
  • scientific -- experience design
  • computer science -- tools and technologies
  • artistic -- portfolio as a design product

...

the artwork

  1. walking figure -- sculpture by Alberto Giacometti,  [Hohl (1971)].
  2. signs -- meteorological symbols,  [ van Rooijen (2003)], p. 214, 215.
  3. photographs -- Jaap Stahlie, commissioned work, using traditional non-digital techniques.

8

virtual environments

augmented virtuality acts as an intelligent looking glass

virtual environments

learning objectives

After reading this chapter you should be able to characterize the notion of virtual context, discuss the issue of information retrieval in virtual environments, explain what is meant about intelligent multimedia and discuss the potential role of intelligent agents in multimedia applications.

From a user perspective, virtual environments offer the most advanced interface to multimedia information systems. Virtual environments involve the use of (high resolution) 3D graphics, intuitive interaction facilities and possibly support for multiple users.

In this chapter, we will explore the use of (desktop) virtual environments as an interface to (multimedia) information systems. We will discuss a number of prototype implementations illustrating, respectively, how paintings can be related to their context, how navigation may be seen as a suitable answer to a query, and how we can define intelligent agents that can interact with the information space. Take good notice, the use of virtual environments as an interface to information systems represents a major challenge for future research!

...

Dam Square, Amsterdam


  • it is a 'real world' environment
  • it has 700 years of (recorded) history
  • it has a fair amount of historical buildings
  • buildings and street life change over time

how can we give access to the 'Dam square' information space

...

virtual context


  • VR model of Dam square
  • selection of related paintings fromRoyalMuseum
  • viewpoint adjustment, to match painting
  • (transparent) overlay of paintings over buildings

augmented virtual reality


  • give user sense of geographic placement of buildings
  • show how multiple objects in a museum relate to eachother
  • show what paintings convey about their subject, and how

problems


  • organised guided tours
  • account for buildings that no longer exist
  • change temporal context
  • allow user queries as input

VRML


  • declarative means for defining geometry and appearance
  • prototype abstraction mechanism
  • powerful event model
  • relatively strong programmatic capabilities

...

virtual archeology


  • variety of archeological sites
  • various paths through individual site
  • reconstruction of 'lost' elements
  • 'discovery' of new material
  • glossary -- general background knowledge

RIF


Retrieval of Information in Virtual Worlds using Feature Detectors

...
On the left is the 2D map of the third floor of CWI, on the right the model generated from it.

the map CWI

the model

...

the query
C O F F E E

navigation
explore discover

explore and discover

...

what are we searching for?

what are we searching for?

types of information

  • viewpoints
  • areas of interest
  • objects
  • persons
  • text

availability

  • static -- always
  • shared -- users
  • dynamic -- runtime
  • temporal -- events
  • hidden -- actions

scanning the scenegraph

  • annotations
  • node types
  • textual content
  • materials
  • textures
  • geometry

choose a metaphor

choose a metaphor

get a viewpoint

  • viewpoints
  • areas of interest
  • objects and persons

answer the query

  • route planning
  • viewpoint transformation

walking

walking

assumptions

  • explicit annotation
  • map for route planning
  • keyword matching

requirements

  • database -- annotations & map
  • 3D (pseudo-immersive) interface

the system

relaxing the assumptions

  • annotation -- incremental and/or automatic
  • (explicit) maps -- based on user navigation
  • (keyword) matching -- text retrieval

...

  • navigation by query is feasible and may help users to find locations and objects
  • determining suitable navigation routes without an explicitly defined map is hard

future work

  • shift in focus -- intelligent agents
  • DLP + VRML -- distributed logic programming

Web Agent Support Program


www.cs.vu.nl/~eliens/research

DEMO

no database, no walking

3D GUI


Wishful thinking about the widespread adoption of three-dimensional interfaces has not helped spawn winning applications. Success stories with three-dimensional games do not translate into broad acceptance of head-tracking immersive virtual reality. To accelerate adoption of advanced interfaces, designers must understand their appeal and performance benefits as well as honestly identify their deficits. We need to separate out the features that make 3D useful and understand how they help overcome the challenges of dis-orientation during navigation and distraction from occlusion.

Ben Shneiderman

Does spatial memory improve with 3D layouts? Is it true that 3D is more natural and easier to learn? Careful empirical studies clarify why modest aspects of 3D, such as shading for buttons and overlapping of windows are helpful, but 3D bar charts and directory structures are not. 3D sometimes pays off for medical imagery, chemical molecules, and architecture, but has yet to prove beneficial for performance measures in shopping or operating systems.

Ben Shneiderman

...

WASP


Web Agent Support Program

DLP


Distributed Logic Programming

RIF + WASP


  • distributed logic programming -- uniform platform
  • agent technology -- subsumes multi-user server

...

multi-user soccer game


  • multiple (human) users -- may join during the game
  • multiple agents -- to participate in the game (e.g. as goalkeeper)
  • reactivity -- players (users and agents) have to react quickly
  • cooperation/competition -- requires 'intelligent' communication
  • dynamic behavior -- sufficiently complex 3D scenes, including the dynamic behavior of the ball

control points


  • get/set -- position, rotation, viewpoint

agents in virtual environments


  • virtual environments with embedded autonomous agents
  • virtual environments supported by ACL communication

Living Worlds


  • scene -- geometrically bounded, continuously navigable
  • world -- collection of (linked) scenes

Shared Object


  • pilot -- instance that will be replicated
  • drone -- instance that replicates pilot

  • pilot agents -- control state of a shared object
  • drone agents -- replicate the state of a shared object

  • object agents -- controls a single shared object (like the soccerball) pilot at server, drone at client
  • controls users' avatar pilot at user side, drone at server or clients
  • autonomous agents -- like football player, with own avatar pilot at server, drone at clients

programming platform


  • VRML EAI support
  • distributed communication capabilities (TCP/IP)
  • multiple threads of control -- for multiple shared objects
  • declarative language -- for agent support

...

taxonomy of agents


  • 2D/3D -- to distinguish between text-based and avatar embodied agents
  • client/server -- to indicate where agents reside
  • single/multi -- as a measure of complexity

PAMELA


Personal Assistent for Multimedia Electronic Archives

  • autonomous and on-demand search capabilities
  • (user and system) modifiablepreferences
  • multimedia presentation facilities

H-Anim


  • control points -- joints, limbs and facial features

presentation agent


Given any collection of results, PAMELA could design some spatial layout and select suitableobject types, including for example color-based relevance cues, to present the results in a scene. PAMELA could then navigate you through the scene, indicating the possible relevance of particular results.

persuasion games


  • single avatar persuasive argumentation
  • multiple avatar dialog games

PAMELA


Persuasive Agent with Multimedia Enlightened Arguments

...

A variety of applications may benefit from deploying embodied conversational agents, either in the form of animated humanoid avatars or, more simply, as a 'talking head'. An interesting example is provided by Signing Avatar, a system that allows for translating arbitrary text in both spoken language and sign language for the deaf, presented by animated humanoid avatars.

Here the use of animated avatars is essential to communicate with a particular group of users, using the sign language for the deaf.

STEP


  • convenience -- for non-professional authors
  • compositional semantics -- combining operations
  • re-definability -- for high-level specification of actions
  • parametrization -- for the adaptation of actions
  • interaction -- with a (virtual) environment

DLP+X3D


The DLP+X3D platform provides together with the STEP scripting language the computational facilities for defining semantically meaningful behaviors and allows for a rich presentational environment, in particular 3D virtual environments that may include streaming video, text and speech.

evaluation criteria

  • effective communication

conversational agents in VR


  • presentational VR, instructional VR, educational VR

applications


  • information agents, presentation agents

system perspective


  • range of agent categories, open standards

user perspective


  • naturalness -- contextual & emotional

initial target(s)


  • build initial (throwaway) prototype
  • explore content creation technology
  • create tutorial(s) for content contribution
  • analyse technological requirements

...
(a) outside view(b) inside view

The history of Second Life is extensively descibed in the official Second Life guide,  [Rymaszweski et al. (2007)]. Beginning 2004, almost out of the blue, Second Life appeared with a high adoption and low churn rate, now counting, March 2007, over 4 million inhabitants. Considering the cost of ownership of land, which easily amounts to 200 euro per month rent after an initial investment of 1500 euro for a single piece of land measuring 65,536 square meters, the adoption of Second Life by individuals as well as companies such as ABN-AMRO, Philips and institutions such as Harvard is surprising.

What is the secret of the success of Second Life?, we asked in  [Eliens et al. (2007)], and we immediately confessed: We don't know! But in comparison to other platforms for immersive worlds, including MMORPGs such as World of Warcraft and Everquest, Second Life seems to offer an optimal combination of avatar modification options, gesture animations, in-game construction tools, and facilities for communication and social networking, such as chatting and instant messaging. Incorporating elements of community formation, commonly denoted as Web 2.0, and exemplified in MySpace, YouTube and Flickr, the immersive appearance, perhaps also the built-in physics and the inclusion of elementary economic principles, seem to be the prime distinguishing factors responsible for the success of Second Life. In addition, the possibility of recording collaborative enacted stories,  [Stories], using built-in machinima certainly also contributes to its appeal.

What has been characterized as a shift of culture, from a media consumer culture to a participatory culture,  [Jenkins (2006)], where users also actively contribute content, (was) for our institution one of the decisive reasons to create a presence in Second Life, to build a virtual platform that may embody our so-called community of learners, where both staff and students cooperate in contributing content, content related to our sciences, that is.

The first idea that comes to mind, naturally, is to use Second Life to offer courses online. But, although we did have plans to give lectures (college) on law, probably including the enactment of a particular case, we did consider this approach as rather naive, and frankly I see no reason to include what may be considered an outdated paradigm of learning in our virtual campus, where there might be more appealing alternatives. Similarly, using the virtual laboratory for experiments might not be the best way to offer courses, although, again, we do intend to provide a model of a living cell, allowing students to study the structure, functionality and behavior of organic cells in virtual space.

active learning


  • experience the world in new ways
  • form new affiliations
  • preparate for future learning

...

8. virtual environments

concepts


technology


projects & further reading

As a project, I suggest the implementation of storytelling in virtual environments, with (possibly) an embodied agent as the narrator.

You may further explore or evaluate the role of agents in multimedia applications and virtual environments.

For further reading in (real) VR, I advice  [Sherman and Craig (2003)], and for gaining an understanding in story telling and applications you may try to get hold of the proceedins, of TIDSE 2003, and TIDSE 2004.

the artwork

  1. another series of dutch light.
  2. virtual context -- Dam Square, Amsterdam, see 8.1.
  3. VU Campus in VRML -- student project.
  4. CWI 3th floor, floormap and model, see 8.2..
  5. query -- on 3th floor of CWI.
  6. navigation -- on 3th floor of CWI.
  7. soccer game -- image from WASP project, see section 8.3.
  8. digital beauties -- taken from  [Wiedermann (2002)].
  9. digital beauties -- taken from  [Wiedermann (2002)].
  10. VU @ Second Life -- screenshots.
  11. signs -- sports,  [ van Rooijen (2003)], p. 276, 277.

9

digital content creation

post-modern design allows for sampling

digital convergence

learning objectives

After reading this chapter you should be able to mention some basic rules of digital content creation, discuss what criteria your portfolio should meet, describe how you would approach the design of a logo, explain the notion of user-centered design, and characterize the issues that play a role in dveloping multimedia for theatre.

Whether your ambition is to become a professional designer or not, also for students of information science and computer science, a course in visual design is a must, I think.

In this chapter, we will treat various aspects of digital content creation. The first section discusses how to approach visual design and gives a number of basic design assignments, that can be used to get experience with visual design. Section 2 discusses the issue of workflow and tools, and investigates how design fits in with the process of developing multimedia applications. In the final section, I will elaborate on a theatre project I was involved in, for which I had to develop an augmented reality application.

...

...

The overall goal of the visual design course is to establish some basic aesthetic awareness, by providing suitable exercises and assignments. In addition, the student is supposed to become familiar with the craft of design, which necessarily, but not exclusively, involves the use of tools and techniques.

track(s) -- perspective


  • styling -- concept and presentation
  • digital content -- material, animation
  • tech track -- special effects

www.178aardigeontwerpers.nl


  • e-motionist -- make emotion, rhythm and movement flow together
  • chaoticus -- who sees chaos within order
  • formologist -- who approaches the fabrication of forms as an art
  • infonaut -- who moves in the twilight zone of information and meaning
  • transformator -- who transforms images and concepts into new matter

...

products


  • web site -- e.g. conference, campaign (browse)
  • 2D/3D animation -- promotion/ad (temporal sequence)
  • virtual space -- game/infotainment (navigate)
  • ebook -- story (sequential experience)

portfolio -- design as a product


  • concept(s)
  • sketches & explorations
  • finalized products
  • evaluation & reflection

www.jaapstahlie.com



  In my perception a portfolio is about the
  past and I feel much more related to the
  present especially in my work as a
  photographer. To me the relation with
  the present and the subject/assignment directs
  my creativity, the experiences over the 
  past draw my skills. My challenge is to be
  truly inspirred, to be present in the present.

There are basic exercises, obligatory for all students, and a final assignment, where you have a choice between three productions, each with a different supervisor. In addition, as explained in the guidelines, all students must write an essay, and give a presentation in class. For deadlines, see the schedule. There will be periodic checks on the status of your work. Each year there will be recommended themes.

basic exercises


  1. develop a logo
  2. create a sign
  3. design a collage
  4. write a story

...

rules


  • be present -- 2 omissions max.
  • be in time -- hard deadlines
  • be online -- have your portfolio available
  • be creative -- don't steal without a reason/mentioning
  • be smart -- there is no 2nd chance

...

multimedia's promise is terribly generalized,

it simply lets you do anything.

shovelware -- multimediocrity


... far from making a killing, it looked as if the big boys ... had killed the industry by glutting the market with inferior products.

if multimedia is comparable to print then yes, we'd be crazy to expect it to mature in a mere ten years.

"Learning how to not fool ourselves is, I'm sorry to say, something that we haven't specifically included in any particular course that I know of. We just hope you've caught it by osmosis."

Richard Feynman


the media equation


We regularly exploit the media equation for enjoyment by the willing suspension of our critical faculties. Theatre is the projection of a story through the window of a stage, and typically the audience gets immersed in the story as if it was real.

engineering


"engineering is the art of moulding materials we do not wholly understand ... in such a way that the community at large has no reason to suspect the extent of our ignorance."

A. R. Dykes.


The best thing is to empower yourself. But before you can do that, you need to understand what you are doing -- which is a surprisingly novel thing to do.

there is no theory of creativity

steps

browse, explore; chew it over; incubation, let it rest; illumination (YES); verification,does it work?

general rules

  • if you aim to please everybody, you will please nobody
  • constraints come with the territory, you must learn to love them
  • emotional charge is the key to success

make your virtual hands dirty.

...

postmodern design


... postmodern design is of a highly reflective nature ... appropriating design of the past ... in other words, sampling is allowed but no plagianarism

game design


  • style -- develop concept, plot and visual assets for a game of choice
  • content -- develop environments, models and animations for a game of choice
  • effects -- develop models, textures and special effects (shaders) for a game of choice

did you ever wonder why cheap wine tastes better in fancy glasses?

seduction


  • physio-pleasure -- of the body
  • socio-pleasure -- by interaction with others
  • psycho-pleasure -- due to use of the product
  • ideo-pleasure -- reflecting on the experience

genre(s)


Abstract, Adaptation, Adventure, Artificial Life, Board Games, Capturing, Card Games, Catching, Chase, Collecting, Combat, Demo, Diagnostic, Dodging, Driving, Educational, Escape, Fighting, Flying, Gambling, Interactive Movie, Management Simulation, Maze, Obstacle Course, Pencil-and-Paper Games, Pinball, Platform, Programming Games, Puzzle, Quiz, Racing, Role-Playing, Rhythm and Dance, Shoot Em Up, Simulation, Sports, Strategy, Table-Top Games, Target, Text Adventure, Training Simulation, and Utility.

levels of design


  • visceral -- what appeals to our intuition (affordance)
  • behavioral -- is all about use (performance)
  • reflective -- its all about message, culture and meaning

...

ICT Games Project


The goal of the ICT games project is to develop immersive, interactive, real time training simulations to help the Army create a new generation of decision-making and leadership-development tools.

  • Mission Rehearsal Exercise -- to solve a potential conflict after a car accident
  • Language Training Simulation -- to learn how to contact local leaders in arabic

Virtual Humans Workshop


  • Is it more appropriate to construct a frame of analysis that encompasses both user and ECA in a single interaction graph?
  • Is it fitting to think in terms of a fixed graph that the user comes to recognize, or is the graph itself a dynamic structure?
  • Is it even appropriate to focus on "affordances to act," or is it more fitting to consider cues that influence the mental interpretations that lead to action (e.g., affordances of control, affordances of valence of potential outcomes, etc.)? How does this relate to intrinsic motivation?

...

usability (ISO DIS 9241-11)


... the effectiveness, efficiency and satisfaction with which specified users can achieve particular goals in particular environments ...

emotional involvement


  • entices by diverting attention -- unlike the common
  • delivers surprising novelty -- not identifiable to its function
  • goes beyond obvious needs and expectations -- it becomes something else
  • creates an instinctive response -- curiosity and confusion

...

...

form and content


Very often people assume that "form" as a concept is the opposite of something called "content". This assumption implies that a poem or a musical piece or a film is like a jug. An external shape, the jug, contains something that could just as easily be held in a cup or pail. Under this assumption, form becomes less important than whatever it is presumed to contain.

We do not accept this assumption. If form is the total system, which the viewer attributes to the film, there is no inside or outside. Every component functions as part of the overall pattern that is perceived. Thus we shall treat as formal elements many things that some people consider content. From our standpoint, subject matter and abstract ideas all enter into the total system of the artwork ( .... )

experimental validation


  • a theory -- in our case: PEFiC
  • a test scenario -- for example, memory tasks in a digital dossier
  • the technology -- to realize applications

validation scenario(s)


  • navigation -- pure interactivity
  • guided tours -- using some narrative structure
  • agent-mediated -- navigation and guided tours

...

Stone and Feather


  • feather: 70 cm, from ostrich, curved
  • stone: 13.5 cm, white marble
  • position: alignment with pedestal, no glue
  • environment: 50 lux of light max.

...

...

...

...

...

Computational Art


The author conducts a simple thought experiment investigating the existence and scope of 'computational art': the utilization of the computer in the visual arts and music. In the experiment he sets the task of constructing an artifact that is capable of producing works of art. Since it appears that the artifact needs at least the capability of imagination, he queries the nature of images and imagery and argues that imagination is strongly intentional. Next he introduces the concept of notational systems, since they seem to govern the artistic activity of (not exclusively) machines. Confronted with the question of whether we are able to develop a computational analogue for taste, he finds that notational systems prove to be necessary for mediating the method of production of an artwork and the appraisal of its artistic value. Furthermore, the author shows that there are certain epistemological limits to the creativity of an imaginative device. Although the outcome of this hypothetical construction task clearly denies the possibility of an autonomously creative artifact, there seems to be no reason to worry about the opportunities for computational art: the computer appears to be a unique tool in exploring the possibilities of artistic production, guided by artists.

google(s)



  <form action="http://www.google.com/search?" method="GET">
  <input name="q" value="+site:www.cs.vu.nl/~eliens/media ">
  <input type="text" name="q" size=40>
  <input type="submit" value=" "> 
  </form>

phases of awareness


  • initiation -- appeal to curiosity
  • promotion -- raising interest
  • progression -- prolonged involvement

...

9. digital content creation

concepts


technology


projects & further reading

As a project, you may develop a dialog engine for non-linear interactive story telling or a collage generator, that produces artworks from a collection of images.

You may further explore the various presentation platforms, and assess the tradeoffs with respect to the support they offer for authoring.

For further reading, I suggest to study interaction design patterns. It is also worthwhile to get some books on modern art, to gain some knowledge about art and design.

the artwork

  1. street logos -- images from  [Manco (2004)].
  2. photograph of oilpaint box.
  3. Mark Veldhuijzen van Zanten -- the six roles in their agency.
  4. Mark Veldhuijzen van Zanten -- to design for the salon, periodic lounge evenings in musea and art institutes.
  5. Geert Mul -- interactive multimedia installation.
  6. Geert Mul -- multimedia installation in dutch consulate in India.
  7. website of Institute of Creative Technologies, showing scenes from Mission rehearsal Exercise (MRE).
  8. street logos -- more images from  [Manco (2004)].
  9. website for Visual Sensations, a yearly VJ contest in the Netherlands, developed by the agency of mark Veldhuijzen van Zanten.
  10. Geert Mul -- Harbour Sound & Vision, 1999
  11. screenshots from virtual atelier of Marinus Boezem.
  12. left: don't spit, a chines poster against spitting during the SARS period, taken from dutch newspaper; right: filmteckarna,  [Wiedermann (2004)].
  13. sketches -- from filmteckarna,  [Wiedermann (2004)].
  14. sketches -- from filmteckarna,  [Wiedermann (2004)].
  15. sketches -- from filmteckarna,  [Wiedermann (2004)].
  16. game -- pizza boy, developed by Headland, see 9.3.
  17. signs -- health and safety,  [ van Rooijen (2003)], p. 258, 259

10

application development

learn the craft, break through the magic of engineering

digital convergence

learning objectives

After reading this chapter you should be able to discuss the multimedia development process, to indicate the need for information system support in the cultural heritage domain, to characterize the notion of digital dossier, to provide solutions for navigating complex information spaces, and to discuss the data representation issues involved.

As you gather from reading this book, the field of multimedia is widely divergent. However, when you develop a multimedia application, you will find that all topics treated so far will become relevant. There will be a need to mix multiple media formats. You will have to find suitable codecs for your video. You will be asked whether search is possible. And, not the least important, you will have to balance navigation and presentation.

This chapter is based on the work we, that is my students, have been doing in the domain of cultural heritage. In the first section, we will introduce the notion of digital dossier and outline our general approach. We will then in section 2 look at some examples, and describe how we deploy concept graphs as a universal navigation tool for complex information spaces. Finally, in section 3, we will explore the options for presenting multimedia material and discuss the design issues as well as the technical issues that have arisen in the course of our work.

...

multimedia casus

The assignment in the multimedia casus is to develop a virtual environment for some cultural or governemental institute or company.

The practicum takes the form a stage, in which external supervision plays an important role.

In the multimedia casus, techniques learned in previous courses (see the afterthoughts) will be applied to create the application.

At the start of the course the actual assignment will be determined.

Examples of possible assignments are: the development of a virtual exposition hall for the Dutch Royal Museum of the Arts, a virtual city square, which gives information about both the present and the past, a virtual shop, with online buying facilities, or an online broker, which offers facilities for inspecting houses.

Given an information space, create a VR that resolves the duality between information and presentation, using intelligent multimedia technology. The VR must offer access to all relevant information entities, organized in a suitable spatial layout, and must allow for presentations from a variety of perspectives, making full use of graphical and rich media facilities.

...
front page of the INCCA website

INCCA


In 1999, a group of eleven international modern art museums and related institutions applied to the European Commission (Raphael Programme) under the umbrella International Network for the Conservation of Contemporary Art (INCCA). The INCCA project was accepted and work started in January 2000 led by the organiser, the ICN (Netherlands Institute for Cultural Heritage) and the co-organiser, Tate, London.

objectives


INCCA's most important set of objectives, which are closely interlinked, focuses on the building of a website with underlying databases that will facilitate the exchange of professional knowledge and information. Furthermore, INCCA partners are involved in a collective effort to gather information directly from artists.

mission


INCCA's guiding mission is to collect, share and preserve knowledge needed for the conservation of modern and contemporary art.

checklist


  • roles -- create a team
  • project goal -- develop a vision
  • production -- construct the assets
  • quality assesment -- test and control
  • delivery -- present and archive
  • manage -- all along
  • document -- track project's history

judgement


  • group -- (2) effort, 5 (product), 3 (documentation)
  • individual -- (4) responsibility, (3) productivity, (3) quality

deliverables


  • group -- project plan, design, project report, product
  • individual -- detailed weekly account of activities

schedule

  1. project organisation
  2. project definition
  3. planning and design
  4. construction and development
  5. integration and delivery
  6. presentation and archiving

...
physical control cubehead-mounted display

application areametaphorfamiliar knowledge
operating environmentdesktopoffice tasks
spreadsheetsledger sheetcolumnar table
object-oriented environmentphysical worldreal world
hypertextnotecardsorganization of text
learning environmenttraveltours, guides, movement
file storagepilescategorizing
multimedia environmentsroomsspatial structures
cooperative workmulti-agentstravel agents, servants

interaction styles


  • command entry
  • menus and navigation
  • forms fills and spreadsheets
  • natural language dialog
  • direct manipulation

digital dossier

Create a VR that realizes a digital dossier for a work of a particular artist. A digital dossier represents the information that is available for a particular work of art, or a collection of works, of a particular artist. The digital dossier should be multimedia-enhanced, that is include photographs, audio and other multimedia material in a compelling manner.

Webster New World Dictionary


  • dossier (dos-si-er) [ Fr < dos (back); so named because labeled on the back ] a collection of documents concerning a particular person or matter
  • archive -- 1) a place where public records are kept ... 2) the records, material itself ...

...
main nodeinterviews

everything must be highly intertwinkled

...
presentation of video clips from Marina Abramovic

...
Reconstruction of Terra della Dea Madre in VRML.

style issues


  • what icons should be used to identify the elements of the concept graph?
  • what categories and relationships are most appropriate?
  • how should the information be displayed, simultaneously or more focussed?
  • how do we allow the user to choose between multiple information items?
  • how do we avoid visually disturbing elements?

...
no lighthalf lightfull light

user-centered design methods


field studies, user requirement analysis, iterative design, usability evaluation, task analysis, focus groups, formal/heuristic analysis, user interviews, prototype (without user testing), surveys, informal expert review, card sorting, participatory design

usability evaluation


  • learnability -- time and effort to reach level of performance
  • throughput -- the amount of work done
  • flexibility -- accomodating changes in the task
  • attitude -- of users to the system

task world ontology


  • task -- activity performed by an agent to reach a certain goal
  • goal -- a desired state in the task world or system
  • role -- a meaningful collection of tasks
  • object -- refers to a physical or non-physical entity
  • agent -- an entity that is considered active
  • event -- a change in the state of the task world

...

...

next generation dossier(s)


  1. adaptation of representation to Dublin Core (+ annotation needed for presentation)
  2. XML-based content management, with php forms (extending what we have now)
  3. there should also be a possibility to present the information and material in a 'plain' web format
  4. as well as in (a new version of) 3D dossiers
  5. we should think about the proper presentation parameters.

structures


  • Video -- to display video fragment, including interviews
  • Picture -- to present pictures of the artwork
  • Artwork -- contains all information connected to a work of art
  • TextItem -- to present text, from the interview or any other source
  • MaterialItem -- to present information about material(s) used
  • GroupNode -- to combine nodes in the concpet graph
  • Information -- acts as the outer container for all nodes

concept graph



  Information {
   informationNodes [
    GroupNode {
  	ID "MAIN"
  	shortName "Main"
  	longName "Main"
  	urlModel "models/conceptGraph/main/modelMain.wrl"
  	description [ "Central information node" ]
  	connectedNodesIDs [ "ARTWORKS", "KEYWORDS", 
  	                         "INTERVIEWS", "REPORT" ]
  	}
    GroupNode {
  	shortName "Artworks"
  	longName "Artworks"
  	description [ "Node that connects to all the artworks" ]
  	ID "ARTWORKS"
  	connectedNodesIDs [ "MAIN", "TRANSITORY", 
  	              "ULAY", "VIDEOINSTALLATION", "ABRAMOVIC" ]
  	urlModel "models/conceptGraph/artworks/artworksGroup.wrl"
  	}
  	## ...
   ]
  }
  


  Artwork {
    shortName "Terra degla Dea Madre"
    longName "Terra degla Dea Madre"
    description ["15:40 min, colour, sound."]
    ID "AV24"
    connectedNodesIDs ["VIDEOINSTALLATION", "DTV24", 
               "TTV24", "PV24", "CV24", "VV24", "G0"]
    urlPreviewImage "images/previewImages/AV24.jpg"
    widthPreviewImage 479
    heightPreviewImage 349
  }
  


  Video {
    ID "CV24"
    shortName "Interview clip Terra degla Dea Madre"
    longName "Interview clip showing Terra degla Dea Madre"
    url "interviewclips/interview_terra_degla.avi"
    width 320
    height 360
    urlPreviewImage "images/previewImages/interview_terra_degla.jpg"
    widthPreviewImage 320
    heightPreviewImage 240
    description [""]
    connectedNodesIDs ["CLIP", "AV24"]
  }
  


  TextItem {
    shortName "Instruction"
    longName "Green Dragon Lying instructions for the public."
    description ["Text explaining the way the public has to interact with the artwork."]
    ID "ITO05"
    connectedNodesIDs ["AO05", "INTERACTION"]
    url "text/AO05_instruction.txt"
  }

Dublin Core


  • title -- name given to the resource
  • creator -- entity primarily responsible for making the content of the resource
  • subject -- topic of the content of the resource
  • description -- an account of the content of the resource
  • publisher -- entity responsible for making the resource available
  • contributor -- entity responsible for making contributions to the content of the resource
  • date -- date of an event in the lifecycle of the resource
  • type -- nature or genre of the content of the resource
  • format -- physical or digital manifestation of the resource
  • identifier -- unambiguous reference to the resource within a given context
  • source -- reference to a resource from which the present resource is derived
  • language -- language of the intellectual content of the resource
  • relation -- reference to a related resource
  • coverage -- extent or scope of the content of the resource
  • rights -- information about rights held in and over the resource

...
location of Tower of Babel project

I-GUARD


Contemporary art is an intrinsic part of our cultural heritage. Installations, performances, video and other forms of media art, as for example web art, have the interest of a small group of adherents, but are in comparison with more traditional art forms, far more difficult to present to a general audience. Another problem presents itself, due to the type of materials used and the context-specific aspects of these art forms, in the conservation of the works.

In our research we address the issue of providing access to these contemporary art forms from a wide variety of perspectives, ranging from the interested layman to the expert that has to deal with archiving, conserving and the possible re-installation of the art works.

intelligent guidance


  • filtering the information space according to the user's perspective, and
  • intelligent agents, that (pro) actively aid the user in searching the information space.

digital dossier(s)


  • representation of information of one or multiple works of art,
  • presentation of that information in a rich media presentation environment,
  • intelligent navigation and interaction, and
  • support for interaction with loosely-structured natural language.

...
outside view of Tower of Babel project

...
Inside view of Tower of Babel project.

(new) media art


  • audio art -- no definition available
  • computer art -- any art in which computers played a role in production or display of the artwork.
  • digital art -- art created on a computer in digital (that is, binary) form.
  • electronic art -- entry to game producer, should be Leonardo.
  • generative art -- art or design generated, composed, or constructed through computer software algorithms, or similar mathematical or mechanical autonomous processes
  • hacktivism -- the writing of code, or otherwise manipulating bits, to promote political ideology
  • interactive art -- a piece of art that involves the spectator in some way.
  • internet art -- art or, more precisely, cultural production which uses the Internet as its primary medium and, more importantly, its subject.
  • performance art -- art where the actions of an individual or a group at a particular place and in a particular time, constitute the work.
  • robotic art -- page does not exist
  • software art -- is an intersection of two almost non-overlapping realms: software and art.
  • video art -- is a subset of artistic works which relies on "moving pictures" and is comprised of video and/or audio data.
  • video game art -- involves the use of a computer game for the creation of a digital artwork.

frame(s) of reference


It is one of the most important formal qualities of film that every object that is reproduced appears simultaneously in two entirely different frames of reference, namely the two-dimensional and the three-dimensional, and that as one identical object it fulfills two different functions in the two contexts.

rules vs fiction


Game fiction is ambiguous, optional and imagined by the player in uncontrollable and unpredictable ways, but the emphasis on fictional worlds may be the strongest innovation of the video game.

...

...

10. application development

concepts


technology


projects & further reading

As a project, develop a data format for text, images and video in XML, and implement stylesheets in XSLT to convert the format for display, for example in HTML frames or using SMIL.

You may further explore the formulation of criteria for selecting software and tool support for developing multimedia applications.

For further reading I suggest, apart from the manuals and learning materials that come with your tools, to study example projects and in particular the workflow, that is the dependencies between stages in the production, as for example explained in  [McCuskey (2002)].

the artwork

  1. website of Montevideo Collection Catalogue. To avoid being parochial here, I should also mention similar institutes abroad, such as Electronic Arts Intermix from New York, USA, and LUX, from London, UK.
  2. website of INCCA.
  3. tangible virtual museum -- from  [Rosenblum and Macedonia (2005)], see section 10.1.
  4. digital dossier -- concept graph for abramovic dossier, see section 10.2.
  5. digital dossier -- presentation gadget in abramovic dossier, with video of Relation in Time, with Ulay.
  6. digital dossier -- installation Terra dea degli madre, as 3D model.
  7. conservator's studio -- Self-Portrait with Braid, see section 10.2
  8. diagram -- task world ontology,  [Welie et al. (1998)].
  9. tower of babel -- location where the event took place, see below.
  10. tower of babel -- projection of tower of babel project, see section 10.3, submitted by Katelijne Arts. The project is a concept of Katelijne Arts, Tineke Goemans, Franka van de Goor, Leidi Haaijer en Bert Vogels.
  11. tower of babel -- a view from the inside of the building.
  12. PANORAMA architecture -- from  [Si & Eliens (2007)].
  13. signs -- sports,  [ van Rooijen (2003)], p. 278, 279.

man is a playful animal

johan huizinga

game development


...

perspectives -- game development


  • artistic -- plot, narrative, style
  • technical -- choice of game engine (SDK)
  • sociological -- sharing within game communities
  • tool selection -- supporting the workflow
  • commercial -- success factors
  • management -- teamwork

...

the artwork

  1. manuscript -- used as a desktop by my favorite student.
  2. signs -- abstract symbols,  [ van Rooijen (2003)], p. 214, 215.
  3. photographs -- Jaap Stahlie, commissioned work.

11

game technology for serious applications

immersion does not require illusion but involvement

game technology for serious applications

learning objectives

After reading this chapter you should have an idea how to approach the development of a moderately complex game, and you should also be able to discuss the notion of immersion and argue why using game technology is relevant for serious applications.

Game playing is fundamental to human life. Not only for entertainment, but also to acquire the necessary skills for survival. Game playing can take a variety of forms, but nowadays the dominant game paradigm is undoubtedly the interactive video game, to be played on a multimedia-enhanced PC or game console. Currently, games are being (re) discovered in the academic field, another serious areas of society, as excellent means for both the transfer of knowledge and, perhaps more importantly, for attitude change.

In this chapter we will look at the various issues in developing a game, and more specifically, in section 11.2 at requirements for a promotional game for our faculty and the issues that came up when giving a masterclass game development for high school students using this game. Finally, we will sketch the history of immersive systems, in particular panoramas, in section 11.3, and we will discuss how immersion is to be realized in a game context.

...

game playing


... in the game we confront a function of the living creature which cannot be determined either biologically or logically ...

visual culture


games are an increasingly important element in our visual culture.

game programming


  • game play programming
  • game engine programming

game engine component(s)


  • rendering system -- 2D/3D graphics
  • input system -- user interaction
  • sound system -- ambient and re-active
  • physics system -- for the blockbusters
  • animation system -- motion of objects and characters
  • artificial intelligence system -- for real challenge(s)

...

fun


  • in the general flow of the game experience and
  • in the individual moments during a playing session.

what is a game?


a game is a series of processes that takes a player to a result.

interactive electronic game


A game is a play activity comprised of a series of actions and decisions, constrained by rules and the game world, moving towards an end condition. The rules and the game world are delivered by electronic media and controlled by a digital program.

The rules and game world exist to create interesting situations to challenge and oppose the player. The player's actions, his decisions, excitement, and chances, really, his journey, all comprise the "soul of play".

It is the richness of context, the challenge, excitement, and fun of a player's journey, and not simply the attainment of the end condition that determines the success of the game.

...
varoomthung blam

battle condition(s)


  • confrontation on well-established area
  • delimited in space/time
  • audience/participants who judge victory/loss

design team role(s)


  • manager(s) -- keep everything together
  • producer(s) -- maintaining focus
  • programmer(s) -- solve problem(s)
  • tester(s) -- control quality
  • designer(s) -- elaborate idea(s)

history


  1. phase i: before space war -- hardwired
  2. phase ii: spacewar on atari -- console with game
  3. phase iii: game console and PC -- separate game development
  4. phase iv: shakedown and consolidation -- player code in data files
  5. phase v: advent of the game engine -- user level design
  6. phase vi: the handheld revolution -- the GameBoy
  7. phase vii: the cellular phenomenon -- larger installed user base
  8. phase viii: multiplayer games -- from MUD to Everquest

...

...
Screens from Samurai Romanesque.

virtual heroes


Serious games and simulations are poised for a second revolution. Today's children, our workforce and scientists are increasingly playing, learning, and inventing in visually intensive "virtual" environments. In our increasingly experiential economy, immersive educational and training solutions are needed to advance the workforce of tomorrow. Game-based learning and technologies meet this challenge.

With regard to the use of 3D we may remark that since ancient times a walk in space has served as a mnemonic device, and as such spatial memory may aid in retention and understanding, which might also provide a decisive argument for the use of 3D in aa serious game, such as a service management game!

peace maker(s)


Q: With the lion's share of strategy games on the market being devoted to ending a conflict through violence, why was it important to you to emphasize the need for a peaceful solution?
A: When we started to work on the project and looked around at other video games, we encountered the notion that war is much more challenging and conflict is essential to engage players. Many people we talked to perceived peacemaking as mere negotiations, where a group of diplomats sit at a table for lengthy discussions and sign agreements. We tried to shed light on what we see as the other side of peacemaking how challenging it is for a leader to gain trust and understanding in the face of constant violence. How difficult it is to execute concessions, while your own population is under stress or feeling despair. In a sense, peacemaking can be more complicated, sophisticated and rewarding than war making, and it is a message that we would like to convey to young adults, the future generation of leaders.

...
black/whitegreyscale sepia thermal

goal(s)


  • to develop a game that could be used for promoting our institute, and
  • to prepare a masterclass game development for high-school students.

...

requirements game


  • the game must provide information about the faculty of sciences of the VU,
  • the game environment must be realistic and sufficiently complex, and
  • the interaction must be of a non-aggressive, non-violent, nature.

requirements masterclass


  • it must be suitable for beginners, in particular high school students,
  • it must explain basic texture manipulation, and
  • offer templates for modifying a game level, and finally
  • there must be a simple (easy to understand) manual.

...
(a) lecture room(b) lecture room (c) student office

...
(a) student office(b) student office(c) student office

...
(a) restaurant(b) restaurant(c) restaurant

technical issues


  • level design
  • game modifications
  • importing models

level design


To give an impression of the overall size of the VU.vmf game level, as map information we obtained 6464 solids, 41725 faces, 849 point entities, 1363 solid entities, and 129 unique textures, requiring in total a texture memory of 67918851 bytes (66.33 MB).

game modification(s)


  • player properties -- players start out immortal, meaning that they cannot "die" while exploring the world. Furthermore, continuous sprinting is enabled, which allows the player to walk around faster.
  • puzzle HUD -- when the player starts out, the puzzle HUD is the only HUD element displayed.
  • puzzle setter -- allows puzzle parts to be displayed on the puzzle HUD.
  • weapon enabler -- allows weapons to be enabled/disabled for the player. Enabling the weapons also enables damage, and swithes from the puzzle HUD to the default Half-Life 2 HUD, which displays weapon and damage information along with a crosshair.

importing models


  • the model must be exported to the custom Valve format smd
  • the model must be compiled from smd to mdl format

assignment(s)


  1. to modify an existing game level by applying different textures,
  2. to create objects within an existing game level, and
  3. (for advanced students only) to create a new level.

instruction(s)


  • an overview of the history of games,
  • a general introduction on modelling characters and objects,
  • the use of the Hammer editor, and finally,
  • an explanation of the assignments.

...
(a) converter(b) VMT tool(c) camera

...
(a) masterclass room(b) room in Hammer editor

hammer editor


  • block tool -- for creating simple object,
  • selection tool -- to select objects for texturing,
  • entity tool -- to select dynamic or interactive objects, and the
  • texture tool -- to apply textures to an object;

Somewhat surprisingly, all students worked directly from the (paper) manual, rather than consulting the online documentation, or the help function with the tool.

...

matrix


competition

To stimulate the participants in their creativity, we awarded the best result, according to our judgement, with a VU-Life 2 T-shirt and a CD with Half-Life 2. The results varied from a music chamber, a space environment, a Matrix inspired room, and a messy study room. We awarded the Matrix room with the first prize, since it looked, although not very original, the most coherent.

civilisation


Media are special cases within the history of civilisation. They have contributed there share to the gigantic rubbish heaps that cover the face of our planet or to the mobile junk that zips through outer space.

dead media project


Together with like-minded people, in 1995, Bruce Sterling started a mailinglist (at that time still an attractive option) to collect obsolete software. This list was soon expanded to collect dead ideas, or dead artifacts, and systems from the history of technical media: inventions that appeared suddenly and disappeared just as quickly, which dead-ended and were never developed further; models that never left the drawing board; or actual products that were bought and used and subsequently vanished into thin air.

machines can die


Sterling's project confronted burgeoning phantasies about the immortality of machines with the simple facticity of a continuously growing list of things that have become defunct.

technology and death


Once again, romantic notions of technology and of death were closely intertwined in the Dead Media Project.

scenario(s)


Michelle is a writer, and she writes a book about Dutch paintings. To collect necessary information, she analyses different sources about Dutch culture, many of them in Dutch language. However, being a beginner in Dutch, she often has to translate phrases from and to English. She also needs to find additional information about particular concepts and facts. Instead of using several tools, such as online dictionaries, definition books and search engines, she decides to compose a simple service that aggregates all the services she needs.

...

dimension(s)


  • scope -- individual vs. corporate
  • platform -- local vs. web-based
  • functionality -- data aggregation vs. interaction
  • developers -- end-user vs. professionals
Additionally we address the issue of
  • licensing -- OS vs. proprietary

service-oriented computing


  • service brokering -- to connect services,
  • service adapters -- to transform data, and
  • integration mechanisms -- to deploy services.

...
camera inputvoice input news reader

analogon of reality


Certainly, the image is not the reality but at least it is its perfect analogon and it is exactly this analogical perfection which, to our common sense, photography. This can be seen as the special status of the photographic image, it is a message without a code.

perspective(s)


  • the organisation of the image, as well as
  • the (optimal) point of view of the viewer.

...

realism


... documentary modality of black and white realism ...

...

visual grammar


grammar goes beyond formal rules of correctness. It is a means of representing patterns of experience. It enables human beings to build a mental picture of reality, to make sense of their experience of what goes on around them and inside them.

virtual reality


the idea of virtual reality only appears to be without a history: in fact, it rests firmly on historic art traditions, which belongs to a discontinuous movement of seeking illusionary image spaces.

immersion


the concept of immersion when implemented as an artwork surrenders most of the essential properties of an artwork.

properties of artwork(s)


  • form -- aesthetic whole
  • structure -- organisation of elements
  • function -- context of display
  • processuality -- (implicit) narrative structure
  • statement -- existential/political implication(s)

collective memory


it is an apparent feature of the concept of immersion that it engages with the spatial and pictorial concentration of the awareness of one's own people, the formation of collective identity through powerful images that occupy the function of memory.

ecstatic transport


using contemporary image techniques, immersive art very often visualizes elements that can best be described as Dionysian: ecstatic transport and exhilaration.

...

realism


a realism is produced by a particular group as an effect of the complex of practices which define and constitute that group.

naturalism


each realism has its naturalism, that is a realism that is a definition of what counts as real, a set of criteria for the real, and it will find its expression in the right, the best, the most natural form of representing that kind of reality, be it a photograph or a diagram.

...

dominant paradigm(s)


the dominant standard by which we judge visual realism (and hence visual modality) remains for the moment, naturalism as conventionally understood, photorealism.

relation(s)


  • response -- object disembodied from self
  • control -- self embodies object
  • reflection -- self disembodied from object
  • belonging -- object embodies self

...

mass medium


thus, one year after Monet's death and fifty years after his Impression soleil levant, a late example of modern art reached the changed artistic landscape of the 1920's, transported in a derivative of the mass medium for images in the 19th century.

new media


a consequence of the constitutive function of artistic-illusionary utopias for the inception of new media of illusion is that the media are both a part of the history of culture and of technology.

cultural convergence


the cultural convergence of art, science, and technology provides ample opportunity for artists to challenge the very notion of how art is produced and to call into question its subject matter and its meaning in society.

tele-presence


  • notions of artificial life
  • fusion with (infinite) virtual image worlds
  • transformation of self into digital data

human aspiration(s)


telepresence also combines the contents of three archetypal areas of human aspirations: automation, virtual illusion and metaphysical views of the self.

cybergnosis


what is being preached is the phantasm of union in a global net community, cybergnosis, salvation through technology, disembodied as a post-biological scattering of data that lives forever.

zealot(s)


what we observe are hyperzealots of a new technoreligion running wild, zapping, excerpting and floating in cyberspace.

aesthetics


since the eighteenth century, aesthetic theories have regarded distance as a constitutive element of reflection, self-discovery and the experience of art and nature.

tool(s)


aesthetic distance is no longer tenable when artist are engaging the same systems used in general communications and research

...

webvolution(s)


  • access -- make available through url
  • find -- using search engine
  • share -- by storing online
  • participate -- by contributing content
  • collaborate -- to create meaningful sites
  • co-create -- to build a (new) world

game event description format


  • name of event -- give a meaningful name
  • event-id -- for administrators only
  • type -- (generic/specific) game/model/video
  • cause -- game play/simulation/exploration
  • feedback/information -- give a logical description
  • player actions -- indicate all (logical) player options
  • description of visuals -- for feedback, information and player options
  • additional information -- give a url with reference(s) to information and visuals
  • relates to event(s) -- give id's or descriptions of related events

scenario(s)


  • context -- general setting, situation
  • problem -- event(s) to occur, problem to solve
  • S-R situation(s) -- stimulus/response (one or more)
  • climax -- action must be taken
  • resolution -- find solution or result

...

11. game technology for serious applications

concepts


technology


projects & further reading

As a project, develop a non-violent game using the Source SDK. For example, you may develop an application that gives a community of users access their personal collections of photographs.

One interesting feature to explore is the use of narratives, that is a kind of guided tour that gives a user an overview of the collection of photographs by means of a story, taking (in other words) the user by the hand in navigating the gane space.

For further reading I suggest, apart from the manuals and learning materials that come with the Source SDK, books on game development such as  [Luna (2003)],  [Gee (2003)] and  [Klabbers (2006)].

the artwork

  1. digital beauties -- taken from  [Wiedermann (2002)].
  2. Masereel, social realist works
  3. Roy Lichtenstein, 1962
  4. Masereel, social realist works
  5. images from Samurai Romanesque, see section 1.3
  6. HalfLife 2 shader programming
  7. VU-Life 2 -- opening screen
  8. VU-Life 2 -- screenshots
  9. VU-Life 2 -- screenshots
  10. VU-Life 2 -- screenshots
  11. VU-Life 2 -- tools
  12. VU-Life 2 -- tools
  13. VU-Life 2 -- masterclass
  14. diagram AMICO core
  15. diagram AMICO applications
  16. Roy Lichtenstein, 1962, Stillives
  17. Monet, Nympheas
  18. Monet, Nympheas
  19. Monet, Nympheas
  20. Web 3Di -- diagram for Webvolution
  21. signs -- abstract,  [ van Rooijen (2003)], p. 146, 147.

12

towards an aesthetics for interaction

experience is determined by meaning

towards an aesthetics for interaction

learning objectives

After reading this chapter you should have an understanding of the model underlying game playing, and the role of narratives in interaction. Furthermore, you might have an idea of how to define aesthetic meaning in a cultural context, and apply your understanding to the creative development of meaningful interactive systems.

As in music, the meaning of interactive applications is determined, not only by its sensory appearance, but to a high extent by the structure and functionality of the application. This observation may, also, explain, why narratives become more and more important in current video games, namely in providing a meaningful context for possible user actions.

In this chapter, we take an interactive game-model extended with narrative functionality as a starting point to explore the aesthetics of interactive applications. In section 12.1, we will introduce a model for interactive video games, and in section 12.2 we will present a variety of rules for the construction of narratives in a game context. Finally, in section 12.3, we will characterize the notion of meaning from a traditional semiotics perspective, which we will then apply in the context of games and interactive multimedia applications.

...

game theory


  • system -- (formal) set of rules
  • relation -- between player and game (affectionate)
  • context -- negotiable relation with 'real world'

...

classic game (reference) model


  • rules -- formal system
  • outcome -- variable and quantifiable
  • value -- different valorisation assignments
  • effort -- in order to influence the outcome
  • attachment -- emotionally attached to outcome (hooked)
  • consequences -- optional and negotiable (profit?)

rules vs fiction


game fiction is ambiguous, optional and imagined by the player in uncontrollable and unpredictable ways, but the emphasis on fictional worlds may be the strongest innovation of the video game.

...

theory of interaction


are games relevant for a theory of interaction?

effective service management game(s)


  • rules -- service management protocols
  • outcome -- learning process
  • value -- intellectual satisfaction
  • effort -- study procedures
  • attachment -- corporate identity
  • consequences -- job qualification

additional criteria


  • scenario(s) - problem solving in service management
  • reward(s) - service level agreements

game play


... structure of interaction with game system and other player(s)

component framework


  • holistic -- playing games as an undividable activity
  • boundary -- limit the activities of people playing games
  • temporal -- describe the flow of the game (interaction)
  • structural -- physical and logical elements of the game system

...

pattern(s)


  • resource management -- resource types, control, progress
  • communication and presentation -- information, indicators
  • actions and events -- control, rewards and penalties
  • narrative structures and immersion -- evaluation, control, characters
  • social interaction -- competition, collaboration, activities
  • mastery and balancing -- planning, tradeoffs
  • meta games and learning -- replayability, learning curve(s)

...

intimate media object(s)


  1. glow tags -- a subtle way to trigger the person who has placed it or who sees it
  2. living scrap book -- to capture and collect information and media digitally
  3. picture ball -- as an object of decoration and a focus for storytelling
  4. lonely planet listener -- enabling people to listen to a real time connection to another place

intimate media experience(s)


  • sensorial -- experience is visual, audible, tactile, olfaric
  • personalized -- objects embody meaning and memories
  • analogue -- people relate to physical objects
  • enhancement -- people already have extensive intimate media collections
  • serendipity -- it supports unstructured and flexible usage
  • longevity -- objects may exist over generations

...

experience as meaning


  • experience occurs during the interaction between the user(s) and the interactive system(s) in the lived environment
  • designers convey meaning (consciously or unconsciously) through the appearance, interaction and function of the system
  • user(s) construct a coherent whole that is a combination of sensual, cognitive, emotional and practical forms of experience

film as art


by still being read, the little treatise seems to prove that in spite of all the changes that have taken place in their form, content and function, films are still most genuinly effective when they rely on the basic properties of the visual medium.

...

illusion


... in film or theatre, so long as the essentials of any event are shown, the illusion takes place

patterns of light


... we can perceive objects and events as living and at the same time imaginary, as real objects and as simple patterns of light on the projection screen, and it is this fact that makes film art possible.

frames of reference


it is one of the most important formal qualities of film that every object that is reproduced appears simultaneously in two entirely different frames of reference, namely the two-dimensional and the three-dimensional, and that as one identical object it fulfills two different functions in the two contexts.

principles of montage


  • cutting -- unit length, whole scenes, cuts within scenes
  • time relations -- synchronized, before/after, neutral
  • space relations -- same place (different time), different place
  • subject matter -- similarity and/or contrast

film technique


  • camera -- position, focus, movement
  • transitions -- fading in/out, dissolving, stills
  • arrangement -- light/shade, color, sound

cinematographic motion


  • movement of objects
  • effect of perspective
  • motion of camera
  • montage of scenes

narrative implication(s)


  • objects -- the items in the image
  • vectors -- (imaginary) lines suggesting interaction
  • gaze -- inward (offer) / outward (demand)

composition


composition, ..., relates the representational and interactive meanings of the picture to eachother, through three interrelated systems.

representation(s)


  • information value -- left/right, top/bottom, centre/margin
  • salience -- foreground/background, relative size, contrast
  • framing -- connecting or dissolving lines

information value


  • left/right -- given versus new
  • top/bottom -- ideal versus real
  • centre/margin -- important versus marginal

...

scientific context


  • mathematics -- matrix algebra, transforms
  • physics -- game physic, particle systems
  • computer science -- technological infra-structure
  • information theory -- compression and delivery
  • media theory -- history of communication
  • semiotics -- theory of meaning

societal context


  • cultural heritage -- digital dissemination of art
  • education & communication -- presentation of concepts and examples

technological context


  • modelling -- objects, characters
  • interaction -- game programming
  • architecture -- game engine design
  • rendering -- programming the graphics hardware

creative context


  • visual design -- style, models and attributes
  • story telling -- narrative structure

learning/meaning


  • actionary level -- action and movements
  • sensory/iconic -- images and impressions
  • symbolic -- language and mathematics

...

basic geometrical shapes


... basic geometrical shapes have always been a source of fascination, even of religious awe. And our scientific age is no exception.

nervous system


(basic geometrical shapes) have been thought to have the power to directly affect our nervous system, for instance by the constructivist artist Gabo: the emotional force of an absolute shape is unique and not replaceable by any other means ...

...

semiotics -- a theory of meaning


  • signifier -- sign/symbol
  • signified -- what is referred to

semiotic modes


... is the move from the verbal to the visual a loss, or a gain?

complexity


... it has to be handled visually, because the verbal is no longer adequate?

multimedia


the multi-modality of written texts has, by and large, been ignored, whether in educational contexts, in linguistic theorizing, or in popular common sense. Today, in the age of multimedia, it can suddenly be perceived again.

quotes


  • myth of transparency -- visual communication is always coded!
  • literacy -- standards for semiotic order
  • semiotic modes -- text, visual, auditive, ...
  • computer technology -- central to semiotic landscape
  • semiotic activities -- production, transformation, development

semiotic landscape


the place of visual communication in a given society can only be understood in the context of, on the one hand, the range of forms or modes of public communication available in that society, and, on the other hand, their uses and valuations.

...

sonic act(s)


what is the meaning of meaning in apparently meaningless expressions

modality


one of the crucial issues in communication is the question of the reliability of messages. Is what we see or hear true, factual, real, or is it a lie, a fiction, something outside reality? To some extent the form of the message itself suggests the answer.

coding orientation


  • technical/scientific -- effectiveness, blueprint
  • sensory -- pleasure principle is dominant
  • abstract -- used by socia-cultural elite
  • naturalistic -- dominant common sense paradigm of realism

aesthetics of shock


it is within the realm of probability that the shock, which Walter Benjamin diagnosed as being film's aesthetic innovation, will undergo renewal and intensification with far more sophisticated means.

voyeurism


the most obvious symptom of this loss of distance will be a voyeuristic, dissecting penetration of representations of objects and bodies.

TV


for the first time in the history of man's striving for understanding, simultaneity can be experienced as such, not merely translated as a succession in time.

sensory stimulation


although the new victory over time and space represents an impressive enrichment of the perceptual world, it also favors a cult of sensory stimulation which is characteristic of the cultural attitude of our time.

direct experience


proud of our inventions -- photography, film, radio, ... -- we praise the educational virtues of direct experience.

communication


when communication can be achieved by pointing with the finger, however, the mouth grows silent, the writing hand stops and the mind shrinks.

...

channels


... the decisive questions remain: who controls the channels, who distributes right of access, and who exercises economic and political authority over the networks?

visions


... the history of technological visions is the history of our dreams, our vagaries and our errors. Media utopias fluctuate, often occurring in a magical or occult ambience.

synopsis course(s)


... the curriculum should emphasise basic principles, and to the extent possible employ open standards and open source. Practical assignments must be centered on local culture, and stimulate the young talent to explore innovative applications for cultural heritage, serious games and artistic expression.

where, what & why


  • where -- Ethiopia & VU
  • what -- introduction multimedia
  • why -- to develop curriculum

environment


  • low end computers -- windows, linux
  • elementary skills -- programming, design

assumptions


  • open source -- flex 2 sdk, Delta3D
  • open standards -- XML, X3D
  • basic principles -- exploratory development

targets


  • local -- present local cultural heritage
  • serious -- develop serious game(s)
  • benefits -- promote local culture and commerce

presentation


  1. philosophy -- pathos, ethos, logos
  2. trailer -- drama, apocalyptic, appeal to player
  3. climate star -- scientific issues & game play
  4. game development -- architecture and project plan

...

phrase(s)


  • art -- rethorics of the material
  • technology -- solving problem(s)
  • science -- establish a theory

perspective(s)


  1. transcendental -- abstract form of experience,  [Kant (1781)]
  2. speculative -- criteria for beauty,  [Kant (1781)]
  3. phenomenological -- self-conscious subjectivity,  [Hegel (1807)]
  4. psychoanalytical -- sub-conscious meaning,  [Freud (1958)]
  5. pragmatical -- art as experience,  [Dewey (1931)]
  6. hermeneutical -- understanding of the senses,  [Hermeneutics]
  7. semiotics -- social construction of meaning,  [Kress and van Leeuwen (1996)]

dimensions of aesthetic awareness


  • spatial -- topological relations, layout of image
  • temporal -- order, rhythm, structure
  • dynamic -- interaction, reflection, involvement

...

12. towards an aesthetics for interaction

concepts


technology


projects & further reading

As a project, explore the ways narratives may be constructed from a collection of images. Deploy the various editing facilities for providing flashbacks, flashforwards, and other (temporal) relations within storytelling.

You may implement this using flash, VRML, or even try to embed such a narration facility in a game level developed with the Delta3D or the Source SDK.

For further reading I suggest you to take a look at more theoretical material from media theory, such as  [Bolter and Grusin (2000)]. Also there is a large collection of books from MIT Media Press that is of relevance for our new visual culture.

the artwork

  1. einzelganger -- walking man of Alberto Giacommeti, taken from an aanouncement of the Ives Ensemble, Amsterdam.
  2. game component framework, from  [Björk & Holopainen (2005)].
  3. diagram MIME
  4. diagram experience as meaning
  5. Roy Lichtenstein, 1962, (a) Kiss II, (b) Masterpiece, (c) Forget it, forget me.
  6. edgecodes -- showing George Lucas and his editoroid.
  7. El Lissitzky, suprematist works
  8. El Lissitzky, suprematist works
  9. Roy Lichtenstein, 1999, Still lifes with brushstrokes
  10. Les Demoisselles dAvignon, Picasso, 1908, regarded as the start of Cubism, and Le Goutier, Jean Metzinger, 1911, often referred to as the Mona Lisa of Cubism.
  11. poster for exhibition of dutch china work.
  12. signs -- abstract,  [ van Rooijen (2003)], p. 228, 229.

afterthought(s)

media equation(s) 1/4


We regularly exploit the media equation for enjoyment by the willing suspension of our critical faculties. Theatre is the projection of a story through the window of a stage, and typically the audience gets immersed in the story as if it was real.

multimedia equation(s) 2/4


multimedia = presentation + context

multimedia equation(s) 3/4


  • context = convergence + information + architecture

where

multimedia equation(s) 4/4


  • convergence = data +platform + distribution
  • information = storage and retrieval
  • architecture = compression + components + connectivity

perfect solutions


Much more than the art of turning base metals into gold, alchemy is a system of cosmic symbolism. The alchemist learns how to create within a sealed vessel a Model of the Universe in which the opposing complementary forces of Male and Female, Earth and Air, Fire and Water attain the perfect synthesis of which gold is the emblem.

multimedia engineering


"engineering is the art of moulding materials we do not wholly understand ... in such a way that the community at large has no reason to suspect the extent of our ignorance."

multimedia courses


  • multimedia authoring -- Web3D/VRML
  • intelligent multimedia technology -- Virtual Environments
  • visual design -- digital content creation
  • multimedia casus -- digital dossier(s)

virtual reality interfaces for multimedia information systems

topical


  1. designed for involving local application (as an anesthetic),
  2. relating to, or arranged by, topics,
  3. referring to the topics of the day or place.

Multimedia Authoring I -- Web3D/VRML


  • product demo -- with descriptive information and animation(s)
  • infotainment VR -- in the areas of Culture, Commerce or Entertainment

www.web3d.org


The term Web3D describes any programming or descriptive language that can be used to deliver interactive 3D objects and worlds across the internet. This includes open languages such as Virtual Reality Modeling Language (VRML), Java3D and X3D (under development) - also any proprietary languages that have been developed for the same purpose come under the umbrella of Web3D. The Web3D Repository is an impartial, comprehensive, community resource for the dissemination of information relating to Web3D and is maintained by the Web3D Consortium.

X3D SDK


This comprehensive suite of X3D and VRML software is available online at sdk.web3d.org and provides a huge range of viewers, content, tools, applications, and source code. The primary purpose of the SDK is to enable further development of X3D-aware applications and content.

abstraction and grouping

geometry and appearance

interaction and behavior

special effects

extensions

protos


  • slideset -- container for slides
  • slide -- container for text and objects
  • slide -- container for lines of text
  • line -- container for text
  • break -- empty text

slide 



  PROTO slide [
      exposedField SFVec3f    translation 0 0 15 
      exposedField SFRotation rotation    0 1 0 0
      exposedField SFVec3f    scale       1 1 1
      exposedField MFNode     children []
  ] {
      Transform {
          children    IS children
          translation IS translation
          rotation    IS rotation
          scale       IS scale
      }
  }

slideset



   PROTO slideset [
      exposedField SFInt32    visible 0
      exposedField MFNode     slides [ ]
      eventIn SFInt32         next
   ] {
      DEF select Switch {
          choice      IS slides
          whichChoice IS visible
      }
  
      Script {
      ...
      }
   }

script



      Script {
      directOutput TRUE
      eventIn SFInt32 next IS next
      field SFInt32 slide IS visible
      field SFNode select USE select
      field MFNode slides []
      url "javascript:
      function next(value) {
      slides = select.choice;
      Browser.print('=' + slide + ' ' + slides.length);
      if (slide >= (slides.length-1)) slide = 0;
      else slide += 1;
      select.whichChoice = slide;
      }"
      }

example



  DEF slides slideset {
   slides [
      slide {
  	children [
  		text { 
  			lines [
  			  line { string ["What about the slide format?"] }
  			  break { }
  			  line { string ["yeh, what about it?"] }
  			  break { }
  			] # lines
  		}
  	        Sphere { radius 0.5 }
  	] # children
      } # slide 1
    
  
      slide { # 2
  	children [
  	        Sphere { radius 0.5 }
  	] 
      } # slide 2
   ]  # slides
  }

timer



  
  DEF time TimeSensor { loop TRUE cycleInterval 10 }
  DEF script Script {
  eventIn SFTime pulse 
  eventOut SFInt32 next
  url "javascript: function pulse(value) { next = 1; }"
  }
  ROUTE time.cycleTime TO script.pulse
  ROUTE script.next TO slides.next

XML-based multimedia


  • introduction: Extensible Markup Language (XML). Extensibility and profiling of web-based multimedia. Streaming. Model of timing and synchronization of web-based multimedia.
  • processing XML: XSLT stylesheets, Java-based XML Processing, SAX, DOM, Java XSL object APIs
  • SMIL: (Synchronized Multimedia Integration Language) SMIL modules: animation, content control, layout, linking, media object, metainformation, timing, and profiles.
  • X3D: (XML-based VRML) Extensible 3D: architecture and based components, profile reference, translation between VRML and X3D. X3D examples: case studies.
  • VHML: (Virtual Human Markup Language) Virtual Human Markup Language, Humanoid, H-anim specification, Speech Synthesis Markup Language Specification for the Speech Interface Framework (Speech ML), Voice Extensible Markup Language (VoiceXML). Text to Speech Technology.
  • STEP: Scripting Technology for Embodied Persona and XSTEP, the XML-encoding of STEP and its processing tools. Embodied agents and multimedia presentation: theory, model, and practice.

slides in XML



  <slideset>
  <slide id="1">
  <text>
  <line>What about the slide format?</line>
  <break/>
  <line string="yeh, what about it">?</line>
  </text>
  <vrml>Sphere { radius 0.5 }</vrml>
  </slide>
  <slide id="2">
  <vrml>Sphere { radius 0.5 }</vrml>
  </slide>
  </slideset>

XSLT stylesheet



  <?xml version="1.0"?>
  <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="text"/>

slideset



  <xsl:template match="/slideset">
  
  ... load (extern) proto(s)
  
  DEF slides slideset {
  slides [
  <xsl:apply-templates/>
  ] # slides
  }
  
  ... include timer or user interface
  
  </xsl:template>

slide



  <xsl:template match="*/slide">
  slide { children [
  <xsl:apply-templates/>
  ] }
  </xsl:template>

text



  <xsl:template match="*/text">
  text { lines [
  <xsl:apply-templates/>
  ] }
  </xsl:template>

line



  <xsl:template match="*/line">
  line { string [ "<xsl:value-of select="@string"/>
  <xsl:apply-templates/> " ] }
  </xsl:template>

etcetera



  <xsl:template match="*/break">
  line { string [ "<xsl:apply-templates/>" ] }
  </xsl:template>
  
  <xsl:template match="*/vrml">
  <xsl:apply-templates/>
  </xsl:template>
  
  </xsl:stylesheet>

conversation


A platform for Embodied Conversational Agents

based on Distributed Logic Programming

Intelligent Multimedia Group

Anton Eliens Zhisheng Huang Cees Visser

Vrije Universiteit, Amsterdam

intelligent multimedia


... intelligent multimedia provides a merge between technology from AI, in particular agent-technology, and multimedia ...

Multimedia Authoring II -- virtual environments


  • intelligent services in virtual environments

DLP


  • extension of Prolog
  • (distributed) objects
  • non-logical instance variables
  • multiple inheritance
  • multi-threaded objects
  • communication by rendez-vous
  • (synchronization) accept statements
  • distributed backtracking

DLP+X3D


  • control points: get/set -- position, rotation, viewpoint
  • event-handling -- asynchronous accept

soccer rule



  findHowToReact(Agent,Ball,Goal,shooting) :-
     get(Agent,position,sfvec3f(X,Y,Z)),
     get(Ball,position,sfvec3f(Xb,Yb,Zb)),
     get(Goal,position,sfvec3f(Xg,Yg,Zg)),
     distance(sfvec3f(X,Y,Z),sfvec3f(Xb,Yb,Zb),DistB),
     distance(sfvec3f(X,Y,Z),sfvec3f(Xg,Yg,Zg),DistG),
     DistB =< kickableDistance,
     DistG =< kickableGoalDistance.

observer



  :- object observer : [actions].
  var slide = anonymous, level = 0, projector = nil.
  
  observer(X) :- 
     projector := X,
     repeat,
       accept( id, level, update, touched),
     fail.
  
  id(V) :-  slide := V.
  level(V) :- level := V.
  touched(V) :- projector<-touched(V).
  update(V) :- act(V,slide,level).
  :- end_object observer.

...

dialog



  <phrase right="how~are~you">
  <phrase left="fine~thank~you"/>
  <phrase right="what do~you think~of studying ..."/>
  ...
  <phrase left="So,~what~are you?"/>
  <phrase right="an ~agent" style="[a(e)=1]"/>
  <phrase left="I always~wanted to be~an agent" style="[a(e)=1]"/>

style parameters



  <phrase right="red" style="[p=(0.5,0,0),persona=0,balloon=0]"/>
  <phrase left="cutie" style="[p=(-0.5,0,0),persona=0,balloon=0]"/>
  <gesture right=1 style=default/>
  <gesture left=1 style=default/>

levels of complexity


basic actions


  • move -- move(Agent,BodyPart,Direction,Duration)
  • turn -- turn(Agent,BodyPart,Direction,Duration)

composite operators


  • sequence -- [Action_1,...,Action_{n}]
  • parallel -- parallel([Action_1,...,Action_{n}])
  • choice -- disjunct([Action_1,...,Action_{n}])
  • repeat -- repeat(Action,k)

example



  script(walk(Agent), ActionList) :-
  ActionList = [
    parallel([turn(Agent,r_shoulder,back_down2,fast),
   	turn(Agent,r_hip,front_down2,fast),
   	turn(Agent,l_shoulder,front_down2,fast),
   	turn(Agent,l_hip,back_down2,fast)]),
    parallel([turn(Agent,l_shoulder,back_down2,fast),
   	turn(Agent,l_hip,front_down2,fast),
   	turn(Agent,r_shoulder,front_down2,fast),
   	turn(Agent,r_hip,back_down2,fast)])
  ], !.

...

interaction


  • test -- test(State)
  • execution -- do(State)
  • conditional -- if_then_else(State,Action1,Action2)
  • until -- until(Action,State)

XSTEP 



  <action type="walk(Agent)">
  <seq>
   <par speed="fast">
    <gesture type="turn" actor="Agent" part="r_shoulder" dir="back_down2"/>
    ...
   </par>
   <par speed="fast">
    ...
    <gesture type="turn" actor="Agent" part="r_hip" dir="back_down2"/>
   </par>
  </seq>
  </action>

STEP


#@include steps.vr

END OF EXERCISE powered by DLP

systems and projects


  • DIVE -- multi-user
  • Parlevink -- applications
  • Jinni -- logic-based
  • Alice -- scripting
  • VHML -- mark up

PRO

  • high level platform -- flexible and powerful
  • clean separation of concerns -- modelling and programming

CON

  • added complexity -- due to hybrid platform
  • performance penalty -- due to EAI communication

TODO

  • models -- movement, behavior, moods
  • mark up -- dialogs, actions, style
  • support -- text-to-speech, interaction

www.cs.vu.nl/~eliens/research

content(s) / list(s) / online(s) / web3d / resource(s) / tool(s) / technology / ? / zaal

resource(s) -- design

resource(s) -- game(s)

content(s) / list(s) / online(s) / web3d / resource(s) / tool(s) / technology / ? / zaal

imaging and graphics

3D modeling

Alias Wavefront Maya

Discreet 3D Studio Max

content(s) / list(s) / online(s) / web3d / resource(s) / tool(s) / technology / ? / zaal

DirectX SDK 9

Wild Tangent

Virtools Software Suite

OpenML

open source technology

outline



     title -- indicating the topic
     name  -- to tell who you are
     abstract -- giving the 'message' of your efforts
     introduction --  clarifying the approach and structure
     background -- explaining the context of the subject
     sections -- to elaborate on the subject
     related work -- characterizing related approaches
     conclusion(s) -- summarizing the main point(s)
     references -- listing the literature you consulted
     appendices (optional) -- providing extra information

perspective(s)


  • review/background -- sketch perspectives, history, viewpoints
  • case study -- analyse assumptions, gather empirical data, and explain!
  • technical analysis -- technology-oriented, work out the details
  • formal study -- clarify in a formal manner, conceptualize and formalize
  • tutorial -- explain for the laymen, but do it very good

authoring convergence standards retrieval
review/background -+++++
case study ++++
technical analysis -++++++
formal study --++-
tutorial ---?-

title(s) / web3d


select:

references
[Role] Adams J. (2002),
Programming Role Playing Games with DirectX, Premier Press
[Animation] Adams J. (2003),
Advanced Animation with DirectX, Premier Press
[Perspective] Alberti L.B. (1435),
On painting and on sculpture, Phaidon, edited by C. Grayson, 1972
[Multiplayer] Alexander T., ed. (2005),
Massively Multiplayer Game Development 2, Charles River Media
[Cyberspace] Anders P. (1999),
Envisioning Cyberspace -- Designing 3D Electronic Spaces, McGraw-Hill
[Angel] Angel E. (1997),
Interactive Computer Graphics -- A top-down approach with OpenGL, Addison-Wesley
[Film] Arnheim R. (1957),
Film as Art, The University of California Press
[Collada] Arnaud R, and Barnes M.C. (2006),
Collada -- sailing the gulf of 3D digital content creation, A.K. Peters Ltd.
[Enhancement] Astleneir H. (2000),
Designing emotionally sound instruction: the FEASP approach, Instructional Science 28, pp. 169-198
[Framework] Atanasova T., Nern H.J., Dziech A. (2007),
Framework Approach for Search and Meta-Data Handling of AV Objects in Digital TV Cycles, Workshop on Digital Television, Proc. EUROMEDIA 2007, Delft, Netherlands
[Glass] Auster P. (2004),
City of Glass, Faber and Faber, adaptation by Karasik P. and Mazzucchelli D.
[PAR] Badler N., Bindiginavale R., Bourne J., Palmer M., Shi J., Schuler W. (1998),
A Parameterized Action Representation for Virtual Human Agents, Workshop on Embodied Conversational Characters, WECC98, Lake Tahoe, CA,Oct 12-15, 1998.
[IR] Baeza-Yates R. and Ribeiro-Neto B. (1999),
Modern Information Retrieval, Addison-Wesley, 1999
[Query] Ballegooij A. van and Eliens A. (2001),
Navigation by Query in Virtual Worlds, Web3D 2001 Conference, Paderborn, Germany, 19-22 Feb 2001
[Archeology] Barcelo S.A., Forte M., Sanders D.H., eds. (2000),
Virtual Reality in Archeology, Bar International Series 843, 2000
[Strategy] Barron T. (2003),
Strategy Game Programming with DirectX 9.0, Wordware Publishing
[Impasto] Baxter W., Wendt J., and Lin M. (2004),
IMPaSTo: A Realistic, Interactive Model for Paint, Proc. of NPAR 2004, The 3rd Int. Symp. on Non-Photorealistic Animation and Rendering. June 7-9 2004, Annecy, France. pp. 45--56.
[Reproduction] Benjamin W. (1936),
The Work of Art in the Age of Mechanical Reproduction, Online Archive -- http://www.marxists.org
[Web] Berners-Lee T., Hendler J., Lassila O. (2001),
The semantic web, Scientific American, may 2001, pp. 28-37
[GamePatterns] Björk S. and Holopainen J. (2005),
Patterns in Game Design, Charles River Media
[Flat] Betsky A. with Eeuwens A. (2004),
False Flat -- Why Dutch Design is so Good, Phaidon
[Vox] Bocconi S. (2006),
Vox Populi: generating video documentation from semantically annotated data repositories, Ph.D. Thesis SIKS 2006-27
[Remediation] Bolter J.D and Grusin R. (2000),
Remediation -- Understanding New Media, MIT Press
[Monet] Boncz P.A. and Kersten M.L. (1995),
Monet -- an impressionist sketch of an advanced database system, In Proc. BIWIT95
[Interactivation] Bongers B. (2006),
Interactivation -- Towards an e-cology of people, our environment, and the arts, Ph.D. Thesis, Vrije Universiteit
[FilmTheory] Bordwell D. and Thomson K. (2003),
Film art, an introduction, McGraw-Hill, 7th edn.
[History] Briggs A. and Burke P. (2001),
A social history of the media -- from Gutenberg to the Internet , Polity Press
[Environments] Broll W. (1996),
VRML and the Web: A basis for Multi-user Virtual Environments on the Internet, In Proceedings of WebNet96, H. Maurer (ed.), AACE, Charlottesville, VA (1996), 51-56.
[Angelic] Broll W., Shäfer L., Höllerer T., Bowman D. (2001),
Interface with Angels: the future of VR and AR interfaces, IEEE Computer Graphics, November/December 2001, pp. 14-17
[Education] Bruner J.S. (1972),
Relevance of Education, Penguin Education
[Avantgarde] Burger P. (1981),
Theorie de Avantgarde, Edition Suhrkamp
[Bush] Bush V. (1945),
As we may think, Atlantic Monthly, July 1945
[Next] CACM 44:3 (2001),
The next 1000 years, Communications of the ACM, March 2001, 44:3
[ Capin T.K., Pandzic I.S., Magnenat-Thalmann N., Thalman D. (1999),
Avatars in Networked Virtual Environments, Wiley
[Annotated] Carey R. and Bell G. (1997),
The Annotated VRML 2.0 Reference Manual, Addison-Wesley
[Prediction] Cesa-Bianchi N. and Lugosi G. (2006),
Prediction, Learning, and Games, Cambridge University Press
[Spaces] Chang S.C. and Costabile M.F. (1997),
Visual Interfaces to Multimedia Databases, In [_Handbook]
[Chapman1] Chapman N. and Chapman J. (2004a),
Digital Multimedia, Wiley, 2nd edn.
[Chapman2] Chapman N. and Chapman J. (2004b),
Digital Media Tools, Wiley, 2nd edn.
[Beauty] Cheng F. (2006),
Five meditations on beauty (in french), Editions Albin Michel
[Video] Christel, M., Olligschlaeger, A., Huang, C. (2000),
Interactive maps for digital video, IEEE Multimedia 7(1), pp. 60-67
[Collaborative] Churchill E.F., Snowdon D.N. and Munro A.J., eds. (2001),
Collaborative Virtual Environments -- Digital Places and Spaces for Interaction, Springer
[Modeling] Conger D. (2004),
Physics Modeling, Thomson
[Hypertext] Conklin J. (1987),
Hypertext: An Introduction and Survey, IEEE Computer 20(9), pp. 17-41
[Design] Danaher S. (2004),
Digital 3D Design, Thomson
[DTV] Daskalova H. and Atanasova T. (2007),
Web Services and Tools for their Composition considering Aspects of Digital TV Workflow, Workshop on Digital Television, Proc. EUROMEDIA 2007, Delft, Netherlands
[Entertainment] Davenport G.(2000),
Your own virtual story world, Scientific American, november 2000, pp. 61-64
[Survey] Davison A. (1993),
A survey of Logic Programming-based Object-Oriented Languages, In Research Directions on Concurrent Object-Oriented programming, G. Agha, P. Wegner, A. Yonezawa (eds.), MIT Press (1993)
[Scripts] Davison A. (2001),
Enhancing VRML97 Scripting, Euromedia\'2001, Valencia, Spain, April 18-20. available from: http://fivedots.coe.psu.ac.th/~ad
[Logic] Davison A. (2001b),
Logic Programming Languages for the Internet, Chapter in Computational Logic: From Logic Programming into the Future, Antonis Kakas, Fariba Sadri (eds.), Springer Verlag (2001). available from: http://fivedots.coe.psu.ac.th/~ad
[Pragmatics] Dewey J. (1931),
Art as Experience, Ref ???
[Vasarely] Diehl G. (1973),
Vasarely, Crown Publishers Inc.
[Distributed] Diehl S. (2001),
Distributed Virtual Worlds -- Foundations and Implementation Techniques using VRML, Java and Corba, Springer
[Reactions] Dijkstra K., Zwaan R. Graesser A.and J. Magliano (1994),
Character and reader emotions in literary texts, Poetics 23, pp. 139-157
[Dix] Dix A., Finlay J., Abowd G., Beale R. (1998),
Human-Computer Interaction (1998), Prentice Hall, 2nd edn.
[Maps] Dodge M. and Kitchin R. (2000),
Mapping Cyberspace, Routledge
[Atlas] Dodge M. and Kitchin R. (2002),
Atlas of Cyberspace, Addison-Wesley
[Humour] Dormann C., Barr P. and Biddle R. (2007),
Humour Theory and Videogames: Laughter in the Slaughter, to appear in XXX
[Virtual] Earnshaw R. and Vince J., eds. (2002),
Intelligent Agents for Mobile and Virtual Media, Springer
[Engine] Eberly D.H. (2001),
3D Game Engine Design, Morgan Kaufmann Publishers
[Physics] Eberly D.H. (2004),
Game Physics, Morgan Kaufmann Publishers
[Eco] Eco U. (1994),
Six walks in the fictional woods, Harvard University Press
[Creativity] Eliens A. (1979),
Creativity: reflection and involvement, Ninth Int Conf of Aesthetics, Dubrovnic, August 1979
[Art] Eliens A. (1988),
Computational Art, First Int Symposium on Electronic Art, Leonardo Supplementary Issue, Pergamon Press 1988, pp. 21-26
[DLP] Eliens A. (1992),
DLP -- A language for Distributed Logic Programming, Wiley
[Workshop] Eliens A. (1996),
workshop: Logic Programming and the Web, \\ http://www.cs.vu.nl/~eliens/online/workshops/www6
[Jamming] Eliens A., Welie M. van, Ossenbruggen J. van and Schonhage S.P.C (1997) ,
Jamming (on) the Web, In: Proceedings of the 6th International World Wide Web Conference --- Everone, Everything Connected, O\'Reilly and Associates, Inc. April 1997, pp. 419-426
[Simulate] Eliens A., Niessink F., Schonhage S.P.C., van Ossenbruggen J.R., Nash P. (1996),
Support for BPR -- simulation, hypermedia and the Web, Proceedings Euromedia\'96, Euromedia, London 1996
[OO] Eliens A. (2000),
Principles of Object-Oriented Software Development, Addison-Wesley Longman, 2nd edn.
[IMVU] Eliens A. (2002),
intelligent multimedia @ VU, http://www.cs.vu.nl/~eliens/research
[Platform] Eliens A., Huang Z., and Visser C. (2002),
A platform for Embodied Conversational Agents based on Distributed Logic Programming, AAMAS Workshop -- Embodied conversational agents - let\'s specify and evaluate them!, Bologna 17/7/2002
[TIDSE] Eliens A., Dormann C., Huang Z. and Visser C. (2003),
A framework for mixed media -- emotive dialogs, rich media and virtual environments, Proc. TIDSE03, 1st Int. Conf. on Technologies for Interactive Digital Storytelling and Entertainment, Göobel S. Braun N.,n Spierling U., Dechau J. and Diener H. (eds>), Fraunhofer IRB Verlag, Darmstadt Germany, March 24-26, 2003
[Slides] Eliens A., Huang Z., Visser C. (2005),
Presentational VR -- What is the secret of the slides?, in preparation
[Navigate] Eliens A., van Riel C., Wang Y. (2006),
Navigating media-rich information spaces using concept graphs -- the abramovic dossier, accepted for: International Conference on Multidisciplinay Information Sciences and Technologies (InSciT2006) October, 25-28th 2006, Merida, Spain www.instac.es/inscit2006
[Odyssee] Eliens A. (2006b),
Odyssee -- explorations in mixed reality theatre, accepted for: GAME'ON-NA'2006, September 19-21, 2006 - Naval Postgraduate School, Monterey, USA
[VULife] Eliens A., S.V. Bhikharie (2006),
game @ VU -- developing a masterclass for high-school students using the Half-life 2 SDK, GAME'ON-NA'2006, September 19-21, 2006 - Naval Postgraduate School, Monterey, USA
[Serious] Eliens A. and Chang T. (2007),
Let's be serious -- ICT is not a (simple) game, In Proc. FUBUTEC 2007, April 2007, Delft
[VUSL] Eliens A. Feldberg F., Konijn E., Compter E. (2007) ,
VU @ Second Life -- creating a (virtual) community of learners, In Proc. EUROMEDIA 2007, Delft, Netherlands
[Recommend] Eliens A. and Wang Y. (2007),
Expert advice and regret for serial recommenders, In Proc. EUROMEDIA 2007, Delft, Netherlands
[Climate] Eliens A., van de Watering M., Huurdeman H. Bhikharie W., Lemmers H., Vellinga P. (2007b),
Clima Futura @ VU -- communicating (unconvenient) science, submitted to GAME ON 2007
[Mashups] Eliens A. Feldberg F., Konijn E., Compter E. (2007c) ,
Mashups in Second Life @ VU, submitted to GAME ON 2007
[NewPanorama] Eliens A. and Vyas V. (2007) ,
Panorama -- explorations in the aesthetics of social awareness, submitted to GAME ON 2007
[Aesthetics] Eliens A., Vyas V. and van der Veer G. (2007d) ,
A theoretical foundation for the aesthetics of interaction and awareness -- making sense of the senses, in preparation
[Beginning] Engel W.F. (2003),
Beginning Direct3D Programming, Premier Press
[Shader1] Engel W.F., ed. (2004a),
Shader X2, Introduction & Tutorials with DirectX 9, Wordware Publishing
[Shader2] Engel W.F., ed. (2004b),
Shader X2 -- Shader Programming, Tips & Tricks with DirectX 9, Wordware Publishing
[Shader3] Engel W.F., ed. (2005),
Shader X3, Advanced Rendering with DirectX and OpenGL, Charles River Media
[Engelbart] Engelbart D. (1963),
A Conceptual Framework for the Augmentation of Man\'s Intellect, Vistas in Information Handling 1(9)
[Managing] England E. and Finney A. (1999),
Managing Multimedia -- project Management for Interactive Media, Addison-Wesley
[Metamorphose] Entabaclaz A. (2003),
Les metamorphoses d\'Ulysse -- reecritures de l\'Odyssee, Editions Flammarion, Paris
[ Ericson C. (2005),
Real-time Collision Detection, Morgan Kaufmann Publishers
[Black] Fan J., Ries E. and Tenitchi C. (1996),
Black Art if JAVA Game Programming, Waite Group Press
[Faulkner] Faulkner X. (2000),
Usability Engineering, MacMillan Press
[Audio] Fay T.M. with Selfon S. and Fay T.J. (2004),
DirectX Audio Exposed -- Interactive Audio Development, Wordware Publishing Inc.
[Cg] Fernando R. and Kilgard M.J. (2003),
The Cg Tutorial -- The Definitive Guide to Programming Real-Time Graphics, Addison-Wesley
[Future] Fischetti M. (2000),
The future of digital entertainment, Scientific American, november 2000, pp. 31-33
[Networked] Fluckiger R. (1995),
Understanding networked multimedia -- applications and technology, Prentice Hall, 1995
[Persuasive] Fogg B.J. (2003),
Persuasive Technology -- Using Computers to Change What We Think and Do, Morgan Kaufmann Publishers
[Graphics] Foley J.D. and van Dam A. (1982),
Fundamentals of Interactive Computer Graphics, Addison-Wesley
[Convergence] Forman P. and Saint John R.W. (2000),
Digital Convergence, Scientific American, november 2000, pp. 34-40
[Witz] Freud S. (1958),
Der Witz und seine Beziehung zum Unbewussten, Fischer Verlag
[Dolores] Fuhr, N., Gövert, N., Rölleke, Th. (1998),
DOLORES: A System for Logic-Based Retrieval of Multimedia Objects, In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 257-265. ACM, New York
[Ludic] Gaver W. (2002),
Designing for Homo Ludens, i3 Magazine
[VideoGame] Gee J.P. (2003),
What video games have to teach us about learning and literacy, Palgrave Macmillan
[Neuromancer] Gibson W. (1986),
Neuromancer,
[Image] Gonzales R.C. and Wintz P. (1987),
Digital Image Processing, Addison-Wesley, 2nd edn.
[VirtualArt] Grau O. (2003),
Virtual Art -- From Illusion to Immersion, The MIT Press
[Handbook] Grosky W., Jain R., Mehrotra R., eds. (1997),
The Handbook of Multimedia Information Management, Prentice Hall, 1997
[Presence] Hallnäss L. and Redström J. (2002),
From Use to Presence: On the Expressions and Aesthetics of Everyday Computational Things, ACM Trans. on Computer-Human Interactions, Vol. 9, No 2, pp. 106-124
[AHM] Hardman L., Bulterman D., van Rossum G. (1994),
The Amsterdam Hypermedia Model: Adding Time and Context to the Dexter Model, CACM 37(2), pp. 50-62, febr 1994
[DL] Harel D. (1984),
Dynamic Logic, In: Handbook of Philosophical Logic, Vol. II, D. Reidel Publishing Company, 1984, pp. 497-604
[Worlds] Hartman J. and Wernecke J. (1996),
The VRML 2.0 Handbook -- Building Moving Worlds on the Web, Addison-Wesley
[Cinema] Hawkins B. (2005),
Real-time cinematography for games, Charles River Media
[Phenomenology] Hegel G.W.F. (1807),
Phänomenologie des Geistes, Felix Meiner Verlag, ed. 1952
[Sein] Heidegger M. (1927),
Sein und Zeit, Niemeyer Tubingen, 1993
[Similarity] Hewlett W.B. and Selfridge-Field E., eds. (1998) ,
Melodic Similarity -- Concepts, Procedures and Applications, MIT Press
[Interactive] Hildebrand M., Eliens A., Huang Z. and Visser C. (2003),
Interactive Agents Learning their Environment, Proc. Intelligent Virtual Agents 2003, Irsee, September 15-17, 2003 J.G. Carbonell and J.Siekmann (eds.), LNAI 2792, Springer, pp. 13-17
[Hix] Hix D. and Hartson R. (1993),
Developing User Interfaces, Wiley
[Mathematics] Hoggar S.G. (1992),
Mathematics for Computer Graphics, Cambridge University Press
[Giacometti] Hohl H. (1971),
Giacometti, Abrams Publishers
[AWC] Hoorn J., Eliens A., Huang Z., van Vugt H.C., Konijn E.A., Visser C.T.,
Agents with character: Evaluation of empathic agents in digital dossiers, Emphatic Agents, AAMAS 2004 New York 19 July - 23 July, 2004
[Taxonomy] Huang Z., Eliens A., van Ballegooij A., De Bra P. (2000),
A Taxonomy of Web Agents, IEEE Proceedings of the First International Workshop on Web Agent Systems and Applications (WASA \'2000), 2000.
[Architecture] Huang Z., Eliens A., and De Bra P. (2001),
An Architecture for Web Agents, Proceedings of the Conference EUROMEDIA 2001, 2001.
[ Huang Z., Eliens A., Visser C. (2001),
Programmability of Intelligent Agent Avatars, Proceedings of the Agent\'01 Workshop on Embodied Agents, June 2001, Montreal, Canada
[Community] Huang Z., Eliens A., Visser C. (2002),
3D Agent-based Virtual Communities, In: Proc. Int. Web3D Symposium, Wagner W. and Beitler M.( eds), ACM Press, pp. 137-144
[Intelligent] Huang Z., Eliens A., Visser C. (2005),
Intelligent Multimedia Technology: An Approach to Combine Agent Technologies with Multimedia, in preparation
[STEP] Huang, Z., Eliens, A., and Visser, C.,
STEP: a Scripting Language for Embodied Agents, in: Helmut Prendinger and Mitsuru Ishizuka (eds.), Life-like Characters, Tools, Affective Functions and Applications, Springer-Verlag, 2003.
[Reach] Huang, Z., Eliens, A., and Visser, C. (2003d),
"Is it within my reach?" -- an agents perspective , Proc. Intelligent Virtual Agents 2003, Irsee, September 15-17, 2003, J.G. Carbonell and J.Siekmann (eds.), LNAI 2792, Springer, pp. 150-158
[STEP-face] Huang Z., Eliens A., and Visser C. (2004),
Facial Animation in STEP, AAMAS 2004 New York 19 July - 23 July, 2004
[STEP-book] Huang, Z., Eliens, A., and Visser, C. (2003c),
STEP: a Scripting Language for Embodied Agents, in: Helmut Prendinger and Mitsuru Ishizuka (eds.), Life-like Characters, Tools, Affective Functions and Applications, Springer-Verlag, (to appear).
[XSTEP] Huang Z., Eliens A., Visser C. (2003b),
XSTEP: A Markup Language for Embodied Agents, Proc. CASA03, The 16th Int. Conf. on Computer Animation and Social Agents
[STEPIMP] Huang Z., Eliens A., Visser C. (2003a),
Implementation of a scripting language for VRML/X3D-based embodied agents, Proc. Web3D 2003 Symposium, Saint Malo France, S. Spencer (ed.) ACM Press, pp. 91-100
[STEP-imp] Huang Z., Eliens A., Visser C. (2003a),
Implementation of a scripting language for VRML/X3D-based embodied agents, Proc. Web3D 2003 Symposium, Saint Malo France, S. Spencer (ed.) ACM Press, pp. 91-100
[STEP-workshop] Huang Z., Eliens A., Visser C. (2002b),
STEP -- a scripting language for Embodied Agents, PRICAI-02 Workshop -- Lifelike Animated Agents: Tools, Affective Functions, and Applications, Tokyo, 19/8/2002
[Magic] Hughes B. (2000),
Dust or Magic -- Secrets of Successful Multimedia Design, Addison-Wesley, 2000
[Modern] Hummelen IJ. and Sill\'e D. (1999),
Modern Art: Who cares?, Foundation for the Conservation of Modern Art and the Netherlands Institute for Cultural Heritage
[Humdrum] Huron D. (1997) ,
Humdrum and Kern: Selective Feature Encoding, In [_Beyond], pp. 375-400
[Experience] Jain R. (2000),
Digital Experience, [_1000], pp. 38-40
[Java3D] Javasoft (2001),
Java3D API, http://java.sun.com/products/java-media/3D
[Participatory] Jenkins H. (2006),
Confronting the Challenges of Participatory Culture: Media Education for the 21th Century, White Paper, MIT MediaLab
[EducationalVR] Johnson A., Moher T., Cho Y-J., Lin Y.J., Haas D., and Kim J. (2002),
Augmenting Elementary School Education with VR, IEEE Computer Graphics and Applications, March/April
[Studio] Johnson B., Skibo C., Young M. (2003),
Inside Microsoft Visual Studio .NET, Microsoft Press
[HalfReal] Juul J. (2005),
Half Real -- Video Games between Real Rules and Fictional Worlds, MIT Press
[Science] Kalawsky R.S. (1993),
The Science of Virtual Reality and Virtual Reality Environments, Addison-Wesley
[Kritik] Kant E. (1781),
Kritik der reinen Vernunft, Felix Meiner Verlag, ed. 1976
[Urteil] Kant E. (1781),
Kritik der Urteilskraft, Reclam, Stuttgart, 1976
[XSLT] Kay M. (2001),
XSLT Programmer\'s Reference, Wrox Press
[Economy] Kassel S., Schumann C-A. and Tittman C. (2007),
Intelligent Advertisement for E-Commerce, In Proc. EUROMEDIA 2007, Delft, Netherlands
[ACOI] Kersten M. L.,Nes M., Windhouwer M.A. (1998),
A feature database for multimedia objects, CWI Report INS-R9807, July 1998
[ Kientzle T. (1998),
A programmer Guide to Sound, Addison-Wesley
[Numerical] Kimmel R. (2004),
Numerical Geometry of Images -- Theory, Algorithms, and Applications, Springer,
[Magic] Klabbers J.H.G. (2006),
The Magic Circle: principles of Gaming and Simulation, Sense Publishers
[Systems] Koegel Buford J.F., ed. (1994),
Multimedia Systems, Addison-Wesley, 1994
[Time] Koenen R. (1999),
MPEG-4 - Multimedia for our time, IEEE Spectrum, Vol. 36, No. 2,February 1999, pp. 26-33.
[MPEG-4] Koenen R. (ed.) (2000),
Coding of moving pictures and audio, ISO/ITEC JTC1/SC29/WG11 N3747
[Fundamental] Körner S. (1973),
Fundamental Questions of Philosophy, Penguin Books
[Semiotics] Kress G. and van Leeuwen T. (1996),
Reading Images: The Grammar of Visual Design, Routledge
[Samurai] Krikke J. (2003),
Samurai Romanesque, and the battle for mobile cyberspace, IEEE Computer, Jan/Febr 2003
[Quaternions] Kuipers J.B. (1999),
Quaternions and Rotation Sequences -- A Primer with Applications to Orbits, Aerospace, and Virtual Reality, Princeton University Press
[ACL] Labrou L., Finin T. and Peng Y. (1999),
The current landscape of Agent Communication Languages, IEEE Intelligent Systems, Vol. 14, no. 2, 1999
[Tricks] LaMothe A. (2002),
Tricks of the Windows Game Programming Gurus, Sams, 2nd edn.,
[GameMathematics] Lengyel E. (2004),
Mathematics for 3D Game Programming & Computer Graphics , Charles River Media
[Fundamentals] Li Z-N. and Drew M.S. (2004),
Fundamentals of Multimedia, Prentice Hall, Pearson
[Game] Luna F. (2003),
Introduction to 3D Game Programming with DirectX 9.0, Wordware Publishing Inc.
[Street] Manco T. (2004),
Street logos, Thames & Hudson
[GPU] McCool M. and Du Toit S. (2004),
Metaprogramming GPUs with Sh, A K Peters, Ltd.
[Effects] McCuskey M. (2002),
Special Effects Game Programming with DirectX, Premier Press
[Hypertext] McKnight C., Dillon A. and Richardson J. (1991),
Hypertext in Context, Cambridge University Press
[Meldex] McNab R.J., Smith L.A., Bainbridge D. and Witten I.M. (1997),
The New Zealand Digital Library MELody inDEX, D-Lib Magazine, May 1997
[Policy] McQuail D. and Siune K. (1989),
Media Policy -- Convergence, Concentration & Commerce, Euromedia Research Group, Sage Publications
[MIDI] Messick P. (1998),
Maximum MIDI -- Music Applications in C++, Manning
[Grammar] Moortgat, M. (1996),
Categorial Type Logics, Handbook of Logic and Language, J. van Benthem and A. ter Meulen (eds.), Elsevier Amsterdam (1996), pp. 93-177
[Categorial] Moortgat, M. (2002),
Categorial Grammar and Formal Semantics, Encyclopedia of Cognitive Science, Nature Publishing Group, Macmillan Publishers Ltd.
[Compare] Mongeau M. and Sankoff D. (1990),
Comparison of musical sequences, Computers and the Humanities 24, pp. 161-175, 1990
[JavaGame] Morrison M. (2005),
Beginning Mobile Phone Game Programming, Sams Publishing
[Mozart] Mozart W.A. (1781),
Twinkle,twinkle little star (Oh! vous dirai-je, mamam), K300, 1781-82
[Social] Munro A., Höök, K. and Benyon D.R. (1999),
Footprints in the Snow, In: Social Navigation of Information Space, Springer
[Hamlet] Murray J. (1997),
Hamlet on the Holodeck -- The future of narrative in Cyberspace , Free Press
[Digital] Negroponte N. (1995),
Being Digital, New Riders
[Nelson] Nelson T. (1980),
Literary Machines, Mindfull Press
[VisualComputing] Nielsen F. (2005),
Visual Computing: Geometry, Graphics, And Vision, Charles River Media
[Nielsen] Nielsen J. and Mack R.L., eds. (1994) ,
Usability inspection methods, Wiley
[Norman] Norman D.A. (2004),
Emotional Design, Basic Books
[Privacy] Oard D.W., Leuski A. and Stubblebine S. (2006),
Protecting the Privacy of Observable Behavior in Distributed Recommender Systems, In: Proc. SIGIR 2003
[AMICO] Obrenovic Z. and Gasevic D. (2007),
Open-Source Software: All you do is put it together, IEEE Software, Vol. 24, No. 5, Sept/Oct 2007
[AMICOCALC] Obrenovic Z. and Eliens A. (2007),
End-User Development with Heterogeneous Software Services -- Using AMICO:CALC within Spreadsheets, submitted to: ICSE 2008
[Selection] Pachet F., Roy P. and Cazaly D. (1999),
A combinatorial approach to content-based music selection, Proc. IEEE Int. Conf. on Multimedia Systems, Firenze
[Music] Ossenbruggen J. van and Eliens A. (1994),
Music in Time-based Hypermedia, Proc. European Conference on Hypermedia Technology 1994, pp. 224-227
[Hypermedia] Ossenbruggen J. van (2001),
Processing Structured Hypermedia -- A matter of style, Ph.D. Thesis, Free University, 2001
[CWI] Ossenbruggen J. van, Geurts J., Cornelissen F., Rutledge L., Hardman L. (2001),
Towards Second and Third Generation Web-Based Multimedia, In Proc. of The Tenth International World Wide Web Conference pp. 479-488, May 1-5, 2001, Hong Kong
[Nympheas] Patin S. (2006),
Regards sur les Nympheas (in french), Editions de la reunion des musees nationaux.
[Video] Pesce M. (2003),
Programming Microsoft DirectShow for digital video and television, Microsoft Press
[Affective] Picard R.W. (1998),
Affective Computing, MIT Press
[Preece] Preece J., Rogers Y., Sharp H., Benyon D., Holland S., Carey T. (1994),
Human-Computer Interaction, Addison-Wesley
[Design] Purvis A.W. and Le Coutre M.F. (2003),
Graphic Design 20th Century, BIS Publishers
[Human] Ratner P. (2003),
3-D Human Modeling and Animation, Wiley, 2nd edn.
[Guide] Riel C. van, Eliens A., Wang Y (2006),
Exploration and guidance in media-rich information spaces: the implementation and realization of guided tours in digital dossiers, accepted for: International Conference on Multidisciplinay Information Sciences and Technologies (InSciT2006) October, 25-28th 2006, Merida, Spain www.instac.es/inscit2006
[ConceptMaps] Riel C. van, Wang Y. and Eliens A. (2006b) ,
Concept map as visual interface in 3D Digital Dossiers: implementation and realization of nthe Music Dossier, CMC2006, Costa Rica, Sept 5-8 2006
[Signs] Rooijen P. van, ed. (2003),
Signs & Symbols, Pepin Press
[World] Rosenblum L. and Macedonia M., eds. (2002),
Projects in VR -- From Urban Terrain Models to Visible Cities, IEEE Computer Graphics and Applications, 22:4, pp. 10-15
[Tangible] Rosenblum L. and Macedonia M., eds. (2005),
Projects in VR -- Tangible Photorealistic Virtual Museum, IEEE Computer Graphics and Applications, 25:1, pp. 15-17
[Context] Rutledge L., van Ballegooij A., Eliens A. (2000),
Virtual Context - relating paintings to their subject, Culture Track of WWW9 in Amsterdam, The Netherlands, May 16th, 2000
[Reusable] Ruttkay M., Huang Z. and Eliens A. (2003),
Reusable gestures for interactive web agents, Proc. Intelligent Virtual Agents 2003, Irsee, September 15-17, 2003 J.G. Carbonell and J.Siekmann (eds.), LNAI 2792, Springer, pp. 80-87
[Conductor] Ruttkay Z., Huang Z. and Eliens A.,
The Conductor: Gestures for Embodied Agents with Logic Programming, in: Recent Advances in Constraints K.R. Apt, F. Fages, F. Rossi, P. Szeredi and J. Vancza (eds.) LNAI 3010, Springer 2004
[SecondLife] Rymaszewski M., Au W.J., Wallace M., Winters C., Ondrejka C., Batstone-Cunningham B. (2007),
Second Life -- the official guide, Wiley
[Algorithms] Sanchez-Crespo Dalmau D. (2004),
Core Techniques and Algorithms in Game Programming , New Riders
[3DEffects] Santiago D. (2005),
Creating 3D Effects for Film, TV and Games, Thomson
[NET] Santos Lobao A. and Hatton E. (2003),
.NET Game Programming with DirectX 9.0, The Authors Press
[Imagination] Sartre J.P. (1936),
L'imagination, Presse Universitaire de France, ed. 8, 1971
[Aesthetics] Saw R.L. (1971),
Aesthetics -- An Introduction, Anchor Books, New York
[Erziehung] Schiller F. (1977),
Uber die ästhetische erziehung des Menschen, Reclam, Stuttgart
[Features] Schmidt, A.R. Windhouwer M.A., Kersten M.L. (1999),
Indexing real-world data using semi-structured documents, CWI Report INS-R9902
[ Schneider P.J. and Eberly D.H. (2003),
Geometric Tools for Computer Graphics, Morgan Kaufmann Publishers
[Gadgets] Schonhage S.P.C., van Ballegooij A. and Eliens A. (2000),
3D Gadgets for Business Process Visualization: a case study, Proc. Int. Web3D/VRML Conference - 2000, Monterey CA, Febr 2000, ACM Press pp. 131-138
[GameDesign] Schuytema P. (2007),
Game Design: A Practical Approach, Charles River Media
[Beyond] Selfridge-Field E., ed. (1997),
Beyond MIDI -- The Handbook of Musical Codes , MIT Press 1997
[Concepts] Selfridge-Field E. (1998),
Conceptual and Representational Issues in Melodic Comparison, In [_Similarity], pp. 3-64
[Hybrid] Setten M. van (2005),
Supporting People in Finding Information -- Hybrid recommender Systems and Goal-based Structuring, Ph.D. Thesis, Telematica Institute Netherlands
[VR] Sherman W.R. and Craig A.B. (2003),
Understanding Virtual Reality -- Interface, Application and Design, Morgan Kaufmann Publishers
[Ultimate] Sherrod A. (2006),
Ultimate Game Programming with DirectX, Charles River Media
[User] Shneidermann B. (1997),
Designing the user interface -- strategies for effective human-computer interaction, Addison-Wesley (3rd edn)
[Laptop] Shneiderman B (2003),
Leonardo's Laptop, Mit Press
[PanoramaWeb] Si Yin and Eliens A. (2007),
PANORAMA -- A Rich VRML Application Platform For Online Gaming, Workshop Web 3D Games, Web3D Symposium 07, Perugia Italy, April 15-18
[VE] Singhal S. and Zyda M. (1999),
Networked Virtual Environments, Addison-Wesley, 1999
[Terrain] Snook G. (2003),
Real-time 3D terrain engines using C++ and DirectX 9, Charles Riven Media
[Computer] Spalter A.M. (1999),
The Computer in the Visual Arts, Addison-Wesley
[Snowcrash] Stephenson N. (1992),
Snowcrash,
[ShaderArt] St-Laurent S. (2004),
Shaders for Game Programmers and Artists, Thomson
[MMDBMS] Subrahmanian V.S. (1998),
Principles of Multimedia Databases, Morgan Kaufmann
[DEEP] Sullivan A. (2005),
3-Deep, IEEE Spectrum 42-4, April 2005, pp. 22-27
[Jinni] Tarau P. (1999),
Jinni: Intelligent Mobile Agent Programming at the Intersection of Java and Prolog, Proc. of PAAM\'99, London, UK, April, see also http://www.binnetcorp.com/Jinni
[Meter] Temperley D. and Sleator D. (1999) ,
Modeling Meter and Harmony: A Preference-Rule Approach, Computer Music Journal 23(1), 1999, pp. 10-27
[CodeArt] Trogemann G. & Viehoff J. (2004),
CodeArt -- Eine elementare Einfhrung in die Programmierung als kunstlerische Praktik, Springer
[Codecs] Vasudev B. and Li W. (1997),
Memory management: Codecs, In [_Handbook], pp. 237-278
[Making] Vaughan T. (1998),
Multimedia -- Making It Work, Osborne/McGraw-Hill, 4th edn
[Essential] Vince J. (1998),
Essential Virtual Reality, Springer
[Web] Visser C. and Eliens A. (2000),
A High-Level Symbolic Language for Distributed Web Programming, Internet Computing 2000, June 26-29, Las Vegas
[Meaning] Vyas, D., van der Veer, G. C. (2006),
Experience as Meaning: Some Underlying Concepts and Implications for Design. , Proceedings of 13th European Conference on Cognitive Ergonomics. ACM Press: NY, (2006), 81-91.
[Affordance] Vyas, D., Chisalita, C., van der Veer, G. C. (2006),
Affordance in Interaction. , Proc. of 13th European Conference on Cognitive Ergonomics. ACM Press: NY, (2006), 92-99.
[SMIL] W3C SMIL Working Group (2001),
Synchronized Multimedia Integration Language (SMIL 2.0), W3C Recommendation 07 August 2001
[Web3D-2002] Wagner W. and Beitler M. (eds), 2002,
Proc. Int. Web3D Symposium, ACM Press
[Advance] Walsh P. (2003),
Advanced 3D Programming with DirectX 9.0, Wordware Publishing Inc.
[Present] Wang Y., Eliens A., van Riel C. (2006),
Content-oriented presentation and personalized interface of cultural heritage in digital dossiers, accepted for: International Conference on Multidisciplinay Information Sciences and Technologies (InSciT2006) October, 25-28th 2006, Merida, Spain www.instac.es/inscit2006
[Games] Watt A. and Policarpo F. (2001),
3D Games -- Real-time Rendering and Software Technology, Addison-Wesley
[Web3D] Web3D Consortium (2001),
VRML97 Standard, \\ http://www.web3d.org/fs_specifications.htm
[Design] Weishar P. (1998),
Digital Space -- Designing Virtual Environments, McGraw-Hill
[Chatting] Welie M. and and Eliens A. (1996),
Chatting on the Web, Proc. ERCIM W4G Workshop on CSCW and the Web, 7-9 febr 1996 GMD St Augustin, Germany, 1996
[Euterpe] Welie M. van, G.C. van der Veer G.C., and Eliens A. (1998) ,
Euterpe - Tool support for analyzing cooperative environments, Proceedings of the Ninth European Conference on Cognitive Ergonomics , August 24-26, 1998, University of Limerick, Ireland, pp. 25-30
[Beauties] Wiedermann J., ed. (2002),
Digital Beauties, Taschen Verlag
[Animovie] Wiedermann J., ed. (2004),
Animation now, Taschen Verlag
[InformationArts] Wilson S. (2002),
Information Arts -- Intersections of Art, Science and Technology, The MIT Press
[Flow] Yellowlees D. and Hargadon A. (2000),
The pleasure principle: immersion, engagement, flow, Proc. of the 11th ACM Conf. on Hypertext and Hypermedia, May 2000,San Antonio, pp. 153-160
[DeepTime] Zielinski S. (2006),
Deep Time of the Media -- Towards an archaeology of Hearing and Seeing by Technical Means, The MIT Press
[AI] Zimmermann D.,
Modeling Musical Structures, Aims, Limitations and the Artist\'s Involvement, Proc. Constraints techniques for artistic applications, Workshop at ECAI\'98 25th August, 1998, Brighton, UK, ftp://ftp.csl.sony.fr/pub/pachet/workshopEcai
\begin{theindex} \item Adams (2003), \hyperpage{x}, \hyperpage{111} \item Alberti (1435), \hyperpage{223} \item Anders (1999), \hyperpage{195} \item Arnaud \& Barnes (2006), \hyperpage{110} \item Arnheim (1957), \hyperpage{x}, \hyperpage{213}, \hyperpage{249}, \hyperpage{291, 292}, \hyperpage{299} \item Atanasova et al. (2007), \hyperpage{171} \item Auster (2004), \hyperpage{155} \indexspace \item Baeza-Yates and Ribeiro-Neto (1999), \hyperpage{x}, \hyperpage{79}, \hyperpage{117, 118}, \hyperpage{128}, \hyperpage{131}, \hyperpage{159}, \hyperpage{162, 163} \item Ballegooij and Eliëns (2001), \hyperpage{x}, \hyperpage{154} \item Baxter et al. (2004), \hyperpage{96} \item Benjamin (1936), \hyperpage{222} \item Betsky (2004), \hyperpage{50}, \hyperpage{175} \item Bj\"{o}rk \& Holopainen (2005), \hyperpage{282}, \hyperpage{287, 288}, \hyperpage{305} \item Bocconi (2006), \hyperpage{86} \item Bolter and Grusin (2000), \hyperpage{x}, \hyperpage{40, 41}, \hyperpage{50}, \hyperpage{195}, \hyperpage{199}, \hyperpage{223}, \hyperpage{249}, \hyperpage{305} \item Briggs and Burke (2001), \hyperpage{9}, \hyperpage{14}, \hyperpage{19, 20}, \hyperpage{50} \item Broll (1996), \hyperpage{329} \item Broll et. al (2001), \hyperpage{80} \item Bruner (1972), \hyperpage{4}, \hyperpage{295}, \hyperpage{310} \item Burger (1981), \hyperpage{50}, \hyperpage{130}, \hyperpage{250} \item Bush (1995), \hyperpage{33} \indexspace \item CACM (2001), \hyperpage{25} \item Cesa-Bianchi and Lugosi (2006), \hyperpage{151} \item Chang and Costabile (1997), \hyperpage{x}, \hyperpage{28}, \hyperpage{33} \item Chapman and Chapman (2004a), \hyperpage{x} \item Chapman and Chapman (2004b), \hyperpage{x} \indexspace \item Daskalova \& Atanasova (2007), \hyperpage{170} \item Davenport (2000), \hyperpage{x}, \hyperpage{5} \item Davison (2001), \hyperpage{329} \item Dewey (1931), \hyperpage{291}, \hyperpage{302, 303} \item Diehl 1973, \hyperpage{50} \item Dodge and Kitchin (2000), \hyperpage{152} \item Dodge and Kitchin (2002), \hyperpage{144} \item Dormann et al. (2007), \hyperpage{301} \indexspace \item Eco (1994), \hyperpage{50} \item Eliëns (1979), \hyperpage{223}, \hyperpage{302} \item Eliëns (1988), \hyperpage{220}, \hyperpage{308} \item Eliëns (1992), \hyperpage{323} \item Eliëns (2000), \hyperpage{37}, \hyperpage{48}, \hyperpage{54}, \hyperpage{79}, \hyperpage{99}, \hyperpage{148}, \hyperpage{168, 169}, \hyperpage{174}, \hyperpage{182}, \hyperpage{240}, \hyperpage{324} \item Eliëns \& Chang (2007), \hyperpage{22}, \hyperpage{82}, \hyperpage{250}, \hyperpage{263}, \hyperpage{287} \item Eliëns \& Vyas (2007), \hyperpage{129}, \hyperpage{221} \item Eliëns \& Wang (2007), \hyperpage{151}, \hyperpage{171} \item Eliëns and Bhikharie (2006), \hyperpage{109}, \hyperpage{263} \item Eliëns et al. (2002), \hyperpage{323} \item Eliëns et al. (2007), \hyperpage{21}, \hyperpage{23}, \hyperpage{25}, \hyperpage{108}, \hyperpage{195, 196} \item Eliëns et al. (2007b), \hyperpage{82}, \hyperpage{109}, \hyperpage{282} \item Eliëns et al. (2007c), \hyperpage{20, 21}, \hyperpage{45}, \hyperpage{251} \item Eliëns {\em et al.} (1997), \hyperpage{37} \item Engel (2004a), \hyperpage{96} \item Engel (2004b), \hyperpage{96} \item Engel (2005), \hyperpage{96} \item Engelbart (1963), \hyperpage{33} \item Entanaclaz (2003), \hyperpage{217} \indexspace \item Faulkner (2000), \hyperpage{212} \item Fay et al. (2004), \hyperpage{111} \item Fernando and Kilgard (2003), \hyperpage{x}, \hyperpage{90, 91}, \hyperpage{96} \item Fluckiger (1995), \hyperpage{x}, \hyperpage{164, 165} \item Forman and Saint John (2000), \hyperpage{x} \item Freud (1958), \hyperpage{302, 303} \item Fuhr et al. (1998), \hyperpage{119} \indexspace \item Gee (2003), \hyperpage{83}, \hyperpage{197}, \hyperpage{284} \item Gonzales and Wintz (1987), \hyperpage{131} \item Grau (2003), \hyperpage{x}, \hyperpage{4}, \hyperpage{22}, \hyperpage{129}, \hyperpage{223}, \hyperpage{276, 277}, \hyperpage{279, 280}, \hyperpage{299, 300}, \hyperpage{303} \indexspace \item Halln\"{a}ss and Redstr\"{o}m (2002), \hyperpage{222}, \hyperpage{290, 291}, \hyperpage{304} \item Hardman et al. (1994), \hyperpage{37} \item Harel (1984), \hyperpage{193} \item Hawkins (2005), \hyperpage{x} \item Hegel (1807), \hyperpage{302, 303} \item Heidegger (1927), \hyperpage{291} \item Hewlett and Selfridge-field (1998), \hyperpage{135}, \hyperpage{206} \item Hohl (1971), \hyperpage{178} \item Huang et al. (2000), \hyperpage{191} \item Huang et al. (2002), \hyperpage{x}, \hyperpage{190} \item Hummelen and Sill\'e (1999), \hyperpage{148}, \hyperpage{155, 156} \item Huron (1997), \hyperpage{138} \indexspace \item Jain (2000), \hyperpage{x}, \hyperpage{6} \item Jenkins (2006), \hyperpage{86}, \hyperpage{196} \item Johnson et al. (2002), \hyperpage{193} \item Juul (2005), \hyperpage{x}, \hyperpage{22}, \hyperpage{250}, \hyperpage{286} \indexspace \item K\"{o}rner (1973), \hyperpage{130} \item Kant (1781), \hyperpage{130}, \hyperpage{302, 303} \item Kassel et al. (2007), \hyperpage{170} \item Kay (2001), \hyperpage{32}, \hyperpage{322} \item Kersten et al. (1998), \hyperpage{x}, \hyperpage{145} \item Klabbers (2006), \hyperpage{x}, \hyperpage{33, 34}, \hyperpage{82}, \hyperpage{206, 207}, \hyperpage{284} \item Koenen (1999), \hyperpage{62} \item Koenen (2000), \hyperpage{x}, \hyperpage{61}, \hyperpage{63}, \hyperpage{87} \item Kress and van Leeuwen (1996), \hyperpage{x}, \hyperpage{223}, \hyperpage{249, 250}, \hyperpage{258}, \hyperpage{274--278}, \hyperpage{293}, \hyperpage{296--298}, \hyperpage{302, 303} \item Krikke (2003), \hyperpage{262} \indexspace \item Li and Drew (2004), \hyperpage{87}, \hyperpage{175} \item Luna (2003), \hyperpage{x}, \hyperpage{111}, \hyperpage{284} \indexspace \item Manco (2004), \hyperpage{224} \item McCuskey (2002), \hyperpage{x}, \hyperpage{252} \item McKnight et al. (1991), \hyperpage{34} \item McNab et al. (1997), \hyperpage{x}, \hyperpage{137, 138}, \hyperpage{149} \item Mongeau and Sankoff (1990), \hyperpage{135}, \hyperpage{137} \item Morrison (2005), \hyperpage{25} \item Mozart (1787), \hyperpage{135}, \hyperpage{149} \item Murray (1997), \hyperpage{42--44} \indexspace \item Negroponte (1995), \hyperpage{3} \item Nelson (1980), \hyperpage{33} \item Norman (2004), \hyperpage{209}, \hyperpage{212} \indexspace \item Oard et al. (2006), \hyperpage{171, 172} \item Obrenovic \& Eliëns (2007), \hyperpage{271} \item Obrenovic \& Gasevic (2007), \hyperpage{272} \item Ossenbruggen (2001), \hyperpage{x}, \hyperpage{37, 38} \item Ossenbruggen \& Eliëns (1994), \hyperpage{37} \item Ossenbruggen et. al. (2001), \hyperpage{75} \indexspace \item Pesce (2003), \hyperpage{99}, \hyperpage{143} \item Picard (1998), \hyperpage{81} \item Preece et al. (1994), \hyperpage{233}, \hyperpage{239} \indexspace \item Rosenblum and Macedonia (2002), \hyperpage{13}, \hyperpage{25} \item Rosenblum and Macedonia (2005), \hyperpage{232}, \hyperpage{252} \item Rymaszweski et al. (2007), \hyperpage{25}, \hyperpage{195} \indexspace \item Sanchez-Crespo Dalmau (2004), \hyperpage{261, 262} \item Santos Lobao and Hatton (2003), \hyperpage{25} \item Sartre (1936), \hyperpage{304} \item Saw (1971), \hyperpage{130}, \hyperpage{223}, \hyperpage{304} \item Schiller (1977), \hyperpage{303} \item Schmidt et al. (1999), \hyperpage{146}, \hyperpage{174} \item Schuytema (2007), \hyperpage{259} \item Selfridge (1997), \hyperpage{150} \item Selfridge (1998), \hyperpage{135, 136} \item Sherman and Craig (2003), \hyperpage{198}, \hyperpage{208} \item Sherrod (2006), \hyperpage{x}, \hyperpage{258} \item Shneiderman (1997), \hyperpage{38, 39}, \hyperpage{174} \item Shneiderman (2003), \hyperpage{25} \item Si \& Eliëns (2007), \hyperpage{129}, \hyperpage{251}, \hyperpage{253} \item Singhal and Zyda (1999), \hyperpage{166, 167} \item St-Laurent (2004), \hyperpage{96} \item Subrahmanian (1998), \hyperpage{x}, \hyperpage{115, 116}, \hyperpage{119, 120}, \hyperpage{122--127}, \hyperpage{133, 134}, \hyperpage{139--142}, \hyperpage{158}, \hyperpage{160, 161}, \hyperpage{165} \item Sullivan (2005), \hyperpage{101}, \hyperpage{112} \indexspace \item Tarau (1999), \hyperpage{329} \item Temperley and Sleator (1999), \hyperpage{138} \item Trogemann \& Viehoff (2004), \hyperpage{107} \indexspace \item van Rooijen (2003), \hyperpage{2}, \hyperpage{25}, \hyperpage{50}, \hyperpage{54}, \hyperpage{87}, \hyperpage{112}, \hyperpage{114}, \hyperpage{132}, \hyperpage{156}, \hyperpage{176}, \hyperpage{178}, \hyperpage{199}, \hyperpage{225}, \hyperpage{253}, \hyperpage{256}, \hyperpage{284}, \hyperpage{306} \item van Setten (2005), \hyperpage{151}, \hyperpage{174} \item Vasudev and Li (1997), \hyperpage{x}, \hyperpage{56--58} \item Visser and Eliëns (2000), \hyperpage{x}, \hyperpage{46} \item Vyas and van der Veer (2006) , \hyperpage{290} \indexspace \item W3C (2001), \hyperpage{61} \item Weishar (1998), \hyperpage{50}, \hyperpage{155} \item Welie et al. (1998), \hyperpage{239}, \hyperpage{253} \item Wiedermann (2002), \hyperpage{199}, \hyperpage{284} \item Wiedermann (2004), \hyperpage{87}, \hyperpage{225} \item Wilson (2002), \hyperpage{4}, \hyperpage{23, 24}, \hyperpage{279--281} \indexspace \item Zielinski (2006), \hyperpage{x}, \hyperpage{4, 5}, \hyperpage{22}, \hyperpage{129}, \hyperpage{270}, \hyperpage{280} \item Zimmerman (1998), \hyperpage{146} \end{theindex} \mbox{}\newpage

topical media & game development

\hspace{-0.35cm}

\vspace{0.0cm} This book provides a concise and comprehensive introduction to multimedia. It arose out of the need for material with a strong academic compontent, that is (simply) material related to scientific research. Indeed, studying multimedia is not (only) fun. Compare it with obtaining a driver license. Before you are allowed to drive on the higway, you have to take a theory exam. So why not take such an exam before entering the multimedia circus. Don't complain, and take the exam. After all it makes you aware of the rules governing the (broadband) digital highway. The book and accompanying material is available at www.cs.vu.nl/~eliens/media

\hspace{3cm}

\parbox{5cm}{© \hspace{0.2cm}Æliens, Amsterdam (2002)}