The next chapter
documents the eight stages of this research project. These have been divided
into two major sections: the preliminary stages during which the research
topic and research questions were developed and; the main stages of the project
in which the final works were developed.
Preliminary Stage
1)
Moving from online media to a medium of sensations
2)
Methods for developing ideas with time-based media
3)
A move to interactive images
4)
Interactive spaces in place of interactive images
5)
Experimentation in physical spaces
6)
The works as a whole
7)
The sound re-mix
8)
The final versions
This material
is summarised from personal journals. The accompanying hypertext version of
the exegesis on the CD-rom contains video documentation of the installations
in progress.
When this research
project was commenced, the main area of my practice was collaborative internet-based
work. The project was started with the working title ‘Order and Chaos
in a digital world’. The intent of this topic was to explore the desire
to control and order information and its fluid uncontrollable nature when
approaching information overload. The plan was to continue working with web-based
media and to integrate streaming media. One of the first activities was attending
the Polar Circuit 2 media arts workshop in Finland [1] . The purpose of attending this event was to be
influenced and to open up new directions. One of the last projects I had completed
prior to starting this research was a collaborative theatre production and
I felt that theatre had many possibilities for and influences on digital media.
I took part in the Kalevala Moo workshop which was a text-based multi-user
environment and performance loosely based around the Finnish myths of the
Kalevala [2] . I was able to attend
workshops by Ken Gregory [3]
and Daniel Jolliffe on sensors
and electronics for artists which created new possibilities for working with
interactive installations.
After attending
this event I spent five days at ZKM Centre for Art and Media [4] in Karlsrühe Germany, researching interactive
artworks in the collections, watching how people moved around and interacted
with the works and viewing ZKM’s extensive collection of video art.
Over the previous five years my practice had moved away from using video;
this time was an opportunity to once again engage with the medium.
On returning to
Hobart I was planning two works, both interactive installations. One was a
video projection where the closer the viewer moved to the screen the more
chaotic the imagery on the screen became. The other was an installation where
there would be three rock-like objects in different parts of the world. When
viewers put their hands over the rock an image of their hand would appear
on the rock in front of them as well as on the rocks in the other parts of
the world. This would allow for a new class of non-verbal tele-presence between
people. Time was spent researching the possible technology to achieve these
works, eg how to sense distance and use of machine vision systems to be able
to realise the rock piece.
New directions
and possibilities had been opened up, important international connections
had been made and a number of international works had been evaluated. The
possibilities of having control over alternative means of sensing user interactions
suggested new directions. Many significant artworks in the area had been viewed
and some of these later informed the central argument of the research. Studio
practice was minimal at this stage. The reason for this became a major area
of my research project, that is, what are the methods for a solo artist to
develop interactive video installations? There were ideas but not concrete
spaces, technologies or research rationales for evaluating and developing
these works. What was needed was a method of working that developed out of
studio-based experimentation, instead of developing ideas from the possibilities
of applying specific technologies.
This stage was
marked by the development of a new methodology. The planning and development
of interactive artworks is a complex activity. One of the main hurdles was
the initial phase when there might be a clear conceptual idea of the finished
project, but there might also be many technical hurdles to overcome before
a prototype could be developed. This can be a time-consuming and convoluted
phase. The possible production models that exist in software, film or game
development, or other areas such as set design or interior design, do not
transfer readily to a comparatively low-budget studio-based research project.
Parts of the process can be related to film production, others to game development
and others still to a more traditional studio practice. To spend time away
from some these complex problems a number of video sketches were developed.
The intent of these sketches was to work with a non-narrative moving image
and to develop a number of possibilities.
The first of these
was an attempt to explore the concept of a sudden fracture in a rhythm.
20. Robin
Petterd, Hit, 1999, video loop.
This evolved into
a video loop consisting of a red stick-like image seeming to hit the surface
of the screen in a tight binary rhythmical way, like a metronome. Then a large
smashing sound is heard and the screen goes black and. The sequence then repeats
itself. The next step could have been to include the smashing sections at
random times. However at this time the focus was not on using interactive
media. The concerns I had with the video loop were:
·
the binary nature of the hit and smash was too literal;
·
it became a narrative and part of the intent was to start to explore the notion
of non-narrative moving images.
The next sketch
was a more subtle way of working with tension and the relationship between
two images and sounds within a frame, instead of across time.
21. Robin
Petterd, Fish loop, 1999, video loop.
This was a video
loop of a fish swimming around the frame and at the centre bottom an image
of a foot stepping in and out of the frame. The image of the fish is fluid
and smooth, while the sound associated with the foot provides contrast with
a mechanical organised sound.
The next development
was an interactive interpretation of the sketch. This was attempted by allowing
the user to alter the speed and the rhythm of the two different sections of
the image by moving the foot image left to right and clicking on the fish
section. This was completed with online streaming media [5] , using Apple QuickTime wired sprites. The video
and sound for both of these was edited using After Effects [6] .
At this stage
a preliminary draft of the research proposal was prepared. It was:
The project will explore the cyclic harmonies that exist in the
relationships between what is structured and highly controlled and what
is fluid and changeable. As part of these cyclic harmonies the act of contemplation
will be introduced in the process of viewing and interacting with new media.
The project will be completed using emerging interactive video technologies
and will use images and sounds of repetitive actions, movements and environmental
images and sound.
At the completion
of this stage the contrast of elements over time, the use of repetition and
of violent interruptions to the flow of events had been adopted. These formal
aspects were more fully developed in later stages of the project. The use
of After Effects allowed the imagery and audio to be highly malleable, allowing
for an almost painterly fashion of working. The linear version of the fish
loop was more successful than the interactive version. The interactivity did
not have a clear interface and lacked a strong linkage to the image. This
type of interface may have been more successful if it had been combined with
more media elements and the users had greater control over their effect on
the media. The type of interactions that were being explored were simple and
subtle and would have been more successful as physical interfaces.
The imagery and
sounds were leading into a method of working that focused on the expressive
nature of the moving image over the importance of narrative. The lack of success
in transferring this to an interactive vision highlighted the issue that the
methods of planning and developing a project define the outcome of the work.
In the later stages of the research the imagery, sounds, spaces and interaction
were worked on in conjunction, instead of attempting to place interactive
or spatial interfaces onto the imagery or sounds after they had been finalised.
The use of video for its expressive sensory qualities developed into the question
of ‘How can video installations be used to evoke sensory experience?’
The first draft
of the research proposal began to define the intent of my research. There
was confusion between attempting to explore a corporeal sensation and the
aim of providing a contemplative experience. While this started to develop
experience design as a core part of the research area, it did not define what
those experiences should be and how they were going to be evoked. The plan
to explore contemplative moments evolved into an interest in experiences that
have a powerful effect on the viewer.
At this stage
I needed to return to the possibilities of physical interfaces as the workshops
completed at Polar Circuit were only introductions. I wanted a system that
would be simple to set up and would give me the ability to work with aspects
of the viewer’s motion and positions. One of the common ways of achieving
this at this stage was with ultrasonic sensors which, for someone new to the
area of electronics, are relatively complex to construct. The resulting sensing
technology used was a machine vision
[7] Xtra [8]
for Macromedia Director [9] developed by Danny Rozin
[10] an artist/programmer working at New York University.
I then attended
Siggraph [11] in Los Angeles,
where part of the Archiving Imagination project was being shown, and Invenção
in Brazil where I presented a paper
[12] . At Siggraph there were a number of interactive artworks
exhibited and the artists were often available and willing to talk about how
they had realised their works. I discovered that most artists working with
motion sensing were using machine vision and they were solving the lack of
processor speed by networking two or more computers together. Another significant
observation was that the works which used simpler interactivity were often
the most successful at an exhibition-style event like Siggraph. When I returned
to Hobart, some time was spent working on building a flexible system to network
computers together using the multi-user compatibilities [13] of Director.
Each computer was able to send messages to other machines on the network.
This system was more complex than was needed at that stage but allowed for
flexibility. One computer was used for the machine vision system and another
for the playback and processing of video. This message passing system was
later used to prototype some of the multi screen systems. The control software
which integrates all of the final installations is based on these scripts.
The next installation
work I developed went through a number of variations. The three most significant
stages are outlined below.
22. Robin
Petterd, Untitled, 1999, interactive video installation, dimensions
variable.
The starting point
emerged from the previous video sketches. The initial
idea was to contrast two images to explore the relationship between a fluid
ephemeral steam image and rhythmic image of someone kneading dough. The layout
of the installation consisted of a rear-projected image floating in the gallery
with a pool of light in front of it. The movement of a viewer in the gallery
triggered the fading-in of the kneading image. This was not pre‑computed,
which meant the fading in and out was computed in real-time. The focus was
on the types of movements people naturally do in an art gallery: walking in
and stopping, observing and then walking out. One of the problems was that
it was too complex and people did not perceive that they were influencing
the projected image. The work need to be more focused, both in terms
of the images and sounds and in the character of the interactions.
23. Robin
Petterd, Untitled, 1999, interactive video installation, dimensions
variable.
The second version
focused on the movement of the imagery reflecting the movement of the viewer,
ie when the viewer moved the imagery moved. This opened up new possibilities
for the research. There was a strong, direct relationship between what the
viewer did and how the imagery responded. Some viewers tried to be as still
as possible, to control it. Others were very active so they could watch the
video in full.
Different aspects
of this interaction explored were:
·
If the viewer moved towards the left, the hands moved towards the left, which
was a basic form of gesture recognition.
·
As the viewer moved more, the sound got louder.
·
Multiple images were trialled; these included those that reflected the viewer’s
position.
24. Robin
Petterd, Unsteady Motions, 2000, interactive video
installation, dimensions variable.
The next stage
applied the same interactive situation to a different type of imagery. An
image of a tree being blown around in a storm was utilised. It responded to
the viewer’s motion but did not respond to a lack of motion. What become
apparent was the interaction did not start a fluid dialogue between the viewer
and work. The interaction was only a simplistic binary reaction that did not
behave in the more complex manners that are possible with interactive technology.
In
the final version of this work the viewer paused in
front of the screen to contemplate the image. The imagery froze if movement
in the gallery stopped as in the other versions. As the viewer starts to ponder
the still image and the silence in contrast to the turbulence of the storm,
the imagery slowly faded to black. When the viewer moved the image erupted
again. The addition of the fade created a dialogue between the viewer and
the work. The focus of the work became the tension between the desire to contemplate
a moment in time and the states of change that are part of stillness. The
piece was titled Unsteady Motions. The opposite to this interaction
was trialled. The imagery moved when the viewer was static and the imagery
paused and faded out when movement was sensed.
This
installation started to move towards a simplified solution to the problems
of contemplation by action as outlined in the research proposal. The viewers
became conscious of their actions and these actions were reflected in a manner
where there was a direct response to both action and inaction.
A
number of concerns that I had with this installation could not be solved or
refined within its parameters.
·
The goal was to engage the viewer in a physical spatial way, but the layout
centred around an image-based viewing situation.
·
The viewers’ actions and experiences of the installation were broad:
some people lept around the room like a storm tossed tree, other people did
nothing.
·
Some people did not realise the work was interactive. It was necessary to
consider whether or not the intent of the research project meant that the
viewers needed to realise they were affecting the installation.
·
An interrelated set of imagery or types of imagery needed to be developed.
While imagery and sounds such as of hands or steam were interesting, the most
successful aspect of them was the movement and the fluid nature of the imagery.
The imagery of the natural environment held more possibilities for this project
than the figurative imagery of the body.
·
What happened when there was more than one person in the room? If the focus
of the project is on reflecting the movement of the viewer, when there is
more than one person in the room then the works cannot function as a personal
dialogue.
From
this stage the question of ‘How can unobtrusive interactivity
be used to enhance the viewer’s engagement with the works?’ developed.
The focus on the movement and being still as common viewer behaviour in a
gallery grew out of this stage and was based on observations of works at Siggraph
and viewers’ engagement with my installations.
The
next two installations were started as a reaction to the above observations
and have evolved into the works in the final exhibition. Around this
time I took an extended family holiday travelling and camping in the north-east
of Tasmania. The weather in late autumn can be changeable. At times the wind
speeds were around forty knots, it became hard to sleep at night, the trees
turned into roaring noise machines above the tent. While I was at Eddystone
Point, the wind died down enough for it to be safe to enter the water. Submersion
in water is unlike other activities. This time the contrast was heightened
because of the recent experience of the tent, wind and rain. The environment
was a soft blue-green blurred space, which induced an eerie calm. The strong
physical experience of this contrast helped to change the direction of the
research. The focus of the imagery and sounds was on aquatic environments
from this stage onwards.
25. Robin
Petterd, Wake – in progress, 2000, interactive
video installation, dimensions variable.
From
this, the first version of Wake was developed. An image of turbulent
shallow water mixing and hitting a rock edge is projected onto the floor of
a room. To the left is a small walkway across which viewers walk. When viewers
stop, the image stops moving. Then, as they pause for long enough to start
to become comfortable a larger wave comes towards the walkway. If they choose
to step off the walkway into the projections, the images and sounds erupt
into a flow of white water that slowly disappears or continues depending on
the viewer’s motion. There was a change from processing aspects of the
imagery in real-time to displaying and delivering different imagery to the
viewer as part of the interaction.
The intention
was to evoke the environment close to water: where water is rushing by, near
rocks or being on board a boat and watching the water slip by beneath. Time
was spent determining how to best use the machine vision system to sense what
was happening, eg where the viewer was and whether or not there was movement.
It was decided a more suitable technique to do this was to use a simple motion
sensor and contact sensors made from non‑conducting foam between plates
of metal mounted inside the walkway. These were interfaced to a computer via
a modified keyboard.
The installation
was complex and needed to be refined. While the walkway was meant to be a
clue to the interaction, viewers did not naturally walk across the board.
This was due to the scale of the walkway compared to the imagery and the space
around the projection. People’s reaction was to walk around the edge
of the image.
26. Robin
Petterd, Wake – in progress, 2000, interactive
video installation, dimensions variable.
In the second
stage of this installation the walkway was removed. When viewers moved the
imagery quickly flicked past the viewer. When they paused, the sound of a
roaring wave played; as it approached the maximum volume the imagery changed
to a soft blurred image and the sounds became gentler. When the viewer moved
again, the image of water rushing by returned.
Another new work
was started, which evolved into Dropping. This was the first experiment with using a waterproof
housing for a video camera. The casing was weighed down with lead weights
and lowered from the side of a boat. While the experience of being in the
water is normally a peaceful, calm experience, combining it with technology
that is fragile and could be destroyed by water means that shooting the imagery
for these installations has never been a relaxing contemplative experience
.
27. Robin
Petterd, Dropping – in progress, 2000, interactive
video installation, dimensions variable.
In the first version
of this work the image was projected into a corner of a room. When the viewer
moved, the image was of the surface of the water; when the viewer paused,
the image dropped down to the bottom of the water to show a barren muddy bottom
with almost no movement. When the viewer moved again, the imagery and sounds
erupted as the image and the camera viewpoint rushed to the surface.
The intent from
the early stages of Dropping was to evoke the experience of slowing down in the water
and the contrast between being at the surface, then diving down and rushing
back up to the surface. As a child I would attempt to sit at the bottom of
the pool for as long as I could and then when I could no longer hold my breath
I would push off the bottom of the pool with all my strength to burst back
to the surface of the water.
This work was
experimenting with using the projector as a source of light and shadow as
well as a source for forming the imagery. Viewers were meant to enter the
lighted space and become surrounded by the image and merge into the projection.
The optimal position was close up to the wall, inside the triangle of the
projection. The viewers did not approach the image very closely; they stood
back and did not enter the beam of the projector.
These early versions
of Wake and Dropping represented
a shift from previous works. The move away from the configuration of the space
as simple cinema-like layouts represented a major development in the investigation.
It became evident that there was a difference between how people moved around
the space and what the intended movement through the space was. A problem
was that most people did not like to enter the light beam from the projection
because of a fear of interrupting the image and affecting the viewing experience
for other viewers in the room at the same time. The quality of the sound compared
to the quality of the image was also a problem.
The issue of how
the viewer interacts had been resolved by simplifying the interaction to the
binary detection of movement. Simpler technology to sense this was used. As
a result of this simplification of technology and interaction, options for
more subtle interactions, in terms of how long someone has been moving or
stationary, became practical. This represented a move from spatial sampling
to temporal sampling techniques. While the interaction at one level is a binary
event those binary events, when combined with statistical temporal sampling
techniques, open up the possibility of more complex interactions [14] .
This stage brought
into the open many new directions which called for a refinement and rewriting
of the research proposal. This finalised the preliminary stages of the project.
The investigation
to this point had clarified the aims of the research and it was time to redraft
the topic to reflect this intent. The research topic was redrafted to:
What are the new aesthetic and technical issues involved in evoking
corporeal sensations related to being submerged in water with interactive
video installations?
This concentrated
the project on the creation and evocation of elements that then create a potent
sensory experience. From the preliminary investigation three questions concerned
with the challenges of working with interactive video installations had developed.
Many of the final solutions to these problems had started to evolve.
28. Robin
Petterd, Dropping – in progress, 2000, interactive
video installation, dimensions variable.
Dropping was
developed in a desktop situation for a period of four months. The room in
which it was next trialled was eight metres long and four metres wide. The
layout consisted of an image projected into two corners of the room. There
were six sound sources: one below and above each of the projections, and two
speakers in the middle of the room at the sides. The sounds panned left to
right and up and down in the space. Each projector had its own sensor placed
so that it sensed the viewer’s movement at that end of the room.
The first version
of the temporal sampling technique was tested. This allowed for logging of
the duration of movement and stillness in the room. The data was divided into
thirds, one third being more than the average activity, another the average
activity and the final third was less than average activity. When the installation
was running, the average of the last eight events was compared with values
gained from previous interactions to determine the level of current activity.
Depending upon the current level of activity one of three different groups
of media was played.
As each projector
was controlled by a separate sensor, when viewers were in only one area of
the space, this could lead to the situation whereby one end of the space could
be at the surface and the other at the sea floor. The two sensors were used
to allow for multiple viewers in the room at the same time. However, this
conflicted with the sensations Dropping was attempting to suggest. The use of the multiple sound
sources was the most successful component of this layout. The panning of the
sounds up and down was hardly noticeable. The layering effect of the multiple
channels was exciting. The system to achieve this used the multiple computer
message software, which was a cheap way of making multiple channel sound,
but it did not give control of the positions of the sounds and required the
use of multiple computers just to play the sounds. This led to the use of
surround sound in the next version of Wake.
The next stage
of Dropping involved the extensive testing of different spatial configurations of
the work, exploration of screen types, size of room and size of the projected
imagery. The use of multiple sensors in a single installation was abandoned
for the reasons mentioned above.
In a larger room
that was eight by four metres, the following configurations were trialled:
·
Two large projected images approximately
five metres wide and three metres high at either end of the room projected
on the walls by projectors placed on the floor;
·
Smaller projected images approximately one
and a half metres wide and one metre high, side by side in a corner of the
room.
The use of interactivity
in a large room means that more people can be in the same room at the same
time, however this makes it harder to create an easily recognisable link between
each of viewer’s actions and the installation’s reaction.
The installation
was moved to a smaller room approximately four metres wide and five metres
long, with an entrance at each end of the room.
The first stage
of experimentation in this room was with projecting the images onto different
parts of the wall, in both high and low positions.
29. Robin
Petterd, Dropping – in progress, 2001, interactive
video installation, dimensions variable.
Custom-made rear
projection screens were introduced. These were placed facing each other and
the projectors were then placed under the screens. There was one metre as
a walkway between the screens and the end walls of the room. The walkway between
the screens and the side walls was half a metre. This forced the viewer to
be physically closer to the image, while not interrupting the projection.
It allowed other viewers to walk through the central walkway of the room and
interrupt the projections. The rear projection screens created a sense of
more sources of imagery than there actually were in the room and the light
from the projections reflected around the room.
30. Robin
Petterd, Dropping – in progress, 2001, interactive
video installation, dimensions variable.
The next stage
was to experiment with three projectors and it was quickly discovered that
working with three projectors in a small room was problematic. The room became
constricted by the technology, the viewer was not able to pass through the
space comfortably without colliding with equipment.
The work was then
moved into a room that was eight metres wide, nine metres long and three metres
high.
A number of layouts
were trialled in this room:
·
The use of three rear projections in a U-shaped arrangement in the middle of
the room. This allowed the viewer to walk around the images (on the outside)
or to stand in the middle of the images.
The most successful
of all these version were the two parallel rear projection screens in the
smaller room. The combination of the size of the room and the placement of
the rear projection screens created a suggestion of being surrounded by and
close to the image. The large projections became a spectacle. The physical
feelings being evoked in Dropping are intimate and close to the body. The spectacle aspect
of the large projections contradicted the closeness of this experience. The
use of screens that could be walked around created a feeling of being in an
environment, instead of viewing cinematic narrative, the screens giving a
structure to the space.
The intention
of Wake was refined to focus on the
experience of water engulfing the swimmer that happens when entering the water
from a beach and the calm once past the surf.
31. Robin
Petterd, Wake – in progress, 2001, interactive
video installation, dimensions variable.
The room was eight
metres long, five metres wide and three metres high. In this version the imagery
was projected on to the floor of the room. When there was movement in the
room the image was of water lapping at the shore/beach. When the movement
ceased the image froze and a roar started. When it reaches its peak an image
was projected onto the wall at the end of the room. The sound was encoded
as surround sound [15] . When there was
an image of the wave coming towards the viewer, there was a corresponding
diagonal panning of the sound through the room. When there was an image of
water lapping around the viewer’s feet the sound was placed in the middle
of the room.
The problems included
the dullness of the projection on the floor. This was due to the projector
needing a new globe and the dark textured carpet of the gallery not making
an ideal surface for projecting onto. The loud rushing sound gave the installation
a strong violent presence, at times it was too loud and harsh.
At the end of
this stage the spatial layout of the works started to evolve towards the final
configuration. The use of surround sound added a presence and physicality
to the imagery. The other development was the recognition of the importance
of time spent working in the gallery with the works set up, to be able to
test and modify aspects of media and scripts rapidly. Testing on a desktop
and the sketching of plans for the works is different to being able to trial,
compare and make alterations to the layout with projectors and other equipment
in a gallery space.
The two projects
developed at this point represented three types of sensations of being in
the water: floating and sinking and attempting to remain stationary in the
surf. The resulting installations have a physical presence but at times they
are uncharacteristically violent. Future developments addressed this by constructing
softer and smoother media for the work. In addition, only the floor projection
started to move away from the paradigm of an image on a wall or vertical surface.
These issues needed to be addressed by developing another installation.
The piece that
was developed out of this need for a ‘soft’ experience is Under.
One of the most pronounced aspects of being in the water is the consciousness
of our breath. This is because we are unable to breathe under water and breathing
out creates odd feelings and sounds from this relatively simple bodily act.
32. Robin
Petterd, Under – in progress, 2001, interactive
video installation, dimensions variable.
The installation
Under is a projection onto the ceiling of a room. When no activity
is detected in the room the image is just under the surface of the water.
When movement occurs bubbles float to the surface. This was the first time
the still state was conceived as the main condition for an installation. When
there was no-one in the room for a period of time the work timed out to a
still state. In the previous two installations the timeout state was a moving
state. A new version of the activity sampling system was a feature of this
installation. The system now mapped the current activity state to a number
between one and twenty. This was based on a standard deviation [16] . If the measured duration for
which viewers in the gallery were still deviated for longer than the standard
deviation, the activity state decreased. The activity state was increased
if the deviation of the duration of movement was greater than the standard
deviation of movement. In the case of this installation, it was then mapped
to twenty different segments of bubble footage that were sorted so the soft,
slow bubbles played when the activity state was low and the faster, larger,
louder bubbles played when the activity state was higher.
In the first version
the shape of the projection was a square. This was changed to a circle, evoking
the sensation of restricted vision through a face mask.
33. Robin
Petterd, Under – in progress, 2001, interactive
video installation, dimensions variable.
A breathing sound
was triggered when movement stopped in the room, in an attempt to suggest
the moment when a breath is taken before diving down under the water. Unlike
s the other sounds, this was recorded out of the water. This crisp sound was
odd when combined with the other sounds.
Under
was installed with Dropping and
the next progression of Wake. This
was the first time that more than one installation had been tested in a room
at a time. This room was twelve metres long by six metres wide.
34. Robin
Petterd, Dropping – in progress, 2001, interactive
video installation, dimensions variable.
This next development
of Dropping tested a different type of screen material. The problem
with the screen material was its shiny surface and tendency to develop creases
easily.
35. Robin
Petterd, Wake – in progress, 2001,
interactive video installation, dimensions variable.
The next version
of Wake incorporated the feedback of twenty different sections of footage based
on the activity state. The issue of the projection’s darkness versus
brightness on the floor was explored by projecting onto a thin layer of sand
on the floor. This created a richer almost tactile projection. The use of
a screen on the ground meant there was a separation between the projection
and the surrounding floor. Only one viewer walked on the sand into the projection.
The conclusion was reached that most people will not step onto or into an
image. They are trained in a protocol when watching video to view the image
from a single viewpoint and not spoil the viewing experience of others by
interrupting the light beam of the projector. It proved practically impossible
to tempt viewers to break out of this pattern.
This was the first
time multiple installations had been set up in a single room, so new problems
and possibilities of how the works could interrelate were introduced. These
included the issues of sounds overlapping and whether or not the installations
should be in discrete areas. Wake and Under had
been planned to be in the same room, separated by a wall. But the large roaring
sound and the hit of wave were in contrast to the soft, rippling bubble imagery
and forced a reconsideration of this plan.
The lack of sound
quality from the microphone in the waterproof casing was in stark contrast
to the quality of the imagery. One of the challenges of underwater sound recording
is that there is a difference between how sounds are perceived underwater
and the sounds that result from underwater recordings. Diving into the water,
it seems the sounds that are heard are slow and have a relaxed heavy character.
This adds to the sensation of time slowing down that occurs when submerged.
Water conducts sounds in a different way to air and sounds are often sharper
and harsher when recorded, or there is complete silence.
This problem with
the sound created an opportunity to rework the sounds and at the same time
investigate using the sounds as a method to integrate the three works.
The purpose of
this stage was to rework the sounds as a means of orchestrating the totality
of the works and to start developing the spatial configuration of the works.
The first step
was re-recording many of the sounds. There are few choices of techniques for
underwater sound recording; these include the use of hydrophones and microphones
inside plastic casings. The sounds recorded by a hydrophone are very different
to the sounds that a person actually hears in the water. Much of what we expect
to hear underwater can only be heard at the surface. Liquid Sensations
only deals with experiences that are near
the surface and do have an audible component. After research into sound design
for underwater film productions and watching films that used underwater footage,
it was found that most sound production for these is completed in pools and
streams. The sounds recorded in synchronicity with the imagery do not give
the impression of the environment that filmmakers are trying to create.
The sounds in
the installations can be divided into groups: the surface sounds, the underwater
sounds and the sounds that are played when activity is sensed in the room.
Many of the sounds such as the breaking wave were not completely rerecorded
but were processed further, mixed and overlaid with new sound recordings.
It is common to associate a roaring sound with breaking waves. Close to the
wave the sound is different to this. The roar of a wave breaking was mixed
and overlaid with the actual sound of the wave hitting the camera casing.
The underwater sounds were recordings taken in streams and pools and were
recorded with a mono microphone encased in plastic covering.
The sounds were
then developed as a linear mix. The use of surround sound was replaced by
a complex stereo sound mix. With three installations in one gallery, using
surround sound would mean fifteen sound sources close together. Some of the
immersive effect of surround sound would be lost. What was adopted was a method
of working where the six sound sources from the installations become a combined
multi-channel sound mix. During the reworking of the sounds I changed from
using the built-in sound features of After Effects to using Pro Tools Free [17] . A number of
versions of the linear mix were developed. One common aspect that developed
was a return to a similar base sound, with a strong cyclic rhythm. This was
based on the sounds and type of sensations experienced when swimming through
the water. It became a form of sound that strokes the viewer’s body.
There was a change in focus from the sounds that exist in water, to the sounds
that I hear in my head whilst in the water.
This stroking
effect and the remixed sounds were then incorporated into the control system
for the installations. The stroking sounds were incorporated in a script where
the current activity state of the three installations was averaged and then
this was used based on a sine function to control the timing, volume and panning
of the sounds [18] .
The number of sounds played in the underwater mix was dependent on this average
activity value. The actual sounds played were chosen randomly from a set of
forty sounds.
The works were
then set up just as a sound installation without imagery. This was tested
to determine if the interactive versions were viable using sound alone. The
louder, more violent sounds of Wake were still a concern compared to the softness of the
sounds of the other works. The room used was eight metres long and four metres
wide; the sound sources were closer together than they would have been with
imagery. If the works were going to function as an orchestrated whole, the
relative success or failure would be evident when they were placed close together.
The installations were set up in a linear sequence. The viewer walked into
the room, into one work, and then moved another two metres forward to the
next.
36.
Plan of possible positions for the works when they
were set up as sound-only installations.
A number of options
were tried to test Wake in different
places, relative to the other pieces. These include:
·
Wake as the first
work that the viewer encountered;
·
Wake as the second
installation in the sequence;
·
Wake as the final
work experienced;
·
The sounds from just one of the installations
in all three positions in the room was also trialled.
These were documented
by a stereo microphone being moved around the room.
What became evident
from this experimentation was that while the aural experience of Wake was
different from the other components, the contrast that it brought to the whole
was important. When Wake was
removed from the sequence a sameness developed across the works. Where Wake was not at the beginning of the sequence, it had a
jarring effect on the overall works. The wave sounds in Wake were synced to the imagery and there was a total of
twenty sequences to reflect all of the possible activity states in the system.
Some of these were short and the shorter ones meant the sounds quickly cut
out. After testing, this led to the conclusion that the sequences with the
wave hitting needed to be reworked and extended. It became evident that the
activity states calculated needed to be more responsive to changes in the
room.
The reworking
and remixing of all the sounds for the installation became a large undertaking,
which was successful. The sound-only experiences were possibly the most tactile,
fluid and liquid environments that had been developed. The overlapping sounds
from the multiple sources created a physical presence that was co-ordinated.
The interaction became a side effect of the viewer being in the room, it was
a reflection of activity in the room instead of a direct relationship with
a single viewer.
The general spatial
sequence of the works had been resolved. Wake
was to be the first installation viewed, then Dropping
and finally Under. There was
a need to rework and extend the wave-hitting sounds in Wake. Likewise Dropping only
had three states and needed to be reworked to reflect all of the possible
activity states. The sound-only versions of the installations were suggestive
of an aquatic environment. Previously the imagery had possibly been too literal.
The reworking of the imagery would allow more ambiguous qualities to be developed
in the imagery.
Removing one aspect
from the installations for a period of time allowed a global approach to the
project to be developed. It reduced the complexity of the installations for
a period allowing refinements to be made to the works as a whole.
The data used
to calculate the activity values was exported from the installation control
software and imported into Excel
[19] .
37.
Plot of stillness durations and moving durations.
Once this data
was plotted in a graph, it was discovered that occasionally there were values
that were larger compared to the balance of the distribution of data and this
occurred most often when inactivity was detected. The Lingo [20] code was checked to make sure
this was not an error in the algorithm. A statistical measure such as standard
deviation would be skewed by these unusual values.
A number
of options were trialled to solve this problem. The first thing that was tested
was reducing the time before the system went into a timeout mode. Then new
data was collected with one of the installations set up. The lower timeout
value did not result in higher values skewing the standard deviation. The
optimal time of two and a half minutes was decided based on the plotted data.
The use of smoothing algorithms to flatten the spikes was investigated. A
median [21]
filter was tested. This is a filter where a value is replaced by the median
value of a range of nearby values. Ignoring these spikes, by changing the
values that are close to the maximum to the median of the data, was tested.
This last solution was found to give the best overall results.
The use of standard
deviations still was not giving an accurate reflection of what was occurring
in the gallery relative to past events. Because the data was in Excel a number
of different solutions could be applied quickly to the data. Probability [22] was found to be most successful in this context
of comparing the current events to past events. A type of ‘model’
of the space developed. Over time the system settled to calculating similar
values if the events were close to normal activity but if an event was unusual
the system would reflect this.
The sections where
the waves hit and the underwater sections of Wake were
re-shot. The descending, bottom and going back up footage of Dropping were re-shot. This allowed many of the successful aspects
of the sounds-only version to be integrated into the visual components of
the work. One example of this was making the edges of the images softer by
a method using sections of footage as masks. The imagery was collaged and
layered together more than in previous versions. The media displayed when
the extreme activity states occur were created to have a greater contrast
to the ‘normal’ states.
Out of the sound-only
experiments, a general outline of the gallery layout had been developed. This
needed to be finalised. A number of sketches were completed of possible configurations
and views from different positions in the gallery. The problem was that these
did not always reflect the correct scale of the projections or where it was
possible to place temporary walls in the gallery. A three-dimensional scale
model of the gallery and installation was developed to allow for greater accuracy
in planning.
Using this 3D
model a number of variations were evaluated. These included:
·
different positions for temporary walls between Wake
and Dropping;
·
the use of fewer temporary walls between the
works;
·
Wake and Under both in the taller section
of the gallery;
·
the addition of more projections on the floor
in Wake and more projections on the
ceiling in Under;
·
the use of a temporary wall to define a small
foyer in the gallery;
The final option
was then tested in the gallery.
Exporting the
data from Director allowed it to be examined in a different way so that an
understanding of what the sensor data was showing could be developed. Working
with a program such as Excel allowed algorithms to be prototyped and the results
to be plotted more quickly than working with Lingo allowed. The use of a three-dimensional
model of the gallery allowed options to be checked rapidly. But this did not
allow for the fully integrated experience of the multiple sounds and interactivity
to be simulated. The reworking of the imagery to reflect the ambiguous effect
of the sounds allowed the works to evoke the sensory experience of submersion
in water more successfully than with images alone.
During the preliminary stages
of the project the research questions were formulated. These developed from
studio experimentation and observation of other interactive installations.
The main stages of the project involved extensive testing and evaluation
of solutions to the research questions while resolving the works. From the
successful aspects of this process the methodologies outlined in Appendix
One were developed.
next
[1] Polar Circuit is an event where a
group of international new media artists and writers gather over a period
of one month to work together, to show work to each other and to take part
in workshops. The focus of the event is developing collaboration and having
longer periods of time than are commonly available in conferences to develop
contacts and projects.
[2] The workshop was run by Adrianne Wortzel. The Kalevala
is an epic collection of poetic stories based on songs and oral stories handed
down from generation to generation of Finnish people which represents an important
part of their identity and culture.
[5] During this time I was working on
a collaborative online project with writer Diane Caney, so I had not completely
abandoned the use of simpler web-based interactive works. The image of the
fish developed out of this collaborative process. This work can be seen at
http://www.archiving.com.au/
[6] Adobe After Effects is an application
used for video effects and animation. It allows for precise control of the
visual aspects of a video image.
[7] Machine vision is a technology which
uses video cameras and computer algorithms to allow a computer to see and
recognise. The most common machine vision solution used by artists is a piece
of software called BigEye produced by Steim (http://www.steim.nl/) in the Netherlands.
It uses midi signals to communicate with other programs. I investigated the
use of Steim’s Image/ine
effects program, but chose not to use it and other midi-based systems such
as Nato 242+55 (http://www.eusocial.com/)
because of the need to introduce even more new technologies such as MAX (http://www.cycling74.com/)
into the process whereas Director was an application that I had previously
used extensively. This meant the working versions of installations could be
developed relatively quickly without the need to learn new applications. The
use of Nato 242+55 had become widespread in the area of interactive video
during the later stages of the project and new software to work with video
with other applications such as jmax (http://www.ircam.fr/equipes/temps-reel/jmax/en/index.php3)
and Pure Data (http://www-crca.ucsd.edu/~msp/) have been developed. But at
this early stage of the project these video plug-ins had not yet been developed.
They were largely developed out of a recognised need for this type of software.
[8] Xtra is a piece of software that
is added to Director to expand the functionality of the program.
[9] Director is an application for developing
interactive multimedia content. It uses the metaphor of a theatre production
for its interface. It includes Lingo which is a scripting language that can
control all aspects of the application and allows for rapid development of
interactive multimedia.
[11] A report on these events was published
in Australian Network for Art and Technology News, Dec 99 Feb 2000, Issue 39.
[12] Invenção
(http://www.itaucultural.org.br/invencao/invencao.html)
in Sau Paulo, Brazil was a conference organised
by Itaú Cultural Institute in collaboration with the Inter-Society
for the Electronic Art, Arts Leonardo/ ISAST Centre for Advanced Inquiry in
the Interactive Arts, University of Wales College, Newport and the Centre
for Science Technology and Art Research, University of Plymouth. It sought
to examine the convergence of art, science and technology. At Invenção
I delivered a joint paper that explored some of the work and ideas that came
out of the text based virtual world work that I completed while in Finland.
The conference paper was titled ‘Rewrapping the real world: using hyper-narrative
in virtual spaces to create un-common realities’ Dr David Casacuberta,
Robin Petterd, Adrianne Wortzel, for INVENÇÃO in Brazil 25th-29th
August online via http://www.itaucultural.org.br/invencao/invencao.html
.
[13] Director’s multi-user compatibilities
are designed to allow messages to be sent between applications developed with
Director. The most common uses are online chat systems and multi- user games.
[14] Jim Campbell in his article Delusions
of Dialogue: Control and Choice in Interactive Art discusses
interaction systems as a spectrum that ranges from controllable systems to
responsive systems. For Campbell binary interactions are a characteristic
of simple controllable systems and systems where the viewer’s actions
are interpreted as complex responsive systems. He states that the use of controllable
systems is one reason why interactive art often does create rich engaging
viewing experiences. The area he sees as having undeveloped potential is the
use of systems where software develops a memory of the environment it is installed
in and reacts in both a short-term and long-term manner. The use of statistical
temporal sampling techniques in my installations means the software is interpreting
events in the room and becomes responsive to the viewer.
[15] This was encoded as Pro Logic Surround
sound, using PanHandler, which is a sound plug-in for Adobe
Premiere. Surround sound systems involve the use of multiple sound sources,
normally arranged with three speakers at the front of the room near the screen
and two speaker at the rear of the room behind the viewing position.
[16] Standard deviation is a measure
of where data is clustered around the mean of a set of data.
[17] Pro Tools Free is a limited capabilities
version of DigiDesign Pro Tools software. It is an application that is used
for multi-track sound editing, mixing and midi sequencing.
[18] thevalue = sin(the current time * averaged activity for all
the installations)
[19] Excel is a spreadsheet program.
[20] Director’s built-in scripting
language.
[21] The median is the value that occurs
most often in a data set. This is less likely to be affected by an unusually
large or small number in a set of data than the mean.
[22] Probability is a measure of the
likelihood a number will occur. It is calculated by dividing how many times
a number has occurred in a data set by the total number of values in the
data set.
next
