interfacing audio & images 1989 - 1993

• 0 abstract

• this text describes several approaches to interface audio and images, all done at the institut for new media in frankfurt, germany. i believe very much, that all of the discussed topics could be easily applied for an interactive stageshow using realtime graphics hardware, which to this dimension noone has performed yet.
  
  additional keywords: audio, multimedia, soundtrack, stageshow, midi, virtual reality, digital signal processing, realtime graphics.

  
  
• 1 introduction

• for the last years, the main focus of my work has been on controlling pictures through audio-input and applying different setups in an art and performance environment. there already exists some tradition in synchronising sound to images like in multimedia-installations or videogames.
  
  a main fact is, that most artistic electronic work in a sense of craftmanship can be automatically done by a computer instead of an operator. there exists already a big library of fileformats both for audio and images that will serve this fact like standard midi files (smf), aiff, tiff and so on. along with special hard- and software there should now exist enough tools for interfacing pictures with sound, even in realtime applications. these setups could then be used for live-stageshows or other interactive purposes. even the production of videoclips could be automized in very complex ways. instead of an exhausting frame-by-frame-rendering of images, 3d-models could so be directly extracted from videoimages or audioinput in any format and the effects could be seen just as it happens.

  
  
• 2 inputmaterials
• • 2.1 audio
    audiodata can be supplied by the following sources:
    • standard midi files (created by softwaresequencers like cubase, vision, digital performer...)
    • midi-data (created by midi-keyboards, midi-guitars, ...)
    • analog soundsources (violins, space shuttle, voice, atomic explosion,...)
    • prerecorded samples on harddisk
  • 2.2 images
    imagedata can be supplied by the following sources:
    • live camera-input
    • laserdisk
    • harddisk (abekas a60, stores 30 seconds of video)
    • videotape
    • workstations that have realtime graphics-capabilities

  by combining these different devices, it is possible, to control images by musical parameters such as frequency, amplitude, spectrum or rythm.

  
  
• 3 imagemanipulation
• • 3.1 non-realtime
    non-realtime techniques can supply visual footage for laserdisks, harddisks or videotapes that can be triggered later on. mainly these techniques include algoritms, that require large amount of calculation time:
    • digital signal processing: fft, filtering, edgedetection...
    • traditional 2d or 3d-animation
    • complex, audiotriggered imagesynthesis
    • standard video postproduction
  • 3.2 realtime
    realtime techniques can be used to manipulate image data in live- interaction:
    • live mixing with videodevices
    • computercontrolled hard- and laserdiskaccess
    • mapping of raw or interpreted audiodata on 3d-grids or models
    • texturemapping on 3d-grids
    • warping
    • morphing
  
  
• 4 soundsynthesis

• along with the manipulation of imagedata, also sound can be processed or changed with the help of midi-controllable devices like samplers, mixing consoles, effects or light-controllers.

  • autocomposition:
  • midibrain: listens to the input and supplies a musical backup
  • midicloning: imitates the input
  • mathematical methods: randomwalk, feigenbaum,...
  
  
• 5 results
• • xtra.trax
    a program creates video-editlists from standard midi files, that will then automatically assemble a videoclip.
    
    
  • a666
    the abekas a60 harddiskrecorder is controlled by a musical instrument. with this, a video is no more cut, but instantly controlled in time.
    
    
  • cybertime
    fractal algorithms create sequences of video and a soundtrack.
    
    
  • flute
    a flute transforms texture-space in realtime.
    
    
  • paula chimes (see extra page)
    a musical instrument made out of 16 chimes and 2 videomonitors is capable of realtime imagewarping and autocomposition of music.
    
    
  • tilt it
    a video, that uses the soundtrack for the direct creation of 3d-computergraphics. for instance, a sample of a guitarsolo is calculated and animated into 3d-space and reflectionmapped with the original picture that created the sound.
    
    
  • dwarfmorph
    a software, that is able to create 3d-models out of 2d-videoimages. in a realtime application, the viewer of a scene can be directly part of the 3d-world he looks at.
    
    
  • elastic guitar grids
    the sound of a guitar creates waves on a static image.
    
    
  • biogrids
    images & sounds create colonies of objects.
    
    
  • hyena days (see extra page)
    the musicians control image- and sounddevices with their instruments (steina vasulka: violin, michael saup: guitar).
    
    
  • soundsculpt
    sound transforms form.
    
    
  • warping the biennale venezia 1993
    the voice of peter weibel warps his own face.
    
    
  • soundmapping
    sounds are mapped onto 3d-objects.
    
  
  
• 6 credits

• brainware: michael saup, institut for new media, frankfurt
  software: michael saup, bob o'kane, gideon may