"We are still very much annoyed by out-of-date notions of time."

Jean Tinguely, untitled Statement, ‘ZERO 3′ (1961)



Metaphone - the project is exploring the connection of body and machine, it was always a challenging task and in many generations this connection was explored in many different ways. The project Metaphone explores the evocative connection of the machine and body and seeks to research thrilling influences resulting in between. How people are connected to the machines and how machines are connected to people is a very broad question to investigate both from scientific and artistic perspectives.

The Metaphone art project is consisting of interactive apparatus, painting technique and sonic art. The apparatus acts as an actuator and produces pictorial artworks by interacting with the audience through sensors. In the first phase artworks are generated from the incoming data and simultaneously the data gets translated into colour scale. While in the second part of the project, audio is generated back from the painting by analyzing visual image through colour recognition and gets translated into sound creating evocative soundscapes.

An important lens through which we can study some of the divisions of what it means to be human and what it means to be a machine in this digital age is through asking questions of authorship. In my work, I have been engaging in creating interactive art. In short, the project have been building on computational models to create some interaction with the audience coming to experience the machine. Authorship is a potentially broad concept, but in this context, I think of it as questioning who is the creator of the art: the artist, the machine that interactively created the output, or the participants that interacted with it, feeding the machine with their input?

Modules

Metaphone #01 - interaction through the audio sensor microphone.

In the first phase of the project (2011), the machine has produced pictorial artworks from sound input: audio frequency got mapped into colour scale, the audio volume was affecting the radius of the painting; secondly, the creation of the soundscape from visuals was executed: the algorithm was translating colour scale of the painting back into audible scape.

Metaphone #02 - drawings produced from EEG sensor.

Additional electroencephalographic (EEG) set receives participant’s brain impulses emission, at the same, the Metaphone machine is drawing pictorials and monitoring sounds from the sensor. Listening of the participant’s brain signals being amplified and exposed publicly also getting them drawn on paper is a potential of such. Personal neuro-sites and para-states have been audio-visualized in 2012.

Metaphone #03 - interaction using bio-sensing technology.

The Metaphone machine collects the participant’s bio-data, Galvanic Skin Response (GSR) and Heart Rate (HR), creating a process of movement, painting and sound. The machine behaves in machine-like, aesthetically evocative ways: a shaft on two large wheels rotates on the floor, carrying paint that is dripped onto a large sheet of aquarelle paper on the floor according to bio-sensor data. A soundscape rhythmically follows the bio-sensor data, but also has its own machine-like sounds.
This bio-sensing module was implement in 2013. In essence, the Metaphone is constituted by three main elements, namely, (1) a bio-ball that fits in the palm of the hand, picking up on the biological signals of the participant, converting it into a stream of bio-data transmitted wirelessly to the rest of the machine, (2) a drawing machine that converts sound as input into drawings on a large aquarelle paper underneath it, (3) a sonic core that both converts the bio-data into sounds (internal, not heard), that the drawing machine can understand, and makes it audible to the participants.
Several research papers were written about this module.

Metaphone #04 – the blowing sensor and control panel.

Exploring various ways of the machine influencing participants, interaction and painting technique, the fourth version of the Metaphone was constructed. Blowing sensor combined with control panel with knobs continues questioning interaction paradigm between human and machine and also raising questions on aspects of control. By blowing the participants feed off the machine with power and then control several parameters of the machine: the amount of each colour, the speed of the moving cart and the frequency of opening valves. Metaphone #04 was exhibited at ISEA in Dubai, 2014.

Motivation

Querying whether authorship is shared between artist, observer and artwork, where contemporary art practices such as mixing, open-source, reflection, interaction and collaboration have become a norm. Sharing artistic qualities with the audience, involving observer into the process of art and transforming observer into participant by allowing taking part in the creative process and finishing the artwork in interaction with the machine. Generative process, interdisciplinary and wide range of traditional and new media is predominant for the Metaphone project. By transforming audience’s experience and bio-data into visual artworks the Metaphone project reflects on interactive aesthetics within phenomenological doctrine.

The project is dedicated to the synthesis of colour and sound, visual music phenomena, from modernistic approach and psychological affect of colour in Wassily Kandinsky's relation between form and colour, his "inner necessity" is the principle of art and the foundation of forms and the harmony of colours with efficient contact with the human soul to revolutionary Jean Tinguely's machinery and linked to his mechanical interactive machine series - Métamatic - producing pictorial artworks.

Three aspects of the machine were approached and revisited repeatedly through the project’s development: (1) the formalism method of analysing the materials, (2) thoroughly examining the content – that is the bio-data provided by the participants, and (3) examining the context in which the machine would be placed.

Exhibitions

Firstly it got exhibited as an open studio at the Open Studios exhibition on the White Balcony in Royal Institute of Art in Stockholm, Sweden. Process based studio work was implemented and numerous pictorials from sound were produced in watercolour, ink and acrylics on paper. Environmental Mandala - the project so called according to the first show visitors, no one could argue the importance of Mandala term in this project.

The list of the project's expositions:

2015 - Crossings, CHI Interactivity, COEX, Seoul, South Korea;
2015 - Light Bounces: Space and Body, Art.CHI online exhibition;
2014 - Location. Electric Tradition. ISEA, the 20th International Symposium on Electronic Art, Al Fahidi district, Dubai, United Arab Emirates;
2013 - Artist Talk, PIVÔ art space, São Paulo, Brazil;
2013 - Art Talks, Jönköpings County Museum, Jönköping, Sweden;
2013 - Research Night, Debaser Medis, Stockholm, Sweden;
2013 - AlbaNova University Center, Stockholm, Sweden;
2013 - Dktus art project space, Stockholm, Sweden;
2013 - Machine Aesthetics, Den Frie art centre, Copenhagen, Denmark;
2013 - Changing Perspectives, CHI Interactivity, Palais de Congrès de Paris, Paris, France;
2013 - European Media Art Festival, Osnabrueck, Germany;
2013 - Bio-tones, Digital Art Center, Kista, Sweden;
2012 - Dual Identity, Konstnarshuset, Stockholm, Sweden;
2012 - Parawise, Mellanrummet, Stockholm, Sweden;
2011 - Remix, Arka gallery, Vilnius, Lithuania;
2011 - Open Studios, Royal Institute of Art, Stockholm, Sweden.

The project has been exhibited in many occasions, both in live performances and in exhibition installations. Throughout the history, several experimental tactics were applied: electroencephalography (EEG) machine (Parawise exhibition, Mellanrummet), drawing from electromagnetic sonics, drone noisescapes (DAC and Jönköpings County Museum), popular music songs (one song, several songs, full album, e.g. The Beatles, Pink Floyd, Beasty Boys, Sonic Youth, etc., and live DJ sets), audio feedback, biofeedback, bio-data, celebrity biological signals, etc. All expositions differ from each other and project has been developed and presented diversely. Environmental sounds, music performances, mix of 2 national anthems and colors of 2 national flags (performance at Dual Identity exhibition, Konstnarshuset), personalized paintings, surround soundscape installation (Bio-tones exhibition accompanied by Machine Art screenings, Digital Art Center), collective performance of several participants playing with 2 bio-sensors (European Media Art Festival) and other tactics were implemented to reach variant results at various locations.

Collaborations

The project started as an exploration of interactive techniques within the arts and gradually evolved into a research project. Most recently the project was shared, developed, and explored by a collaborative multidisciplinary collective and turned into an open artistic research platform.

The project at different stages has been developed in collaborative work with Anders Lundström (Royal Institute of Technology, KTH), Tore Nilsson (Royal Institute of Art, KKH), Jordi Solsona Belenguer (Mobile Life Centre), Kristina Höök (Mobile Life Centre), Magnus Dahl (KKH), Vincent Lewandowski (KTH) by now. Various artists, designers, technicians and researchers at different stages of the project took part in creating and implementing the interactive art project as well as conducting the research.

Research

Research is being conducted alongside the artistic practice and several conference papers were published:

CHI conference, Art.Chi workshop and catalogue paper, Vygandas Šimbelis, “Metaphone: Distinguishing Human and Machine”, ACM Press, 2015;
CHI conference, Full Paper, "Metaphone: Machine Aesthetics Meets Interaction Design", V. Šimbelis, A. Lundström, K. Höök, J. Solsona, and V. Lewandowski, ACM Press, 2014;
CHI conference, Extended Abstract for Interactivity, "Metaphone: An Artistic Exploration of Biofeedback and Machine Aesthetics", V. Šimbelis, and K. Höök, ACM Press, 2013;
CHI conference, Extended Abstract for organising a workshop, "Changing Perspectives of Time in HCI", S. Lindley, R. Corish, P. Ferreira, V. Šimbelis, and E. Vaara, ACM Press, 2013.

-

Documented pictures of the machine, paintings in spiral pattern, initial sketches and some detailed elements are showed below. By now, here are about 150 drawings, most of them are already sold out or gifted away. Dimensions of paintings vary from pocket book size till diameter of 2 meters.

The project has been discussed on National Swedish Radio, in several publications and in online articles. The painting of the Metaphone was published on the front cover of the Royal Institute of Technology (KTH) in Stockholm, Sweden, for its 2014-2015 year catalogue.

Other machines with diverse patterns like spirograph will come out soon and will join the Metaphone family. We are hoping for the great continuation!

Metaphone.net is on its way to spread the bio-sonic visual.



Mellanrummet gallery, Stockholm 


exposing the code on the screen 


expanded and exposed process 


Metaphone #04– the blowing sensor 


the blowing sensor in action 

experiencing the Metaphone module 4
Erkki Huhtamo media archeologist: experiencing the Metaphone module 4


preparation at Digital Art Centre 


surround soundscape at DAC 


at the White Balcony, KKH 


kung-fu kick at the study 


the interest at CHI conference 

Jordi Solsona and Mattias Jacobsson
experimental study with sensors: Jordi Solsona and Mattias Jacobsson


dead participant 


performance with EEG sensor 


Cultural Commentator at work 


relaxing interaction at DKTUS 


bio-ball sensor flashes and blinks 


2 bio-balls resting 


design process of the wax ball 


open studio at KKH 


start of the performance 


blending colours 


Metaphone in process 


the valves and loaded tubings  


spots and lines of ink 


close up of painting technique 


close up 


various experiments with paints 


some paint effects 


experiment with acrylics 


mix of ink and aquarelle paint 


painting from EEG, 2x2 meters 


painting on aquarelle paper 


readable colours and lines 


blended colours and flowing fluids 


fluids and wrapped paper effect 


painting on wet paper 


mapped sounds in colour lines 


PJ Harvey - Down By The Water 


mixed colours of long painting time 


separated lines of slower process 


minimalist drawing 


the very first Metaphone painting 


Sonic Youth - Daydream Nation 


pattern explorations 


the early sketch of the system