Image Player

Sound Art + Mobile Application


Image Player is a mobile phone application that turns cell phones to its users・ personal sonic art player. With Image Player, captured images from the phone・s camera become the basis to creation of sonic art. Image Player works by analyzing and turning byte data of a captured image into midi sequence.

Programming on Mobile Processing.

Download Image Player.

Tested on Nokia N82. Please inform any bugs to help make Image Player better. SourceCode obtainable via email.

Listen to sound clips.

Production Essay: An F-Word Project

This is a project full of failures. Some are real failures that bring fatal harm to the project. Some open up new space for exploration. Some do both. The outcome may not sound as appealing as the original idea does. But we hope that the process working through failures makes up to it.

Behind this sonic art project are two notions. The core is the focus on everyday experience. This project should be capable of including as many people as possible and in which they should be endowed with a high degree of freedom to adjust and make their own pieces and to enjoy in their ways. Pervasiveness, individuality, sense of control and multiple usages are keywords in our pursuit. Mobile phone is therefore a very suitable platform.

The second core is casualness. We think that users should be able to understand and make sense of the project with their own knowledge. They should not be forced to look through any special goggle or theory. This is not to teach. They should come and enjoy throughout. And so we also do not scratch our phone to make sound. We do not employ destructive approach.

Mobile phone by its nature is a very dynamic device born to perform real-time data processing. Generative process as a sonic art practice therefore should go along well with mobile phone. Among a number of data streams on a mobile phone, we have chosen the camera for its visual strength that may strengthen the applicability on notation.

The project started with an idea of a color sensor meant to work with real-time stream of images from cell phone・s camera. Color would be analyzed and turned into synthesized sound. Therefore it would be applicable to all images and, also in its way, traditional notation system.

Constrained by cell phone・s processing power and limitations on development tool, Image Player not only does not work with color. It also does not play synthesized sound. It analyzes byte data and plays midi. Frankly, the current Image Player is a degraded outcome of the original project but is the best of possible at the moment.

Now we discuss technical details. This project is full of failures. First it failed to analyze color and this is fatal. Compare to analyzing byte data, color could provide a higher sense of control to users. In this sense the application failed to ally its logic to the logic of users. The application now interpret image in a way different from users・. Otherwise users could manipulate more directly the final generated sound. This also links to the issue on notation. Now without this alliance and sense of control users are unable to do notation. The claim that original claim that any image could become notation therefore is now not plausible. The possibility of the emergence of a new form of notation is killed.

Another main failure is its failure to play synthesized sound and has to settle on midi. This brings the problem on purity and the relationship between data and sound. With synthesized sound the relationship between data and sound is clearer. In the ideal, the higher value of byte data would generate synthesized sound in higher frequency and continuous identical value would result in longer synthesized sound. So the idea that byte data is directly affecting the acoustic quality and duration of sound is more obvious with synthesized sound. Now with midi the effect is still there but is not as visible. Another problem for midi is that generated sound is very much conditioned by midi channels unless the whole midi channel is programmed brand new. On the other hand these disadvantages are at some degree compensated. Midi generates more pleasant and comfortable sound as it was what we usually listen to. Maybe Midi would encourage a longer and more attentive listening practice? In this sense users may also be able to make use of the application in their way.

The third is the failure to work with real-time image. This is more bittersweet. Real-time processing can provide users an experience on direct manipulation that they would be really playing with the device. And the device would be more like an instrument. On the other hand, now working with a captured image, the device is more like a player the way a CD player does. By capturing an image users are like putting a CD into a CD player. Then what they have to do is to listen. These two approaches both have their strength that provides very different experience.

So come back to the notions. Pervasiveness, individuality, sense of control, multiple usages and casualness. How much does the project achieve? Image Player coming through failures is still an application that transform image to sound, and now in a pleasant way. Image Player is also easy to use. Everyone should be able to download one to their phone and create. Sense of control is area that scores the least marks.

Now you may probably have a feeling that this project is one that struggles through constraints and is one that tries to get the best out of failures. This outcome is not as appealing as the original idea. The maker also shares this feeling. But this is how far the project can reach as at today. Experiments are worth of its process. Counted its process the maker still feels very happy for this project and thinks it is a lovable piece.

On Sonic Art Quality

It・s an open work! Chance Operation! It・s so John Cage!

This is easy to understand. Image Player is an open work for its openness to variation on data input that generate different results. As mentioned, users enjoy a degree of freedom and sense of control. In terms of sound quality, different sound chips of mobile phones will also play in different sound quality. On synthesized sound this could be the difference on sound quality. On Midi this may result in different choice of Midi instruments which can make really wide variation.

From the audience・s point of view, looking at Image Player generating sound may give a similar experience as looking at John Cage playing his mysterious composition with bathtub and blender and his prepared piano, that as audience you may know the notion behind, but the actual process of operation is somehow hidden to the bare eyes. :What・s his composition?; audience might wonder. This is also the case when they come to Image Player. :Does a byte represent a pixel of color? Where are we in the picture now?; The disalliance of human・s visual logic and the player・s in this sense is John-Cagious. It only allows audience an understanding to the composition to a point where unexpectable chance operation can still work. They might know what is going on there. But they never know what exactly would come next. A higher note or a lower one? You never know.