Dr. Benedikt von Walter - head of MTV's digital research team for Northern Europe

So as expected: there has been quite a lot of the usual hooblah and blaahblaah about innovation and digital technology and other such buzzwords that everybody seems to spit out these days (what do these words even mean anymore?).  I guess when you have worked in this area as long as I have, the intellectual output of such workshops is seldom the main point any more. Rather, such workshops provide us with the best way to experience the different kinds of creative energies and experiments that are going on in regions such as the Baltic States.  So despite some of the usual recycling of American cliches of digital openness, participation and digital revolution plus a few good speakers, the best part of the week so far has been the guided visits to the local incubators, art collectives and cultural centres such as the Culture Factory Polymer where probably the most interesting art/technology/business synergies are happening today.

The first week in Estonia is now done. Now looking forward to seeing a few more of such places in the lovely town of Riga.

This here is a very quick experiment where I composed a palette with 5000 particles moving around semi-randomly in 3D space with their color subtly affected by the movement in the 3D-space (the z-axis). Instead of doing anything more fancy, the aim here was merely to experiment with the possibilities of this and to prove the concept,. Even this might look a bit more complex, in fact, I actually only used rectangles and ellipses to compose these images. I let the program run for a few hours and see what comes of it.

This image was taken in an anonymous warehouse that you had to enter through a backdoor…

The ‘loot’ comes from everywhere, each store has its own network spread across the country and outside (you would find stuff from Sri Lanka, parts of Africa…)…

Obviously, photography is not allowed. However, a few gentle words helped break into their world.

As I started digging into the history of the place, I found that the Wikipedia entry about the place itself had been hijacked! An excerpt below :-

Chor Bazaar is an area in South Mumbai famous for its second-hand goods. Although the name Chor means “thief” in Hindi. This area can be considered one of the tourist attractions of Mumbai (Bombay). It is a basically an “organized” flea market, where one has to rumage through junk and hopefully find treasures. The reason it is know as “thief’s market”, is because it assumed that goods sold there are stolen. Chor Bazaar if off the beaten path, but everyone knows about it.

In addition, the name Chor Bazaar was adopted by an Indie Indian Fused tshirt label based out of Brooklyn, NY with roots in, India. link title

Our designs are meant not just to be “cool” but to evoke memories of experiencing India, the India that our parents were raised in and the one that exists today. Both are far different but both are still very Indian.

Our mission is to expand the Indian-fusion art form to another realm. Most have experienced this, “fusion”, in music and literature but have hardly seen this transpire into urban apparel. We utilize the medium of our graphic t-shirts to assist in creating an identity that stems farther than mainstream’s portrayal of Indian culture.

Dang!

The primary problem I have with more classical animation, drawing and painting (and 3D) is that it is very time-consuming.  Being sometimes peripatetically cross-displine and cross-media, I am interested in doing things in almost every possible format that I get my hands on.  But, say, if I wanted to create a digital character with some facial expressions and mix it with some photography or video to get some idea across, doing these with the old ways would take days to complete.  A simple expressive character, in the end, is rather difficult and laborious to create properly and with style.

Now take a game engine such as Spore.  What these virtual reality and/or game engines allow one to do is quickly create a prototype of some idea or another — develop an “element” that can be used to explore an idea at its initial stages.  It is not perfect by any means; you do not get full control of what you get.  You get instead a rather rich set of parameters to play around with but this is still not the quasi-complete freedom you get with doing things analogically. However, what you do is get possibilities do things that would not have been available before unless you wanted to spend days and days creating every little element yourself from beginning.

So a part of the Uncanny Valley experiment is to develop a workflow that allows the quick production of such “mixules” and / or sketches that can be later worked with to develop more complete projects and designs.  I have played around with Spore mostly here as its the most recent of these games but will probably do my rounds around Sims, Second Life etc and whatever will be useful.  Each of these sketches have taken — on average — about 30 minutes to complete.  Most of that to try to work shadows etc (though in some pictures I have not spend enough time on this admittedly…)

The theory here could perhaps broadly be said a play around “locative media” and “ubiquituos computing” where the overlap of virtual and non-virtual reality is predicted to be the next evolution of the Internet. I am especially interested in conceptualizing the blurry notion of reality between the virtual and non-virtual as an experiment in both content and form.  Something close to evolutionary art but not quite.  I will do a series based on this eventually when the idea crystallizes but meanwhile just working out some of the techniques here — also thinking how some of these engines could be perhaps used to do short videos etc.

I will also post a tutorial up soon about how such “rapid prototyping machines” can be used …

[Tools used: Pentax K20D, Spore, Photoshop Cs3, San Miquel]

However, Olympus E520 serves my purpose for the moment – inbuilt image stabilization + four-thirds mount tremendously expands the line-up of lens that it can handle. The kit lens is one of the best in the range, and I love the controls (over my previous Nikon series of cams).

These are very first HDR experiments (literally amongst the first twenty photographs captured with the cam). Both have been shot near fountain, VT station, Mumbai. Yet to work on the style, as these are just thematic samples at the moment.

An another…

My friend / bcg partner Matti has perfected his approach (his kit: Pentax K20D) with HDR and has done an outstanding series on Mumbai, where is signature style is very visible. Check it out here.

And a detail of this:

[Tools used: Processing, CS3)

So instead of looking at the the individual as the primary means of representing the urban experience — as has been again and again — I was more interested in seeing the city as a wider assemblage of different spaces and speeds through which people have to navigate in their daily existence. Therefore, instead of taking pictures of people, I was more interested in seeing a kind of an a-anthropocentric vision of the world: not seeing frozen moments, but seeing fluctuating frame-rates, seeing different timescales of existence from cars to people to buildings to nature bubbling in-between.

What was interesting as I was showing the first “sketches” of this series around friends and other professional photographers, the most common question was: how did I do this? How was I able to achieve such a frame-rate/time-based effect? Did I use a special camera?  How did I achieve the multiple-exposure and ghosting effect?  So while this technique is still in development, I decided to start off here with a brief tutorial of how to do such time- and/or framerate-based photography and the possible techniques that can be used for further projects.

Here are the steps explained below for the first time:

1)  MULTIPLE EXPOSURES / AUTO-BRACKETING: First of all, I developed here a customized technique of photography that combines multiple exposure-photography with high-dynamic-range-imaging (HDRI) and digital painting to give the pictures a time-based feeling and movement to them. So to understand how this works, you have to understand some of the principles of HDRI photography. When you HDRI photography, you basically take multiple shots and exposures of the same scene instead of one that you do in normal photography.  So when you take a picture of a scene, instead of doing one picture, you take, for instance, the following five exposures: -4/ -2/ 0 /+2 /+4.  What we therefore get is a full dynamic range of the dark areas and highlights of the picture which is not possible through normal photograpghy.  HDRI photographic techniques are explained more carefully HERE.

2)  HDR MERGE: Once you have done the different different exposure of the scene, you need to somehow combine the images.  To do this, I then used a HDR software called Photomatix.  Photomatix allows you to take multiples pictures and it automatically merges and created a composite of the 5 different pictures with the maximum dynamic range between lights and shadows.

3)  RANDOM MOVEMENT / COMPOSITE: The thing about HDRI photography is that it does not work really well with movement — at least that is what we are supposed to believe.  The software cannot calculate the dynamic range for moving objects as they come in 5 different places.  Normally you need to use a tripod to get as static images as possible. There are certain ways to avoid this problem such as auto-aligning images but they seldom work and people and crowds are notoriously difficult to capture because of this problem of time.   However, here is also the trick!  If we do not even try to get rid of the multiple exposures when you create a HDR composite, the software calculates a value for all these different exposures. Specifically this happens when you get rid of the software’s own “auto-alignment functions” and “reduce ghosting options”. So when there is movement, this creates a ghosting effect that we see above.  This process, as far as I have experimented so far, is quite random.  You get some degree of control to the exposures of these images but the different shapes and forms that emerge are quite unpredictable. You can see some of some of the effects below from close-ups of the pictures where figures and forms break into each other and into occasional noise.

4)  DUOTONE: The rest is pretty simple. I wanted this series to be in duotone so the next thing I did was move the image to Photoshop and used a set of filters to achieve the effect I wanted.  Specifically, I did the following pretty standard Photoshop CS3 adjustments across the different images to get the desired effect:

–> duplicate layer
–> soft light, opacity 20-30%)
–> gaussian blur 20px
–> adjustments – black and white – with green filter
–> greyscale to duotone (light brown tone) to rgb color
–> adjust master saturation -30%

5) DODGING AND BURNING:  Finally, I used a Wacom graphics pad to paint over the original images to exaggerate some of the ghosting effects of the images and overall give it the surreal slightly dark atmosphere.  Specifically, what I did here was to us “Dodge – Highlights” and “Burn – Shadows” to get the specific effects that I wanted.  No major strategy here: this is building on a technique I have been developing for years of painting with light on images which can give rather interesting effects such as in this in one of my earlier series below.

Anyway, much more I could and will write here.  Things such as time of day, light conditions etc affect how this effect works.  Also we could use neutral density filters that also would allow you to further control light and exposure times with more precision.  There are multiple variables here that can still be experimented with and I am probably going to do the part II of this experiment in London when the weather gets dreary and colors grey.  Meanwhile, hoping to get this series exhibited soon — either solo or together with my crazy photographer friend from Northeast India in a joint exhibition about people caught up in spaces not of their own making.  He works with tribal borderlands and fragile border spaces; this series is about urban spatiality — somehow the contrast and the overlaps, we feel, would be a great mix.

This, I believe, is a good example of how classical photography, creatiuve use of software and a little bit of randomness can create effects that were perhaps not possible before through classical methods and can be rather effective to achive the artistic effect you are after.

[Tools used: Pentax k20d, Photomatix, Photoshop CS3 and a fair amount of Old Monk]

The idea is simple: I create a set of microprograms (I fondly call them ‘brats’!), each of which have their own selfish agendas, decision processes, survival rules, and their own I/O probes that constantly monitor the surroundings for resources they need (and perish soon if they don’t find them)… and then I spawn each variety of agent multiple times and see an overall behavior emerge.

The first set of such agents are already sweating their necks, making highly stupid decisions… and yet surviving, but my goal is to now take it a step further.

I now I want to go in two directions:

- train a GA or reinforcement learning based system to figure out the initial configurations.

- spread the agents across multiple computers on the web, thus creating a virtual cluster, an ecosystem that would be very interesting to study.

You may ask, why is this any different than Cellular Automata, the system that Stephen Wolfram brought back into fad through his book ‘New Kind of Science’? The fact is, though similarities may exist on the surface, a deeper inspection would reveal that my agents are far more inspired by ant colony / multi-agent based emergent behavior systems than CA — the only difference being, I am trying to far more flexible about the toolkit, instead of sticking with just pheromones!. Also, at the end of the day, I am just trying to learn my tools!

[Tools used: Ruby, RubyGems, MySQL, a lot of caffeine, and box dvd set of The Wire]

The first is that “things” (in technical language “identities” or “forms”) are the emergent outcomes of complex repetition of patterns through which complexity if formed. These can then be reverse engineered and computationally modelled so that, say, a fractal algorithm can be used to model complex patterns in nature. This is especially common today in some of the 3D and visual compositing software that we use today and all kinds of experiments with genetic algorithms and cellular automata etc – simple rules, complex outcomes.

The second is that human intelligence itself is made of, to a large degree, of such repetitions. This idea is rather much more elusive than the first but is one of the key presuppositions behind artificial intelligence research. Here the key question is that what algorithms could be used to model the way humans think and thus be used to guide machines to perform complex tasks. The philosophical implications of this are even more profound than getting a robot to recognize faces or clean a non-linear toilet bowl. That is, if human intelligence is, in fact, highly programmable, what then defines humans from machines? This goes two ways: machines-as-humans and humans-as-machines. In other words, AI defines rationality a certain way with certain presupposition of what logic, thinking and consciousness are and how they can be pragmatically simulated in computers. But as importantly, if we look at the concept of rationality and how it has been historically constructed, this has always presupposes a certain “image of thought” that has excluded all that would not fit into the sphere of rationality (intuitions, insanity, madness, illogic, spontaneity, absurdity ….). So how would we then understand the blurred boundaries of man and computer (as intelligent forms, which neither technically speaking are) and the human-computer assemblage that is making the old notions of rationality/humanity perhaps increasingly difficult to defend? Humans as (programmed?) repetitions: computers as programmed repetitions: natural intelligence: artificial intelligence: natural stupidity: artificial stupidity …

Anyway, all this probably seems rather abstract here. However, what was a concrete outcome of this philosophical babble was a project called AML (or what we like to tentatively call Aesthetics Mark-Up Language) that we are now currently trying to get our head around and develop. Similar to the more philosophical questions above, the idea here would be to experiment with how we detect patterns in certain visual styles such as certain genres in film etc. These patters could be then transposed/translated to other instances so that, say – yes, however preposterous this may sound – a Bollywood Film could edit a Hollywood film!

To perhaps see more what we mean by this, please find below an excerpt of some of the behind-the-scenes work-in-progress. The email conversation explains the idea better than I could re-write.

In film theory, for instance, the following elements are often talked about in the semiotics of the filmic image (well it is more complex but bear with me for the time being …)

1) Images;
2) Phonetics = speech;
3) Noise = background noise etc;
4) Text
5) Music.

For our purposes, we will probably have to stick to images for now. Tracking classical elements of film, in the beginning, is far too complex to get started with. Such as depth of field, montage / contrast etc. So what could be easily tracked to get started?

Let’s start with images. We could start off with the following variables:

- movement (speed of movement = speed of change in pixels?) This could be later used to analyze some rhythm of change.

- brightness and contrast (how would this be tracked = the relationship or average of pixels in any given location on the video?)

This could also later be use to analyze things such as harmony of composition, direction of lines in the mise en scene, etc. We would have to come up with a set of principles from art history and composition and see how these could be determined in the screen etc?

- color range (this would probably have to be RGB values in the image itself). This would probably move us into the realm of things such as monochromatic color schemes, bright colors, harmonious colors, contrasting / oppositional color … ie to use some notion of color theory to provide patterns in certain styles of video etc. I’ve
studied this in high school so will be fun to revisit some principles of classical painting.

So I suspect what we need to do is set up a very simple experiment / structure in place that can be developed and extended depending on need. In other words, we need to develop … AML (Aesthetics Meta Language) … a basic language structure that would describe what the variables are within any analyzed video. This language, I suspect, could be then developed into the interface between the language of aesthetics and the computer. Something like this:

//AML: “DEBBIE DOES DALLAS”

<contrast>
<high>134</high>
<low>12</low>
<average>58</average>
<mean>22</mean>
</contrast>

You get the point. The interesting thing about such an approach would be that we would be developing an entire syntactics of the aesthetics-computer interface that could be taken to any given direction we want. And then, of course, transposed back to how we would edit any clips or piece of media that we choose to run through the framework.

So how would this relate to the aesthetics machine? Basically, determining before hand the parameters for aesthetics is impossible. In film theory, there are some ideas such as structuralist semiotics that try to describe the language of different films but – as you know – aesthetics is notoriously difficult to pinpoint, especially when we are stuck with pixel-level pattern recognition on the computer. So what the aesthetics machine could be is the extension of the AML meta-language that would emerge from experimenting with as much visual material as possible. Say, we make AML public and get film-buffs and media-freaks to go through the entire ouevre of, say, German Porn genre to see what patterns emerge. These are then described in the wider structure of the AML language that we start developing So we get:

//AML: “DEBBIE DOES DALLAS” AESTHETIC STRUCTURE

<crossfade>
<number>12</number>
<duration>145</duration>
</crossface>

<contrast>
<high>134</high>
<low>12</low>
<average>58</average>
<mean>22</mean>
</contrast>

etc etc. Perhaps we could even open this language up so that some elements could be done analogically by volunteers (ie Amazon Turk Model) to further build the language beyond what is possible through pattern recognition. As this language develops, then the aesthetics machine will then merely be the instantiation and translation of these variables to other media material.

So if this approach would be the best, we need to develop / conceptualise three things:

1) The pattern recognition engine to get started experimentation with
simple data;

2) The syntactic structure that would describe any given visual material and that could be populated both by computers as well as analog humans;

3) The basic experimentations of how the AML language could be then
used to edit / re-mix some other media element.

So one reason why we are interested in Processing is exactly that it is the ideal platform for such experiments to emerge. In any case, watch the space here as our ideas eventually develop and we start getting the first concrete experiments up.

In this second experiment, my attempt is to slowly explore the aesthetics that one can achieve through minimal code. This was written in ten minutes, and is just a version 0.001 of what is possible. At the moment, each is driven by a random generator, and controlled by three variables… but that will be the first thing to change in the upcoming update. I am yet to decide on the real world pattern that I will feed to the algorithm, but I am getting inclined towards live video feed…