So in our never-ending quest for the Idea, we have been recently throwing around some ideas around AI, aesthetics and creativity. To understand this particularly strange one, however, I probably will need to provide a few words of background to the wider project that we are interested in. I will try to keep it simple. Basically, the problematic/question that I am working around has to do with the “image of thought” in today’s digital cultures. One of the key tropes we are seeing emerging today is that the universe is made of patterns; or “abstract machines” as I prefer calling them after the French philosopher Deleuze. Without getting into unnecessary theoretical complexity here, what this means can perhaps best be seen inflected and explained in two ways:

The first is that “things” (in technical language “identities” or “forms”) are the emergent outcomes of complex repetition of patterns through which complexity if formed. These can then be reverse engineered and computationally modelled so that, say, a fractal algorithm can be used to model complex patterns in nature. This is especially common today in some of the 3D and visual compositing software that we use today and all kinds of experiments with genetic algorithms and cellular automata etc - simple rules, complex outcomes.

The second is that human intelligence itself is made of, to a large degree, of such repetitions. This idea is rather much more elusive than the first but is one of the key presuppositions behind artificial intelligence research. Here the key question is that what algorithms could be used to model the way humans think and thus be used to guide machines to perform complex tasks. The philosophical implications of this are even more profound than getting a robot to recognize faces or clean a non-linear toilet bowl. That is, if human intelligence is, in fact, highly programmable, what then defines humans from machines? This goes two ways: machines-as-humans and humans-as-machines. In other words, AI defines rationality a certain way with certain presupposition of what logic, thinking and consciousness are and how they can be pragmatically simulated in computers. But as importantly, if we look at the concept of rationality and how it has been historically constructed, this has always presupposes a certain “image of thought” that has excluded all that would not fit into the sphere of rationality (intuitions, insanity, madness, illogic, spontaneity, absurdity ….). So how would we then understand the blurred boundaries of man and computer (as intelligent forms, which neither technically speaking are) and the human-computer assemblage that is making the old notions of rationality/humanity perhaps increasingly difficult to defend? Humans as (programmed?) repetitions: computers as programmed repetitions: natural intelligence: artificial intelligence: natural stupidity: artificial stupidity …

Anyway, all this probably seems rather abstract here. However, what was a concrete outcome of this philosophical babble was a project called AML (or what we like to tentatively call Aesthetics Mark-Up Language) that we are now currently trying to get our head around and develop. Similar to the more philosophical questions above, the idea here would be to experiment with how we detect patterns in certain visual styles such as certain genres in film etc. These patters could be then transposed/translated to other instances so that, say - yes, however preposterous this may sound - a Bollywood Film could edit a Hollywood film!

To perhaps see more what we mean by this, please find below an excerpt of some of the behind-the-scenes work-in-progress. The email conversation explains the idea better than I could re-write.

In film theory, for instance, the following elements are often talked about in the semiotics of the filmic image (well it is more complex but bear with me for the time being …)

1) Images;
2) Phonetics = speech;
3) Noise = background noise etc;
4) Text
5) Music.

For our purposes, we will probably have to stick to images for now. Tracking classical elements of film, in the beginning, is far too complex to get started with. Such as depth of field, montage / contrast etc. So what could be easily tracked to get started?

Let’s start with images. We could start off with the following variables:

- movement (speed of movement = speed of change in pixels?) This could be later used to analyze some rhythm of change.

- brightness and contrast (how would this be tracked = the relationship or average of pixels in any given location on the video?)

This could also later be use to analyze things such as harmony of composition, direction of lines in the mise en scene, etc. We would have to come up with a set of principles from art history and composition and see how these could be determined in the screen etc?

- color range (this would probably have to be RGB values in the image itself). This would probably move us into the realm of things such as monochromatic color schemes, bright colors, harmonious colors, contrasting / oppositional color … ie to use some notion of color theory to provide patterns in certain styles of video etc. I’ve
studied this in high school so will be fun to revisit some principles of classical painting.

So I suspect what we need to do is set up a very simple experiment / structure in place that can be developed and extended depending on need. In other words, we need to develop … AML (Aesthetics Meta Language) … a basic language structure that would describe what the variables are within any analyzed video. This language, I suspect, could be then developed into the interface between the language of aesthetics and the computer. Something like this:

//AML: “DEBBIE DOES DALLAS”

<contrast>
<high>134</high>
<low>12</low>
<average>58</average>
<mean>22</mean>
</contrast>

You get the point. The interesting thing about such an approach would be that we would be developing an entire syntactics of the aesthetics-computer interface that could be taken to any given direction we want. And then, of course, transposed back to how we would edit any clips or piece of media that we choose to run through the framework.

So how would this relate to the aesthetics machine? Basically, determining before hand the parameters for aesthetics is impossible. In film theory, there are some ideas such as structuralist semiotics that try to describe the language of different films but - as you know - aesthetics is notoriously difficult to pinpoint, especially when we are stuck with pixel-level pattern recognition on the computer. So what the aesthetics machine could be is the extension of the AML meta-language that would emerge from experimenting with as much visual material as possible. Say, we make AML public and get film-buffs and media-freaks to go through the entire ouevre of, say, German Porn genre to see what patterns emerge. These are then described in the wider structure of the AML language that we start developing So we get:

//AML: “DEBBIE DOES DALLAS” AESTHETIC STRUCTURE

<crossfade>
<number>12</number>
<duration>145</duration>
</crossface>

<contrast>
<high>134</high>
<low>12</low>
<average>58</average>
<mean>22</mean>
</contrast>

etc etc. Perhaps we could even open this language up so that some elements could be done analogically by volunteers (ie Amazon Turk Model) to further build the language beyond what is possible through pattern recognition. As this language develops, then the aesthetics machine will then merely be the instantiation and translation of these variables to other media material.

So if this approach would be the best, we need to develop / conceptualise three things:

1) The pattern recognition engine to get started experimentation with
simple data;

2) The syntactic structure that would describe any given visual material and that could be populated both by computers as well as analog humans;

3) The basic experimentations of how the AML language could be then
used to edit / re-mix some other media element.

So one reason why we are interested in Processing is exactly that it is the ideal platform for such experiments to emerge. In any case, watch the space here as our ideas eventually develop and we start getting the first concrete experiments up.