Everything is Miscellaneous

Thanks to Joyce Shintani for blogging this. I found it relevant to the mpeg7 discussion, among other things that I like in general about interenet culture.

Let The Games Begin?

Hi all! I'll be taking a somewhat different tack in my contributions here because of my different background. I am a social worker, with some backgound in neuropsychology and cognitive science from my undergraduate experience. I am also a phenomenologist and attempted writer, but have very limited knowledge of cinema per se. My primary interest in these technologies are:

a) in the feedback mechanisms which could potentially be used to incorporate viewer response into the process of "recomposing" quantum films
b) the subjective experience of viewing such a "self-composed" film
c) the social consequences of the loss of conventional shared narratives
d) applications of quantum cinema outside "art"

I want to start off this conversation by saying that quantum cinema already exists and it is called video gaming. Whether Pac Man qualifies for the medium is an academic discussion that is hardly relevant at this point. The new wave of game systems offer games in which characters can be designed, rendered somewhat realistically, and directly controlled to move within certain specifications. This creates virtually limitless possibilities of narrative in a sense, although games are almost universally unsatisfying as stories.

One reason is that the stories tend to bottleneck at certain points (usually during the preconcieved "cinematic" sequences during which the character does not have control). This creates the existencial conundrum of having absolute control but no optins.

A second reason is that the system gives too much control and predictability for the brain to translate into an emotional response. Most intense experiences I have had with cinema have been accompanied by the intense feeling of not being able to control outcomes and therefore being dependent on the story. One good example is watching an episode of "Curb Your Enthusiasm" and becoming painfully uncomfortable with the actions of the characters, feeling for example the intense need to apologize to the other characters for the actions of Larry David while at the same time feeling a great deal of sympathy for/with Larry David because of the actions of his wife.

The reason linear cinema is so succesful is that it allows the film maker to speak directly from the story and therefore "make a statement". If you ask a videogame player (full disclosure: I am one, I especially love shooters) what makes a game great, the most common answer is not the story, the music, the graphics, or even the design of the maps or the artificial intelligence. It is something called The Engine, the invisible but explicitly felt algorithms which determine the ways in which the manipulation of the controls translates into modifications of the game world. The well known Halo franchise, for instance, won out largely on the strength of it's engine. Will fans of cinema one day stop talking about stories and start talking about engines?

Quantum Camera Components

The quantum camera has a brain and ears.

Speech recognition software performs best when you train the software by speaking a series of important words and sentences. It seems to me that the technology already exists to make prototypes of a system that would embed videos with full text transcriptions of everything spoken on camera as metadata. Nevermind the fact that if what you were shooting had a screenplay you could feed the entire screenplay to the camera before you shoot so it can anticipate what the actors will say.

Techniques such as dynamic time warping would accomodate the variations between different takes of the same shot so that they are all regarded as versions of one another. Seems like mpeg7 would be the best possible way to store this data today.

Frameline 47 lets you take advantage of the mpeg7 standard quite fully and its not surprising at all that a company like Eptascape would be in the security and surveillance market. What we need is the metadata to be entered half-automatically during the shooting process combined with a learning curve for the camera where you have it learn what the actors look like and sound like. A sophisticated infra red camera would help define actors from the background. Using three cameras (one on the X, Y and Z axis) could help to define the physical space and depth. And now a dog in infrared:

Wrinklers In Time

I have just been researching wormholes, string theory and Stephen Hawking's chronology protection conjecture.

Quantum editors will be called tesseracters because the medium they compose with will have four dimensions instead of two.

The Tyranny of The Frame

I have been thinking about Greenaway's "Tyranny of the Frame". It is pefectly acceptable to me that we should have a frame, whatever the format of the rectangle. I also have nothing against those artists who like to project on round things or whose fetish it is to go beyond our peripheral vision in 360° panorama.


The real tyranny of the frame of celluloid cinema is that it is in two dimensions. Above is an animated projection of a rotating tesseract. Why should the video frame be flat?

The camera of the last 100+ years of cinematic history is essentially a glorified eyeball with sophisticated spectacles.

The Quantum Camera will attach these free-floating, bespectacled eyeballs to brains. Brains capable of perceiving reality more like the way our nervous system works. When I was at University of Maryland I had a class about visual communication where the professor had us read a book about visual perception. Humans have depth perception. We can tell the foreground from the background without any trouble, and if something that was moving ceases to move, we can still distinguish it as a separate entity. We have had multi-track audio recording equipment for years. We need multi-depth video cameras. Cameras that record the background and the foreground to separate layers of video. Goodbye keying and matting, hello alpha channels!

But it needs to go further than just what keying can accomplish. Layers are still a 2D concept. Containers (I admit I borrow the word from conversations I have had with Philipp) are a much more appropriate model. Here is a film still from Lost In Translation:


The quantum camera would ideally see (at least) these containers:
(Restaurant (Charlotte) (Table (Food) ) (Steam) (Bob) )

Of course our eyes can discriminate an incredible level of detail:
(Bob (Costume (Sweater (Shirt) ) (Wristwatch) (Pants) )

And we can even infer things which we cannot see, such as socks, underwear and shoes.

It may also be necessary to "teach" the quantum camera in order to get it to learn to recognize these containers and their IDs. With the above example, I can imagine that it would work that first you would show the camera the empty table and benches and ID it Restaurant. Then you could put the food on the table and ID it Food. Then a threshold knob would be adjusted to catch the steam and ID it Steam. Lastly Bob and Charlotte would each be added to the composition and IDed respectively. Or perhaps you could use a combination of RFIDs and threshold settings for brightness, depth and movement on the camera.

When I was at Bard I attended a screening of Let's Get Lost by Bruce Weber (see this site for video clips). The screening was presented by Bard alumni Jeff Preiss, an accomplished cinematographer. With a successful career in commercial advertising, Jeff knew some secrets of the image industry. He mentioned that he had heard of major corporations developing cameras that photograph all surfaces of physical reality in the hope of creating photographic 3D space.

What are they waiting for?