Object Recognition using Kinect on the PC

If this kind of thing is possible with a live video stream it should be possible to have seamless hardware and software integrations in the future. Either the camera sees objects and writes semantic objects into video metadata while recording, or video could be processed after being recorded.

One Step Closer to Universal EDL

Last week I attended the Mozilla Drumbeat Festival in Barcelona. It gave me an opportunity to collaborate with an amazing ad hoc team of people in the context of the Open Video Lab, chaordinated by Brett Gaylor and David Humphrey. Together over the course of a two day sprint, a big team of us collaborated on a demo of the popcorn.js javascript library that really shows off the potential beauty of web made movies. The vimeo video above is just a screen capture; for the live demo visit this page.

Photo by Samuel Huron BY-NC-ND

It was a very rewarding experience to contribute to the aesthetic and conceptual process. I enjoyed the challenge of conducting interviews in languages I don't speak, and collaborating with the multilingual Xabier Cid on the editing process. I was honored to be able to address the audience at the "BEST of the FEST closing variety slam showcase" for the need for new approaches to film school in the face of scrum/agile approaches to storytelling.



What is great about the demo is how it utilizes time-coded metadata to retrieve live content from flickr and twitter in real time. It shows how as we move towards an object-oriented moving image we will continue to redefine what cinema is and also our notion of editing. The tweets are aggregated from the #futureofeducation hashtag. The flickr photos that appear in the demo are called based on timeline metadata that I approximated by putting dummy content (the blue events in the screenshot above) on the timeline to get a sense of a rough rhythm. I then gave a rough approximation of that timecode information to Berto Yáñez, the programmer who did much of the heavy lifting on the demo. Oscar Otero helped with the design of the page. Oscar, Berto and Xabier all work together at the Galician web company A navalla suíza.

Photo by Homardpayette

This process, which involved swapping lots of data across computers via USB sticks, underscored the need for a Universal Edit Decision List (EDL). This was something I identified  about a year ago as part of my rubric for open source cinema. The Universal EDL got discussed quite a bit during the video lab, and together with the amazing work that's already been done creating a web-based timeline interface with Universal Subtitles, it seems like the seed of inspiration to take things a step further has been planted. I am very excited to have contributed to these developments towards an object-oriented open source cinema!

It would be great to see all the names of the participants in the workshop added to the demo. During the demo Laura Hilliger, David Humphrey and I put together a nice cloud-based credit concept for solving the dilemma of crediting multiple parties with multiple credits. Laura should have a rough list of names and roles, and those who are missing could use the #drumbeat #videolab hashtags on twitter to ID themselves, or comment on the video, so we can round everybody up.

Tesseracting in Action

Now let's see them grab the sound too!
Thanks to notthisbody for the link.

Postmodern vs Hypercubist

PhotoSketch: Another Tool for Atomizing the Frame

There's some great controversy stirred up over on Vimeo about whether this is fake or not. Regardless of whether it's legitimate science or great science fiction, this video is another beautifully clear example of how we are moving towards an object-oriented image making paradigm.

Volumetrically Enhanced Dance Performance from United Visual Artists

This subtle and elegant dance performance from United Visual Artists uses some type of volumetric recording technique to generate a real-time object-oriented video backdrop. It's not surprising that the data points are being displayed in a similar aesthetic to the House of Cards video which I love talking about so much; photo-realistic data sets would be a burdensome load for a real-time or even non-real-time process.

Anyone have a prediction on how soon we'll see photo-realistic real-time volumetrics? Let's start a wager! ;)

Thanks to my friend and colleague Carsten Goertz for the link to the video!

Object-Oriented Byproducts of Image/Video Manipulation

These technologies will simultaneously help increase the demand for volumetric cinematography while being made obsolete by its widespread arrival.

Cubism vs. Hypercubism

I thought I would take a stab at a concise definition of Hypercubism, a word I use quite often and have until now perhaps not defined so exactly. For the sake of reference, here is an excerpt of Wikipedia's definition for Cubism:

In cubist artworks, objects are broken up, analyzed, and re-assembled in an abstracted form — instead of depicting objects from one viewpoint, the artist depicts the subject from a multitude of viewpoints to represent the subject in a greater context. Often the surfaces intersect at seemingly random angles, removing a coherent sense of depth. The background and object planes interpenetrate one another to create the shallow ambiguous space, one of cubism's distinct characteristics.

In contrast we might consider this summation of Hypercubism:

In hypercubist artworks, objects are particlized, analyzed and synthesized in a realistic form — instead of depicting all objects from one temporal perspective, the artist (or artists) depict the subject from a multitude of temporal perspectives to represent the subject in a greater temporality. Often the surfaces of intersect seamlessly, creating a coherent four-dimensional spacetime illusion. The background and object planes are always distinct to create deep concrete space, one of hypercubism's distinct characteristics

Besides being a bit of encyclopedic revisionism, these two definitions set up a useful theoretical dichotomy between Cubism and Hypercubism. Simply put:

Cubism shows multiple spaces in the same time while Hypercubism shows multiple times in the same space.

200 Cameras, 2.5 million frames and 20,000 Gigabytes worth of Toshiba Magic

Although we've seen this effect used since The Matrix in numerous advertisements, films and spoofs, this example reveals in its reduced, contemporary aesthetics the object-oriented nature of the hypercubist revolution on its way. The subtle interplay of the different players in this "time sculpture" (as the ad agency people call it) reveals the short-comings of 2D compositing while also hinting at an entirely different image world.

Of course my contention is that the methodology used in this process is already theoretically obsolete. I predict that the scanning technologies of the future should be able to yield clouds of photo-realistic data without the necessity of 200 distinct video cameras. Nonetheless, state-of-the-art Californian motion capture company MOVA is also using a highly complex strobing multi-camera system together with phosphorescent makeup.



Their white paper on volumetric cinematography is a great quick read, written in accessible, non-scientific language, and resonates strongly with my own hypercubist theory.

These developments surely must be fascinating for the early pioneer of blue-screen proto-hypercubism, Zbig Rybczynski. His 1980 film Tango was an essential forerunner of such object oriented imagery.

The concept sketches, diagrams and film still from the finished composite of Tango reveal a direct hand-crafted analog predecessor of the Toshiba spot;

The Explosion of Cinematic Time

This coy video from Dan Goldman of Adobe Systems alludes to some pretty fundamental concepts of what an object-oriented cinema might entail;

• Drawing on Objects
  (making graphic changes to objects which stay with the object in time, as opposed to simulating object-oriented change by altering every frame)
  
  
• Delineating Paths of Objects
  (tracking and displaying an objects path through space)
  
  
• Attaching Visual Metadata to Objects
  (text annotations in the form of cartoon speech bubbles)
  
  
• Object Timeline Scrubbing
  (using a "click and drag"approach to scrub the timeline)
  
  
• "Throwing" Objects
  (giving objects the ability to be manipulated using analog velocity controls)
  
  
• Segmentation of Objects
  (enabling puppet-like effects)
  
  
• Hypercubist Time
  ("drag and drop" manipulation of individual objects through their own timelines to create composite hypercubist time)

Dan explains that his software system analyzes the movement of points and allows the system to recognize moving objects. The video also demonstrates how algorithmic analysis will be able to accurately calculate hypercubist compositing to eliminate visual artifacts.

The curious and persistent question in my mind is: when will this hypercubist vision for moving images be embraced by the video camera manufacturers? Why not displace some of the processing power needed to identify the objects into the camera system itself? There are certainly high-end special effects tracking systems for the commercial industry to do all kinds of compositing and layering of 3D and real images, but I think lower price point systems have a genuine appeal for numerous less glossy applications. It seems like Dan is already on this tip in his experiments with puppets.

The Dynamic Graphics Project at the University of Toronto has also been contributing to this field with its Dimp video player prototype which allows to browse video clips by directly dragging their content.



These interface prototypes are a great step forward in establishing hypercubist video aesthetics. I think 2009 looks auspicious for these projects. With YouTube's recent launch of annotations even with a silly example like this its not hard to imagine more and more regular consumers demanding sophisticated object-oriented video tools.

Bill Viola Interview and Marc Aschenbrenner's Zweite Sonne

I like Bill Viola's comparison of video cameras to reindeer bones with notches cut into them. He is profound in his assertion that its the telling of stories and leaving of objects that is the fundamental activity of human existence.



I recently attended the Düsseldorf and Köln art fairs and it was transparently clear that the contemporary art world is dominated by a marketplace driven by the collection of objects. What little video I did see I found remarkably boring and derivative, with the exception of Zweite Sonne by Marc Aschenbrenner on exhibit at the Olaf Stüber Gallery booth in Köln. It exhibited a property I would like to associate with my slowly developing theory of Quantinuity; namely that it transported an otherwise virtual object into physical space. The video was projected inside a giant black plastic balloon-suit that also appears in the video itself as the central subject of the video.