SimpleMovieX and the iPad

First, a comment about my predictions: I got them plain wrong.
As many other commentators, I have overestimated the weight of the Technical in innovation: Setting aside the A4 processor, a true technical barrier of entry for competitors, the iPad is first and foremost a device that stands on the shoulders of a Giant: the iPhone.

Here I consider the whole iPhone ecosystem as the sum of the “Touch” user interface and the AppStore and its 140,000 existing applications. For developers, it’s indeed the same platform.

The iPad re-purposes the iPhone ecosystem into the future mainstream utility to get things done. The magic is in simplifying things to make them iPhone-like, and the genius is in artificially restraining some technical capabilities.

Your mythical grand-ma cannot buy or use a computer alone, but she downloads and uses apps on her iPhone, alone. That’s the whole differentiating point of the iPad, and because it’s not a computer, Apple has met its goal to give whole new demographics their Internet-age appliance.

Now, let’s go back to SimpleMovieX. I have no doubt that the iPad will be a successful platform, that eventually will displace current platforms. That is definitively a place I will be unless I want my business to address only niche technical markets ten years from now.

SimpleMovieX is a lightweight video editor. From the scope point-of-view, it can be a good fit for the iPad, that is primarily a consumption and lightweight creative device. SimpleMovieX uncluttered user interface could easily be transposed to iPad screens.

The first problem is how you get your data into your iPad. Movies syncing through iTunes works well, but it is not the channel that people will use to import content that they want to modify. According to published information, the iPad doesn’t have a built-in camera, nor does it connect directly to video cameras.
The iPad seems too disconnected from video production workflows and from video devices to be a place for SimpleMovieX to live. At least today.

From the technical stand point, it looks really bad too: All indicates that the QuickTime framework that powers SimpleMovieX is not available on the iPad. Instead, the modern and efficient QuickTime X, used in the iPhone and iPad for audio and video playback, supports very few codecs and formats, and have near-zero editing capabilities.
This will improve over time, for sure, but Apple is like a car with no reverse gear: QuickTime X will grow towards the future, not towards ensuring full backwards compability with legacy QuickTime.

In other words, if SimpleMovieX someday exists on the iPad platform, it will have nothing in common with today’s SimpleMovieX. Except maybe the skin and purpose. Time will tell.

The Everest of movie repairs

From time to time, I receive an Everest-class repair.
Mount Everest, the highest summit on earth, can barely be climbed by men. Only an elite of alpinists can reach the 8848m without help of oxigen.
In some way, it is measuring the capacity of humans. A bit higher and it would be physiologically impossible to climb.

Similarly, every two or three months, I receive a movie repair that is at the edge of impossible. It has this rare quality of being too damaged to be repaired with known techniques, but at the same time I foresee that there is a small chance that a new technique may fix it.

Officialy, I don’t call it Everest-class, but rather Investigative Repair, to convey to the customer three important ideas:
- I have to develop something radically new. A unique solution for a unique case.
- it’s gonna be expensive (I charge usually one order of magnitude more for those repairs)
- at the end, I can come to the conclusion that the repair is impossible. Results are not guaranteed.

I do such repairs primarly to challenge my repair ability and to push the enveloppe of the discipline. As boring as repairing movies may seem, an Everest-class repair is when there’s adventure, struggle and achievement.

The last Everest-class repair came last month: Canon EOS 5D Mark II files with some bitstream corruption. Mr Szot, the polish director, did not notice the problem until the production was finished. The clips, needed for a 15 minutes short movie called “An Exclusive”, could not be shot again.

I observed around fifty defects in a dozen of files. The footage is encoded in H264, where one wrong bit can propagate nefarious effects over a dozen of frames. Even with “creative editing”, it would be impossible to tell the story: For some important takes, the footage was unusable.

In 99% of the repairs, the problem consists in re-indexing the clip: Audio and Video media is fine but the “table of contents” that tells where the data for each frame, is missing.
But here, it’s the opposite. The table of contents is fine, but the media is corrupt.

Not that corrupt, according to real world standards, since the amount of bad data is only 1 for 100 millions. Like a rare disease that would affect just 70 people in the world population. The typical needle-in-a-hay-stack problem. Change needle for bits and haystack for a file with one billion bits and you get the idea.

Bits Flipping Party

The repair technique lies on a simple idea: when decoding the frame, the wrong bit will cause errors and exceptions that eventually stop the decoding process. That is what we observe in a defective frame: the top is ok, then something occurs, then the rest of the frame is not decoded.
The location where it stops should be close to the wrong bit. Never before, at a short distance after.
From here, we will go backward and flip the bits one by one, until we get something that decodes without error.

I made a bold assumption here: that wrong bits are extremely rare. So rare that we can consider that there is only one wrong bit involved in every decoding error.
I had nothing to really back this assumption: If it’s true, we would be able to repair. If it’s wrong, it would be impossible. Checking the assumption thus became a priority before engaging into time consuming developments.

There’s a second assumption: The distance between the wrong bit and the error is small.
One iteration will be needed for every bit, and iterations are slow since they decode several video frames each. Not just one frame, but the whole group from the I keyframe to the damaged frame. That can take several seconds, so if we have thousands of iterations, it would become impracticable.

Prototype and Automation

I assembled a prototype from various pieces, an open-source H264 decoder, a couple of small programs to flip bits and to detect errors and generate pictures from potentially successful iterations.
I tried it on a first defect, a slow process since almost everything had to be done by hand, and results have to be carefully interpreted.
When I finally managed to get a good picture out of the prototype, it was unbelievable. I had found the needle.

Since I had almost fifty defects to fix, I spent some time automating the process. At the end, I would only have to send a few command-line commands, choose manually the starting point of the search, and launch it.
The computer would run for minutes, sometimes hours, until it starts spitting pictures.
I had to review the pictures one by one, until I could find one that was perfect.
Then, I would modify by the wrong bit in the movie and verify that it fixed the frame and also the rest of frames in the “Group of Pictures”.

In some cases, fixing a frame would just unveil a defective one a few frames later. Just as one train may hide another…

Finally, I managed to clean completely all but one defect. This was quite a surprise. I would never have anticipated such a desperate repair attempt to work with a 98% success rate.

My customer, Mr Szot, is happy. His film “An Exclusive” will be presented this week to the polish public.

A force-sensitive back-side interface for the Tablet?

Yesterday’s post was the fruit of trying to imagine how a tablet could work, from the bare usability standpoint, and I came to the conclusion that the device would be operated from the back side with a “touch sensitive” surface where your fingers would play.

Today, by doing casual research on “back side touch” keywords, I found a very interesting 2007 patent filing by Apple:
Back-Side Interface for Hand-Held Devices

The patent is credited to Apple engineer John Elias. This guy is the founder of Fingerworks.com, a company dedicated to develop devices than can be controlled by gestures. Apple hired him 5 years ago, and surprise! the website has been shut down this week…

Timing is perfect for a 27 January presentation.
I don’t intend to convert this blog into a rumors place, but God! speculation is exciting!

My Tablet

With 2 weeks missing for Apple Tablet presentation, speculation is ramping up.
As smart commentators point out, there is a major problem with such a device:
How would you use it, from bare usability stand point?

Either it’s on resting on a flat surface, and you have unacceptable ergonomics, with your hands hiding the screen and your neck bent at an angle that your EHS department won’t permit.
Or you’re holding it in your hands on front of you. But how do you use the touch screen if your fingers are behind the tablet?

Apple will for sure come with the solution, that will feel so natural and simple that all speculations like mine will look a bit stupid two week from now.

My take on the Tablet:
Why not make the back side of the tablet a touch sensitive surface and show the fingers as an overlay of the user interface?

Fingers that are merely resting on the surface would show a small, transparent “finger print” while the active finger would be more visible, for example with a black circle representing the pressure applied.
This would enable multi finger gestures, and give natural feedback, both visual and tactile.

Such a device with a screen on the front side and a tactile surface on the back side would work well in the two natural positions:
- Held with both hands, mid-air
- Held with both hands, but with hands resting on a table. This one provides a nice 45 degrees angle, and it’s comfortable for long sessions (like watching a movie).

I am not excluding that the screen would also be tactile.

What about the keyboard?
It’s hard to imagine interacting with a software keyboard operated from behind the screen, but why not? With the appropriate visual feedback, it seems plausible.

Such a device layout has another advantage: the touch surface doesn’t need to be exactly as big as the screen, it can be dimensioned to work well with standard hands size.

If ever it comes true, remember that you’ve read it here first!