Similarly, every two or three months, I receive a movie repair that is at the edge of impossible. It has this rare quality of being too damaged to be repaired with known techniques, but at the same time I foresee that there is a small chance that a new technique may fix it.

Officialy, I don’t call it Everest-class, but rather Investigative Repair, to convey to the customer three important ideas:
- I have to develop something radically new. A unique solution for a unique case.
- it’s gonna be expensive (I charge usually one order of magnitude more for those repairs)
- at the end, I can come to the conclusion that the repair is impossible. Results are not guaranteed.

I do such repairs primarly to challenge my repair ability and to push the enveloppe of the discipline. As boring as repairing movies may seem, an Everest-class repair is when there’s adventure, struggle and achievement.

The last Everest-class repair came last month: Canon EOS 5D Mark II files with some bitstream corruption. Mr Szot, the polish director, did not notice the problem until the production was finished. The clips, needed for a 15 minutes short movie called “An Exclusive”, could not be shot again.

I observed around fifty defects in a dozen of files. The footage is encoded in H264, where one wrong bit can propagate nefarious effects over a dozen of frames. Even with “creative editing”, it would be impossible to tell the story: For some important takes, the footage was unusable.

In 99% of the repairs, the problem consists in re-indexing the clip: Audio and Video media is fine but the “table of contents” that tells where the data for each frame, is missing.
But here, it’s the opposite. The table of contents is fine, but the media is corrupt.

Not that corrupt, according to real world standards, since the amount of bad data is only 1 for 100 millions. Like a rare disease that would affect just 70 people in the world population. The typical needle-in-a-hay-stack problem. Change needle for bits and haystack for a file with one billion bits and you get the idea.

Bits Flipping Party

The repair technique lies on a simple idea: when decoding the frame, the wrong bit will cause errors and exceptions that eventually stop the decoding process. That is what we observe in a defective frame: the top is ok, then something occurs, then the rest of the frame is not decoded.
The location where it stops should be close to the wrong bit. Never before, at a short distance after.
From here, we will go backward and flip the bits one by one, until we get something that decodes without error.

I made a bold assumption here: that wrong bits are extremely rare. So rare that we can consider that there is only one wrong bit involved in every decoding error.
I had nothing to really back this assumption: If it’s true, we would be able to repair. If it’s wrong, it would be impossible. Checking the assumption thus became a priority before engaging into time consuming developments.

There’s a second assumption: The distance between the wrong bit and the error is small.
One iteration will be needed for every bit, and iterations are slow since they decode several video frames each. Not just one frame, but the whole group from the I keyframe to the damaged frame. That can take several seconds, so if we have thousands of iterations, it would become impracticable.

Prototype and Automation

I assembled a prototype from various pieces, an open-source H264 decoder, a couple of small programs to flip bits and to detect errors and generate pictures from potentially successful iterations.
I tried it on a first defect, a slow process since almost everything had to be done by hand, and results have to be carefully interpreted.
When I finally managed to get a good picture out of the prototype, it was unbelievable. I had found the needle.

Since I had almost fifty defects to fix, I spent some time automating the process. At the end, I would only have to send a few command-line commands, choose manually the starting point of the search, and launch it.
The computer would run for minutes, sometimes hours, until it starts spitting pictures.
I had to review the pictures one by one, until I could find one that was perfect.
Then, I would modify by the wrong bit in the movie and verify that it fixed the frame and also the rest of frames in the “Group of Pictures”.

In some cases, fixing a frame would just unveil a defective one a few frames later. Just as one train may hide another…

Finally, I managed to clean completely all but one defect. This was quite a surprise. I would never have anticipated such a desperate repair attempt to work with a 98% success rate.

My customer, Mr Szot, is happy. His film “An Exclusive” will be presented this week to the polish public.

For Windows users, I have recently found a good technique to create a disk image from the damaged disk or card, which is the first step towards recovery of footage.
Note that this can also be useful to Mac users if the disk is not visible in the Finder, but is visible on a Windows PC.

In a nutshell, we use a small utility called dd for Windows that is able to read the raw data of a disk. (From now on, I will just use the word disk, and it can refer to a hard disk, a memory card or a USB drive)

Under the premise that the disk is readable, ie Windows detects it when you plug the disk or insert the card in the reader, “dd for Windows” should be able to create a file with about the size of the disk.

Here is the process, step by step:

http://www.chrysocome.net/downloads/dd-0.5.zip

This program is free and distributed under GPL license. For more information, refer to the product page.

dd --list

And you will get a list of the devices connected to your computer.
Here is what I get for example:

rawwrite dd for windows version 0.5.
Written by John Newbigin
This program is covered by the GPL. See copying.txt for details
Win32 Available Volume Information
\\.\Volume{aef46cf9-3e3d-11de-b8c6-806d6172696f}\
link to \\?\Device\HarddiskVolume3
fixed media
Mounted on \\.\c:

\\.\Volume{aef46cf8-3e3d-11de-b8c6-806d6172696f}\
link to \\?\Device\CdRom0
CD-ROM
Mounted on \\.\d:

\\.\Volume{6f41f4b2-d11a-11de-b318-001d4f88486c}\
link to \\?\Device\Harddisk1\DP(1)0-0+5
removeable media
Mounted on \\.\f:

Don’t be afraid, it’s just a list of the devices connected to your computer.
Here the interesting device is the F:\ volume described as removeable media: this is the card that I want to recover. Take note of the corresponding volume name:
\\.\Volume{6f41f4b2-d11a-11de-b318-001d4f88486c}

This volume name starting with \\.\Volume{ and ending with } will be used in step #6.

dd if=\\.\Volume{6f41f4b2-d11a-11de-b318-001d4f88486c} of=c:\usb.img bs=1M

This command creates a file called c:\usb.img that is a carbon-copy of the damaged disk.
The operation can take a long time. For 1GB, it can take one minute or more. For 120GB, it will take several hours.

Note that you can monitor the size of the c:\usb.img file as it grows from 0 to the size of the disk: It will give you an idea of the progress.

What do I do with the .img file?
Now if everything went fine, you should have a .img with a size similar to the disk capacity.
If the disk still contains footage, the footage must now be inside this file also, and we can do a diagnostic of the file and hopefully, repair it.

For Mac users, Diagnostic and Repair have already been covered in a previous post.
For Windows users, the .img file can now be used to create an extract that will be uploaded to Aero Quartet. This is explained in the Movie Repair Guide in the Windows page.

The first and most important thing is to prevent the file from being deleted by the system. As it’s a temporary file, rebooting the computer could delete the file.

Do not reboot your computer!

We will find the file and move it to a safe place. The location of the file depends on the volume on which you were recording:

In Boot volume:
Boot volume is usually your internal hard disk. It’s the disk that contains the operating system and your home folder.
In this case, the movie file is located in /private/var/folders/

This folder is not visible in the Finder. It’s hidden because you are not supposed to store your data here. But there’s a way to explore this folder anyway:

From the Finder, use menu Go > Go to Folder … and type /private/var/folders/

A Finder window opens, revealing a few subfolders. Now you can search inside, until you find one or several files with extension .mov and whose date of creation matches the QuickTime recording incident.
Just move those files to the Desktop. Once on the Desktop, the files are safe.

In an external drive, or a volume that is not the boot volume:
In this casem the file should be in this folder:
/Volumes/nameOfExternalDisk/.TemporaryItems/folders.501/TemporaryItems/


Note the . before TemporaryItems, it makes the folder invisible in the Finder.
You must use Go > Go to Folder… and type the path like this:
/Volumes/nameOfExternalDisk/.TemporaryItems/

Do not forget to replace nameOfExternalDisk by the real name of the volume!

Then explore the subfolders until you find the .mov files whose creation date correspond to the QuickTime session, and move them to a new folder on the same volume.

Now the files are in a safe place. You can reboot your computer if needed.

As the recording process was interrupted, the files are not playable: QuickTime gives an error message when you try to open them.
Fortunately, there is Treasured, and you can diagnose and preview the file.

• Repair less files: Use Treasured to preview your damaged videos. If only one clip is really important to you, and if it’s less than 500MB in size, then it will cost you only $89 (Local Repair)
• Repair smaller files: Only files under 500MB apply for a cheaper Local Repair. If your file is bigger than that, but you are only interested in fixing a segment of the movie, then you can use Treasured to extract the segment, and request a repair only on this segment.

  Utilities > Sample Extraction… in Treasured menu, creates a file called extract in your repair folder. Rename it and run Treasured on it.

• Specify that you want only Audio or only Video: Usually, synchronization of audio and video accounts for a important slice of the final cost. Treasured lets you uncheck audio or video if you feel it’s not necessary.
• Don’t check Absolute Privacy if you don’t need it. Regular level of privacy is good enough for the vast majority of repairs. Absolute Privacy commands an extra cost.
• Be patient… Repair costs will go down as repair techniques improve. Check the repair cost again after 3 months. Maybe it’s significantly lower.
]]> http://aeroquartet.com/wordpress/2009/06/26/5-tricks-to-pay-less-for-your-movie-repair/feed/ 0 http://aeroquartet.com/wordpress/2009/06/17/your-best-ally-for-corrupt-card-recovery-disk-utility/ http://aeroquartet.com/wordpress/2009/06/17/your-best-ally-for-corrupt-card-recovery-disk-utility/#comments Tue, 16 Jun 2009 22:07:15 +0000 Benoit Joossen http://www.aeroquartet.com/wordpress/2009/06/17/your-best-ally-for-corrupt-card-recovery-disk-utility/ When your memory card gets corrupt with tens or hundreds of megabytes of footage, video clips or movies, you have not one but two problems:
• Extract data from the magma that has now become your P2 card or your SDHC. (I told you not to buy the cheapest!)
• Organize this data into playable movies.

For the second problem, I have the solution: Treasured for diagnostic, then a Repair Kit that I develop for you.
But for the first one, until recently, I could just recommend to use a data recovery software. Stuff relatively expensive, and that would give average-to-good results only: Data Rescue 2, FileSalvage, Stellar Phoenix 3, Boomerang, UnDelete Plus, Recover4all, …

I also would have proposed to use photoRec, a free, open source utility, that seems to give the same average-to-good results as commercial products mentioned above. (But most people will not feel comfortable with its command line interface)
I even went as far as recording a rudimentary tutorial to help you use photoRec.



Until I discovered Disk Utility!
Disk Utility is a program that sits in your Utilities folder. It ships with every copy of Mac OS X.
And it’s not even a data recovery program.

This small program is capable of creating a disk image of your damaged card. It’s like copying the card, but instead of copying the files as the Finder would do (remember, it’s corrupt, it won’t help), Disk Utility copies the card, as is.
Imagine instead of a corrupt card, a damaged book with some missing index pages. The Finder would say it can’t copy anything because without index it doesn’t know where the chapters are.
Disk Utility would just photocopy the whole book, without asking questions.


Creating your Disk Image from the damaged drive…

This disk image, a .dmg file, contains all the recoverable media. Because it contains all what the card contains.
The beauty of the disk image is that we no longer have files, just raw data.
So if you use Treasured on the .dmg file, you can preview all the media. From all files as if they were just one.


Image Format: read-only
Compression: none (or it won’t work!)

Finally, the Repair Kit, working on the .dmg file, will restore in one shot all the media contained in the card: All the clips will be merged into one playable movie file.

Now let’s be a bit more technical:
Why does Disk Utility work better than data recovery products?

• First and foremost, because Treasured and Repair Kits work on raw media data, so the file structure is not helping (and their lack-of is not disturbing).
• And because data on cards is usually not very fragmented: Usage model is to fill the card (shoot until you can), then empty it, then fill again: Data layout is most likely linear on the card.

I discovered this by chance! Disk Utility was not on my radar, until a customer, Karen, told me she had done a disk image of the card, just in case.
Suddenly, a flash in my mind: Why not using the image on Treasured?
And it worked beautifully!

Thanks Karen.

]]> http://aeroquartet.com/wordpress/2009/06/17/your-best-ally-for-corrupt-card-recovery-disk-utility/feed/ 0 http://aeroquartet.com/wordpress/2009/03/05/can-i-make-movie-repairs-effortless/ http://aeroquartet.com/wordpress/2009/03/05/can-i-make-movie-repairs-effortless/#comments Thu, 05 Mar 2009 14:52:40 +0000 Benoit Joossen http://www.aeroquartet.com/wordpress/2009/03/05/can-i-make-movie-repairs-effortless/ A couple of weeks ago, I’ve released a new version of Treasured with some long-awaited improvements.
It’s not yet fulfilling my vision of a diagnostic and repair application, but still a huge step in the right direction.

Having a damaged movie repaired must become natural and almost effortless.



Remember two years ago, when all was done manually? I had to give instructions on how to extract a sample of the file, upload it. You had to spend one hour of your time, maybe more, just to do the first step. And only a few hours later I would send you a preview and propose a deal.

Repairing a movie required a certain engagement. It was hard, for me and for you.

Today all this is automated. Treasured shows the preview in a matter of seconds, extracts and uploads the file in a couple of clicks. A quotation for the repair is provided to help you decide.

But it’s not enough.

Video professionals are always extremely busy, fighting to meet deadlines. When you stumble upon an unplayable clip that jeopardizes the delivery of the project, it becomes a crisis situation immediately. Every second counts.

I understand that. The value lays not just in repairing the movie. It’s also in doing it fast, and spending as little of your time as possible.

Treasured is just the tip of the iceberg. Improvements in Repair Service are mostly underwater:

Today, where do you spend/waste time?

• 1. Finding my service, and deciding to try it.
  If you need to do 20 Google searches, follow 50 links, try several products, until you finally find the one that works, you’ve probably wasted half a day.
• 2. Getting started with Treasured and sending a repair request.
  This one is already well optimized. Not a priority today.
• 3. Waiting for the Repair Kit.
  Developing a repair kit takes time. As I’m accumulating experience and building powerful repair tools, the lead time tends to go down.
• 4. Getting the Repair Kit to work.
  Repair kit can quickly become unfriendly applications. I know what are the common pitfalls when you try to use the kit for the first time. Big improvements can be done here to shorten this phase.
• 5. Getting through payment.
  I’m using a standard solution here, that works well.
• 6. Entering the code and running the repair again.
  I see it technically feasible to remove this step. Watermarks could be removed from movies in a breeze.

Therefore, my vision of an effortless movie repair experience is still far from reality. But if I keep the same pace of work during 2009, I can probably walk half the distance.

]]> http://aeroquartet.com/wordpress/2009/03/05/can-i-make-movie-repairs-effortless/feed/ 0 http://aeroquartet.com/wordpress/2008/10/31/treasured-is-finally-out/ http://aeroquartet.com/wordpress/2008/10/31/treasured-is-finally-out/#comments Thu, 30 Oct 2008 22:22:27 +0000 Benoit Joossen http://www.aeroquartet.com/wordpress/2008/10/31/treasured-is-finally-out/ That’s a breakthrough in movie repair technology. For the first time, you can preview your damaged files and decide what deserves to be repaired.

It’s early to tell if all the goals of Treasured have been met, but for sure the landscape has changed for me:
Ten to twenty times more people contact me for a movie repair now. Treasured is definitively the hook that my Movie Repair Service needed.
But two thirds of them are not actual customers, they are just playing with the app and uploading a trashed DivX or WMV file.

The remaining third is a mix of my traditional customer base of video professionals, and a new category of customers. By lowering the barrier to request a repair, I’m attracting new types of repairs, new types of files, for which Treasured was not initially designed or optimized.

Bottom line: My claim of 90% accurate diagnostic, 80% with preview, that I religiously checked before Treasured was released, is wrong due to the diversity of movies I’m seeing now.

]]> http://aeroquartet.com/wordpress/2008/10/31/treasured-is-finally-out/feed/ 0 http://aeroquartet.com/wordpress/2008/09/01/teach-a-man-to-fish/ http://aeroquartet.com/wordpress/2008/09/01/teach-a-man-to-fish/#comments Mon, 01 Sep 2008 19:34:32 +0000 Benoit Joossen http://www.aeroquartet.com/wordpress/2008/09/01/teach-a-man-to-fish/ (Third installment of my “Movie Repair Guide”, where you’ll learn how a movie can be extracted from a larger file)


When you repair movies, you never know what you can find.
Yesterday I had an old shoe moment while looking for some mountain footage in a corrupt file.
Some obscure AVI Windows file containing this infamous “Download” animation surfaced when I least expected it:

Today I’ll teach you to fish, in other words I’ll explain how to rescue movies from inside larger files. It will only work if certain conditions are met, but it’s an exciting experience… that can end with an old shoe. You’ve been warned.

-=-=-

Container Structure Correction is a repair technique that acts upon the container structure data, leaving the media data and the index and tables untouched.
Unlike reindexing or other techniques that need to act on hundreds of audio or video frames, a structure correction is usually a question of few bytes to correct or a misplaced block of data. For this reason, it can be done manually.

If the same correction has to be applied on a collection of files, then maybe it’s worth spending time automating the structure correction.

Most common structure damages and how to fix them:

• Embedded movie

Full movie data, including container, is embedded in a larger file, for example after a data recovery.
In this case, the data can be extracted and saved with the container suffix and it usually works.

In this example, we have examined the file with an Hex editor and have found an intact RIFF structure in the middle of a file. Note that the length of the structure 0×49e4 is encoded just after RIFF in 32 bits, big-endian.
A quick check shows that after 0×49e4+8 bytes, there is suddenly no data, thus confirming that we are into something.

02a3970: 0000 0000 0000 0000 0000 0000 0000 0000 ................
02a3980: 0000 0000 0000 0000 0000 0000 0000 0000 ................
02a3990: 5249 4646 e449 0000 4156 4920 4c49 5354 RIFF.I..AVI LIST
02a39a0: 2004 0000 6864 726c 6176 6968 3800 0000 ...hdrlavih8...
02a39b0: 8545 0100 f816 0000 0000 0000 1008 0000 .E..............
02a39c0: 0500 0000 0000 0000 0100 0000 7c0b 0000 ............|...
02a39d0: 3c00 0000 3100 0000 0000 0000 0000 0000 <...1...........
02a39e0: 0000 0000 0000 0000 4c49 5354 d403 0000 ........LIST....
..........
02a8310: b9b9 b9b9 b9b9 b9b9 b9b9 b9b9 b9b9 b9b9 ................
02a8320: b9b9 b9b9 6964 7831 5000 0000 3030 6462 ....idx1P...00db
02a8330: 1000 0000 0400 0000 7c0b 0000 3030 6462 ........|...00db
02a8340: 1000 0000 880b 0000 7c0b 0000 3030 6462 ........|...00db
02a8350: 1000 0000 0c17 0000 7c0b 0000 3030 6462 ........|...00db
02a8360: 1000 0000 9022 0000 7c0b 0000 3030 6462 ....."..|...00db
02a8370: 1000 0000 142e 0000 7c0b 0000 0000 0000 ........|.......
02a8380: 0000 0000 0000 0000 0000 0000 0000 0000 ................

By copying this data to a new file and saving with an AVI suffix, we now have a valid movie.
Make sure you copy exactly the required bytes, starting from the R in RIFF, ending after idx structure, otherwise it won't work.

• Multiple moov atoms found, but last one is corrupt

When you edit a movie in-place (ie without writing again all the file), some edition softwares just add a new moov container at the end of the file, without bothering to remove older ones. If the file has become corrupt, those old containers can still work.
The trick consists in finding the moov structures and redirect towards an older one.
You'll need an Hex editor and good knowledge of QuickTime file format to do that.

• Mispositioned blocks of data

Files from a data recovery can present defects: contents not matching the file name, truncated files, mashed contents coming from several files, moved or missing blocks.
As data recovery acts at the lower filesystem level, the data is always presented in blocks, for example exactly 2048 bytes.
A movie can fail to open because some blocks have been moved, duplicated or missing. The fact have blocks always have a length of 2048 bytes helps to detect and correct those problems. It's like solving a puzzle and is usually complex.

]]> http://aeroquartet.com/wordpress/2008/09/01/teach-a-man-to-fish/feed/ 0 http://aeroquartet.com/wordpress/2008/08/16/zip-test/ http://aeroquartet.com/wordpress/2008/08/16/zip-test/#comments Sat, 16 Aug 2008 13:24:24 +0000 Benoit Joossen http://www.aeroquartet.com/wordpress/2008/08/16/zip-test/ (Second instalment of “Movie Repair Guide”)

When confronted with a corrupt movie file, there’s a quick way to check that it still contains the media data, which is a necessary condition to repair it: the “Zip test”

We create a compressed version of the file, which tells us the compression rate. For example, 50% means that the zip file is only half the size of the original one.

Audio and video data will give a low compression rate whereas other type of data will be around 50% or higher.



In Mac OS X, you can use the zip utility, either from the Finder, or from the Terminal:

zip --filename--

In Windows, you can use WinZip utility.

Here are aproximate compression rates you get:

• For data completly erased (file full of zeroes), 90 to 100%
• For alien data, 25 to 100% (Office documents: 50%, Applications: 40%)
• For audio integer 16, 21%
• For AIFF, 17%
• For DVCHD Pro, 11%
• For DV, 5 to 12%
• For other video codecs, DivX with mp3, Intermediate, MPEG2, H264, JPEG: 0 to 8%

There is a big gap between audio/video data and any other type of data. If you have a corrupt file, and the “Zip test” gives you a result over 25%, probably a complete repair is impossible because the data is partially or completely gone.

You can also refine the test by measuring on segments inside the file:

With a 500mb file, we split in 5 segments of 100mb and run the zip test on each segment with this two Mac OS X Terminal commands:

split -b 100m corrupt.mov
zip deleteme xaa xab xac xad xae

(The first one, split, cuts the file in chunks of 100mb each, called xaa, xab, xac, xad and so on.)
(The second creates a compressed archive called deleteme from xaa, xab, xac, xad and xae.)

This gives for example:

xaa (deflated 5%)
xab (deflated 5%)
xac (deflated 12%)
xad (deflated 100%)
xae (deflated 100%)

Which tells us that the media data in last two segments is probably lost. If a repair is possible, maybe 50 to 60% of the original file will be restored.

]]> http://aeroquartet.com/wordpress/2008/08/16/zip-test/feed/ 0