Once upon a time, there was an action camera called GoPro. It quickly became the top seller of its category, with 9 millions cameras shipped since 2009, and made its CEO a mad billionaire.
And since action cameras have a rough life, with no surprise did we start receiving several GoPro repair requests every day.
GoPro corrupt footage was routinely repaired, using deep parsing and re-indexing techniques.
One Day…
One day, we received a damaged GoPro file that couldn’t be cleanly repaired. Re-indexing seemed to work fine, and repaired videos were playable, but they were full of glitches, and audio was repeated every couple of seconds, like a mad echo effect.
At first we didn’t pay a lot of attention to this, we just said to the customer that the video could not be cleanly repaired due to presence of “alien data” inside the file. Alien data is material that should not be in the damaged file, and that makes re-indexing more difficult or even impossible. This is a bit like waste recycling that can only be efficient if waste has been properly triaged beforehand.
We were not wrong about “alien data” contamination, but not for the expected reasons… A few days later we received one, then two more damaged videos from different customers that were having the exact same symptoms:
- Damaged files preview perfectly with Treasured.
- However, after repair, it was a mix of clean and garbled video, with audio repeating (like echo) every two or three seconds.
At this point I considered having discovered a new failure mode, and I guessed that it was probably a side effect of how new GoPro cameras operate.
Cracking New GoPro
The most intriguing fact about this failure mode is that audio repeats, so we started our investigation with audio.
After close examination, I noticed that not only audio was repeating, but video as well! The only difference is that secondary video wasn’t directly visible, indeed the glitches inside repaired video are our secondary video stream. So we have a camera that records two streams at once, both with same audio but incompatible video.
Does it sound familiar?
It didn’t for me at first, but when I read the list of changes in new GoPro firmware and cameras, everything became clear…
LRV for Low Res Version
It turns out that GoPro camera now records an LRV clip at the same time as the HD clip. The LRV file is a lower resolution version of the video recording used for preview purposes on the phone and tablet GoPro applications (less data to transfer wirelessly to the device running the application, and less decoding raw power required as well).
During recording, your GoPro camera produces two streams of videos, one high resolution (represented by big orange images on illustration, and one low resolution, whose volume is one fourth to one tenth, hence the small blue images.
The Locker Room
Now let’s figure out what happens when the camera writes the two streams on the SD card. In computers, but also in phones and cameras, a program called “file system” is in charge of storing data and keep track of where it is stored to be able to read it later.
You can think of the SD card as a locker room, with thousands of lockers, all of the same size. People who want to store their stuff give it the person in charge, the file system, who will put it in one or several lockers and keep track of what goes where.
During recording, every second the GoPro guy comes to the locker room with new boxes that he hands over to the file system: 4 orange boxes, that fill 4 lockers, and 4 small blue boxes, that fit in one locker. To go faster, the file system fills the lockers in rows, so orange and blue boxes are interleaved.
Disaster Strikes
Unfortunately, before our file system could write in the registry where boxes have been stored, fire alarms suddenly start shrieking, and everyone leaves the building. A few minutes later, people is authorized to go back to work, but our file system no longer remember where video has been stored. Our GoPro is desperate… and calls SoS!
Cleaning up this Mess
Finding footage is easy, just open all the lockers, put all items together, and pray. This is what Data Recovery utilities do. Unfortunately, this won’t work because besides fragmentation, we have LRV data interleaved with HD video, that will cause glitches and echoing audio once fixed.
As you have guessed, we now have two problems:
- Separating LRV from HD video
- Re-indexing video
Triage
There’s no such thing as orange and blue data. Both streams contain H.264 video and AAC audio and are nearly indistinguishable. Yet we need to surgically separate them. Cutting just one byte off would generate glitches in several frames.
Fortunately, two things makes triage possible:
- Streams are mixed in locker-sized chunks. Cuts can only happen every 131072 bytes exactly, at discrete locations.
- Every H.264 frame starts with a pattern that informs of its exact bytes length, so we can verify whether it’s in one piece – or – whether 131072 bytes of “blue data” are contaminating the frame.
Combining those two hints, I have managed to remove LRV from the raw data. Now we are back to the normal situation of a corrupt video in need of re-indexing.
Amazing: Kayak Footage Restored!
The final step is to re-index the cleaned – yet still corrupt – raw file. Needless to say, when you have cracked a new failure mode like this one, this is a lot of excitement the first time your repair program finally cranks out clean video.
The first video that I have managed to fix shows some guys paddling in kayaks, so I have decided to call this feat: to pull a kayak
Frame from repaired clip, courtesy of Gordon Scott
If ever you face a corrupt video problem with your GoPro, even if it’s not a “kayak”, just send us a request through Treasured and we will do the paddling for you. 
March 17th, 2013 in
Movie Repair |
Comments Off
I have finally managed to put my hands on corrupt footage coming from new flagship Canon camera.
Canon claims that its EOS-1D C is the first 4K resolution DSLR camera.
I have examined the damaged footage: video is actually encoded in Motion JPEG, 4096 x 2160 and audio is in Linear PCM 16 bits.
At almost 3 MB per frame, the camera has to write its 4K/24p footage at an astonishing speed of 500 Mbits/s. Needless to say, the fastest CompactFlash cards in the market can barely sustain this write speed, and actually the camera uses a Dual CF set-up to cope with that.
Freeze, Zoom, Focus … Read the Newspaper!
4K videos recorded by Canon EOS-1D C produces images so detailed and sharp that you can crop 1% of a frame and still read the newspaper! A Blade Runner-ish performance!
Highlighted image represents just 1% of the original video frame!
Repairing Corrupt 4K Videos
Such demanding bandwidth and storage specs will inevitably cause problems. Corrupt videos will likely be produced by “usual suspects”:
- lack of space in card
- lack of speed of card
- camera failure in hot conditions
At this point there is no evidence that firmware 1.0.2 is responsible of the failure. We will track the failures by firmware versions as we did during Canon 7D epidemic outbreak.
The damaged footage that I have examined comes in a QuickTime MOV file, but without the moov atom at the end since recording did not end gracefully. This missing moov atom should contain the media tables and index that are needed for the video to be playable.
Fortunately, Motion JPEG media is easy to parse, thanks to the markers used to encode the image, in particular FFD8 (start of image) and FFD9 (end of image). With a standard re-indexing technique, we have managed to make the video playable, restoring original quality.
Therefore, as of February 2013…
- Treasured cannot detect and preview corrupt Canon EOS-1D C videos yet.
we will try to update Treasured in March with Canon 4K detection and preview.
- But we can repair Canon 4K videos if you send us a repair request with Treasured.
February 28th, 2013 in
General |
Comments Off
Many users of Treasured and MP4repair.org are very surprised to discover that their videos, after being recovered or undeleted by data recovery tools, still cannot open or play!
After having burnt days trying several tools and spent money on the most promising one, their recovered video files are still corrupt and they finally come to us because only our diagnostics tools Treasured (or MP4repair.org) can preview them and give them some hope.
Why does it work for my PDFs but not for my videos?
Data recovery tools are usually good at undeleting small files, like PDFs, Word and Excel documents, or low-res images. But as file size increases, the chances of recovery plummet.
Even for JPEG pictures in the 2 – 5 MB range, about 15% of files appear missing or corrupted.
For video files, whose size is usually in the hundreds of MB, and even in the tens of gigabytes for post-production formats, the chances of recovery are almost zero.
The reason is FRAGMENTATION: In your disk, everything is stored in fixed-sized “blocks”, like the shelves in the picture below. But if the file is big, it will not fit in one shelf.
Video files occupy many shelves. If the file is really big, let say 20 shelves, it’s not even possible to store it in contiguous shelves. In picture below, it’s obvious that really big files will end up spread over different walls.
Why Fragmentation Kills Recovery
Fragmentated files are split into several pieces, each piece being stored at a different place. Data is no longer contiguous.
This is not a problem for the operating system, that knows where those pieces are stored. But after a file deletion or a disk failure, this “map” also disappears and the file is now lost in the middle of thousands of shelves.
All data recovery tools (except high-end forensics tools, more about that below) use simple file carving techniques to recover deleted data.
Most common technique is Header/Footer carving. It consists in identifying the beginning of a file of a certain type, using a signature or pattern. AVI files, for example, start with RIFF word. MP4 files have a movie atom at the end of the file (footer) that can easily be detected using pattern matching.
Once a header and a footer are found, the data recovery program assumes that all data in between is part of the video file. This will only be true is the file was stored in one piece, without fragmentation.
This rarely happens. Even in disks with plenty of space available, fragmentation often happens as a side effect of video recording. For example, XDCAM cameras write as the same time several files (MP4, BIN, PPN) and this causes a systematic fragmentation of the MP4 video file.
Alien Data
Therefore, any fragmented video file recovered will contain “alien data”.
This will cause the following problems, from less serious to more serious:
- Glitches in video and audio.
- Possible crashes during playback
- Missing footage, footage from different clip inserted in video
- File no longer opens due to inconsistency of video container
- File not even recovered if software doesn’t find footer
Recovery is not Repair
Data recovery tools are very good at restoring small files of many different formats, by identifying superficial features of the data (header signatures, footer patterns) and assuming that the file is not fragmented.
But this assumption is not true for video files. The only effective technique that can be used to recover fragmented files is called “deep carving”.
Tools implementing this technique are very few, very specialized, more expensive, slower, and are based on probabilistic algorithms specific to the video and audio formats to recover.
- Recovery tools achieve good results on small files, but not on videos, due to fragmentation.
- Recovery tools can’t actually repair fragments of video files, because they can only identify generic and superficial file features
- Video Repair tools are specialized. They are built to repair one type of videos only,for example Canon EOS 1080p24 or XDCAM 720p30.
- Video Repair tools identify video frames and audio blocks deep inside the file and re-index them into a playable video.
This is the reason why our repair service works when other generic recovery tools don’t: We build specialized tools that can assemble the puzzle, fit the parts of the disk together to restore a functional video file.
February 9th, 2013 in
Movie Repair |
Comments Off
GoPro has recently shipped its new line-up of HERO3 cameras with impressive specs.
Users have started reporting problems with the camera in November 2012: One of the most annoying failure mode is the dreaded “lock up during video recording” that leaves you with a corrupt video. We are now receiving new damaged clips to fix every day.
Affected Firmware Versions
Lock-up problems can be traced back to firmware versions that include Protune features. All GoPro models with Protune are affected.
As of January 2013:
- All GoPro HERO 3 cameras except White model (because it doesn’t include Protune)
- GoPro HERO 2 cameras with Firmware upgrade 8.12.198 (October 2012) that enables Protune
Two important points:
Tip: Save $20 on GoPro Repairs with Treasured and MP4repair.org
We have been busy in the last months catching up with new GoPro cameras. Current releases of Treasured (Mac app) and MP4repair.org (web app) can detect and preview corrupt GoPro videos, even from new models and formats.
However, GoPro special pricing is not always correctly applied. When video is identified as Media: H.264 instead of Brand: GoPro HERO, price quote will be higher. We will fix that by end of February.
In the meantime, just add a comment in your repair request or send us an email to notify that it’s GoPro footage and we will apply the correct price to your repair.
January 31st, 2013 in
Movie Repair |
Comments Off
