SpaceX Falcon First Stage Landing (Part III)

This post is the third of a serie, explaining how at Aero Quartet we managed to recover the only existing pictures of the Falcon rocket “landing” on the Atlantic Ocean, first published on Elon Musk (SpaceX CEO) twitter on April 28.

In the previous posts, we explained why we have to focus our efforts on recovering frames F199 and F215, which are the only two keyframes that are almost error-free.

Brute Force Bit Flipping

In F199 we expect that by changing only one bit somewhere in the frame the picture will improve. The brute force approach consists in trying hundreds of positions until we find the right bit to flip.

F199 is about 20 kb in size, so we have 160,000 bits to test. But we can easily narrow down to a small region by finding where things start to go wrong, decoding-wise.

At first glance, decoding works fine in top 40% of the picture. Since MPEG4 Video is encoded from left to right, top to bottom, just as we read text on a page, and as image is made of blocks of 16×16 pixels, it’s easy to draw the line between good and bad in the picture. The highlighted rectangle below is where things go wrong: Before it all is good, after it all is bad.

So we can focus our bit-flipping effort on this area only. The cause of the problem must be here.
To find the bit positions inside the frame corresponding to this area, we can use several tricks. For example you make copies of the frame and write garbage information at a given position, and then you look at how the image is affected. By making a few iterations you will determine the position corresponding to beginning of area and to end of area. No rocket science involved here.

We have determined with this method a range that contains only 896 bits: from byte 6898 to byte 7010.
Now all we have to do is to try flip those bits one by one…

the Script to Flip Bits

Of course we don’t do this by hand, it would take days. I have written a couple of scripts to have the computer do the heavy lifting.

First of all, we will divide the range to test into 7 smaller ranges of 128 bits each. This is to make things easier for us humans who will have to visually verify the results eventually.
We will work on a video file (download) containing the single frame F199 that we try to repair. The 7 ranges have the following starting addresses (in hexadecimal):

0x4890 0x48a0 0x48b0 0x48c0 0x48d0 0x48e0 0x48f0

For each range, we run this ugly little bash script (everything is done under Mac OS X) that creates 128 copies of the video frame. Each copy differs from the original by exactly one bit that has been flipped.

mkdir f199-4890 # new folder where the 128 copies will be created
cd f199-4890
z="4890"; i=0; 
xxd -p -s 0x$z -l 16 ../f199.mov | ../flipBits.pl | sed "s/^/$z: /" | while read LINE; do
  cp ../f199.mov ./f$z-$i.mov ; echo $LINE | xxd -r - ./f$z-$i.mov ; let "i=i+1"; 
done

First, xxd extracts the 16 bytes at position 0x4890 from the original video f199.mov, and outputs them as an hex string: DF4E2D41FBBFFFB51D8412FF118A2C4F
Then flipBits.pl script (download) takes this hex string as input and creates the 128 variants, that it outputs, one per line:

5F4E2D41FBBFFFB51D8412FF118A2C4F
BF4E2D41FBBFFFB51D8412FF118A2C4F
... and 125 more and finally the last one ...
DF4E2D41FBBFFFB51D8412FF118A2C4E

Finally, we use xxd in reverse mode to apply the 128 variants back to the original video, and we create one file per variant.

You end up with a folder containing 128 video files, that you can review visually one by one until you find the needle in the haystack… Hey, f48e0-73 looks promising!

I’ve made it look easy, but I actually spent a few hours on this back in April, and my eyes were red after reviewing a few thousand pictures, before I stumbled upon f48e0-73.

Clean, Rinse, Repeat

Once you have improved an image, you can try to repeat to process on the same image:

• Figure out the range where decoder stops working
• Generate hundreds of variants of this image by flipping all bits in the range, one by one
• Visually pick the images that show an improvement

On the second frame F215, I have used this technique a well.

F215 in raw video stream

Macroblock area where decoder fails (highlighted)

F215 improved: Boosters are now in action to achieve zero speed, generating a cloud of vapor

Other Techniques to Improve Further

Unfortunately, at Aero Quartet we have very limited resources and we couldn’t work on this repair as many hours as we would have liked.
After delivering the two improved pictures to SpaceX, we actually stopped our effort, but Robert from SpaceX had the excellent idea to crowdsource the effort to the many SpaceX fans on the Internet.

I have followed what they have been doing (and this series of posts is also aimed at helping them), and I’m very impressed by the results.

They have not only used the “bit-flipping” technique that I’ve explained above, but they also have built a customized ffmpeg tool that lets them play with macroblocks inside the image. That allows them to skip some bad macroblocks and have the decoder catch-up later when good macroblocks are found again.

The best results from are visible on this web page: http://spacexlanding.wikispaces.com

SpaceX Falcon First Stage Landing (Part II)

This post is the second of a serie, explaining how at Aero Quartet we managed to recover the only existing pictures of the Falcon rocket “landing” on the Atlantic Ocean, first published on Elon Musk (SpaceX CEO) twitter on April 28.

Why the video stream can’t be recovered

First of all, I will explain why it’s pointless to try to recover the video, and why one must focus on recovering cherry-picked frames.

The rocket on-board camera has transmitted the video stream by radio to a plane flying in expected splash-down area. Unfortunately, the radio transmission was bad, probably due to distance and bad weather conditions.

The stream file has now been made public by SpaceX (download), and it’s identical to the file that we had access to a few days prior to public release: It’s a short (4 MB) file containing transport stream headers and footage encoded in MPEG4 Video format at 704 x 480.

I’m a fan of SpaceX, so when I first watched the video I was devastated: There’s not a single good frame in there, you can only guess some movement and boosters firing towards the end.

MPEG4 Video encoding is not redundant, so any error, even a single bit missing or flipped, will cause decoding to fail, leaving us with a garbled frame. Even worse, since MPEG4 Video uses temporal compression, a fault in a keyframe will cause the following 15 frames to be garbled as well.

I’ve been repairing videos full time for the last 7 years, so I immediately knew that the bitstream was corrupt and that at best we could hope to partially fix some frames. Maybe a 50% chance of recovering one or two frames.

Based on previous “edge-of-impossible” repairs, I know that the error rate must be close to zero. A 20kb frame, for example, contains 160,000 bits. If more than 2 or 3 of them are wrong, it becomes impossible to repair it.

This gives us a maximum error rate of 1/50,000.

Using Transport Stream Headers as Estimators of Error Rate

Transport Stream headers (TSH) are quite predictable: Every 188 bytes (this can also be 192 bytes when error correction is used) we have 4 bytes that follow this pattern:

47 03 E8 10
47 03 E8 11
47 03 E8 12
47 03 E8 13
47 03 E8 14
47 03 E8 15
47 03 E8 16
47 03 E8 17
47 03 E8 18
47 03 E8 19
47 03 E8 1A
47 03 E8 1B
47 03 E8 1C
47 03 E8 1D
47 03 E8 1E
47 03 E8 1F
47 03 E8 10 ... and repeating 11, 12, 13, ...
(note that this has been simplified)
(there are also special headers for padding and for beginning of frames)

TSHs, because they follow this pattern, are perfect to estimate the error rate in the stream: Assuming that bit errors happen randomly, if we find errors in the TSHs, then we also have errors in the payload and repair becomes impossible.
But if we find a segment of the stream without any error in the TSHs, we can have good confidence that the segment is almost error-free, hence maybe it can be repaired.

So let’s plot the good and bad areas of the file. For this, I’m using a tool called X-Rays that uses space-filling curves to represent properties of large files graphically. Don’t search for X-Rays on the Internet. It’s a proprietary tool used at Aero Quartet only. We may release it to the public in the future, but that’s another story.

In this particular case, I have programmed a custom filter that displays a black pixel where TSHs are corrupts or missing, and a white pixel where it looks healthy:

Necessary but not Sufficient

At this point, you may say that over 80% of the image is white, so the situation isn’t that bad.

But remember that this is encoded video with temporal compression, so in order to decode something, we need large areas free of black pixels, and such areas must contain at least a keyframe. That leaves us with very few possible spots that could be repaired.

Below I have plotted the file again with X-Rays, but with a filter that shows only keyframes.
Each Tetris-shaped area corresponds to a keyframe. Pixels in white correspond to good TSHs inside the keyframe, and pixels in grey to bad ones.

As you can see, most keyframes are severely “contaminated” by bad TSHs.

I have seen in some forums that people is trying to improve the video by fixing the TSHs.
This doesn’t hurt, but it will not improve the video, because while TSHs are easy to fix, a bad TSH indicates that probably several bits in the neighborhood are also bad. And those are the critical ones, and video won’t improve until you fix them all.

As mentioned previously, an error rate higher than 1/50,000 is impossible to fix. And if you have bad TSHs you probably have an order of magnitude more errors than that in the neighborhood.

Cherry Picking Frames

Now it’s time to cherry-pick the frames that we will try to repair.
When confronted to such a hard problem, I always start with the easiest case. And here, there’s clearly a winner among the keyframes: F199
F199 is almost entirely error-free. Second comes F215, then the rest is probably not worth looking at.

F199 stands for frame #199 of re-wrapped video.

Indeed, during the rest of the recovery I’m working on a re-wrapped video file.
I’ve used ffmpeg to rewrap TS in MOV. This operation doesn’t affect encoding, it only removes TSHs and gets rid of areas where TSHs are corrupt:
ffmpeg -i raw.ts -vcodec copy raw.mov

raw.mov is smaller than raw.ts, but contains the same information. What has been removed corresponds to black pixels in first X-Rays image.

In the raw video, F199 is unsurprisingly the best frame, from a visual point of view. The upper third of the picture is perfectly decoded. The rest is the picture looks like smear applied vertically, which is what is displayed when the MPEG4 Video decoder gives up.

In the next post I will explain how I managed to go from this to that:

F199 in raw video stream

F199 after repair effort

Read next post

SpaceX Falcon First Stage Landing: Pictures are from us!

Last Sunday (April, 27th) I was contacted by Robert from SpaceX to repair the historical footage of first-ever landing of a commercial rocket back on earth.

The footage was in really bad condition, actually you don’t see anything in the video, due to bad radio transmission between the rocket and the plane used to record the stream.

When I examined the stream, I immediately understood that there was too much data corruption and that recovering the video was impossible, but maybe recovering some frames from the stream was doable.

So I have cherry-picked two frames in the stream with fewer errors that the rest and I have worked on fixing them, and the result can be seen below. I reckon that those pictures have glitches, but they are the best graphical documents of the landing, and have been published on Elon Musk’s twitter here and here:

First stage of Falcon 9 rocket in controlled descent over Atlantic Ocean,
a few seconds before splash-down. The 120 feet rocket is seen from top, with
2 of the 4 landing legs deployed and visible.

A few seconds later, boosters are in action to achieve zero speed, generating a cloud of vapor

How I’ve done it

I know that SpaceX has “crowdsourced” the video recovery effort, but after one week it seems that nobody has done better than I did on April 28 and April 29, so explaining the process I’ve followed can be interesting for people trying to recover more frames.

Unfortunately someone passed away in my family last week, I have been traveling long-distance to attend to funerals, so I have a backlog of work and it will take me a few days of explain in details how I have done it.

This picture represents in gray the areas of the video file where data is in better condition. I have used space-filling curves to dray it, and I have cherry-picked the frames to repair based on this information. I’ll explain all this in detail in next post.

Why MXF Format will eventually Win

Predictions are hard, and Internet is unforgiving, but let’s risk a bit on this one:

<claim>MXF will become the standard format for Production in the next years.</claim>

I believe that this video format has significant advantages over alternatives like .MOV or .MP4 and that recent industry moves will accelerate its adoption.
MXF stands for Material eXchange Format and as production workflows become more complex, a versatile container format that can be used from capture to post-production is exactly what professionals want.

A Slow Starter

MXF isn’t exactly a new container format for us: Back in 2008, DVCPro HD was a very popular recording format, and most of the time we had to deal with MXF damaged files.
But for a format standardized over a decade ago, this has been a very slow start.

Recently, the success of XDCAM HD422 camcorders like Sony PWM-200 and of Canon XF products like Cinema EOS C300 has given MXF a promiment place in professional workflows. Ageing of QuickTime .mov container and Apple missteps with Final Cut X have also helped.

MXF enjoys a growing support among industry players

It’s the Workflow, Stupid

A few months ago, I started to receive requests to repair footage from new Sony CineAlta F5. This 4K cinema camera uses a new format called Sony RAW, which is recorded in MXF files.

Until now, we had delivered our repaired MXF footage as .mov files. There was a lot of possible workflows to use the repaired videos, and our customers found that it was only a minor inconvenience (compared to a loss of footage…)

But the times they are a-changin, you know the song… and now customers want their files repaired as MXF more often than not, and they have excellent reasons for that:

• Final Cut has lost share in postproduction world, so many people is now in 100% Windows set-ups where QuickTime files are not very friendly, whereas MXF is first class citizen in AVID and Adobe Premiere.
• In 4K and Cinema productions, where workflow is king, end-to-end processing including color grading simply cannot be done without MXF source files.

Therefore, when our first Cinema repairs started to arrive, we saw it necessary to develop tools that could output MXF files:

a Repair Kit that can repair Sony XAVC videos in both MOV and MXF containers

The 4 Advantages of MXF

So what significant advantages will make MXF win?

I believe that there are 4 important things that MXF is doing extremely well. Other containers format can perform well on one or two of them, but only MXF nails the 4:

Robust

MXF is a verbose and redundant format, which makes it more robust and easier to repair than MOV or MP4. That’s a strong advantage for professionals.
At Aero Quartet, it’s always a pleasure to work on corrupt MXFs, because we know that we will be able to provide a repaired video with same quality as original.

Treasured can detect and preview video inside unplayable or recovered MXF files. DVCPro HD, IMX, Sony RAW, Sony XAVC, XDCAM, Canon XF and other formats supported.

For the more technically inclined, we explain how MXF is internally organized to provide robustness.

Workflow-friendly

MXF can be used from capture to post-production. The big advantage is that all metadata “travels” inside the MXF files from beginning to end of workflow, which saves a lot of headaches to all the production and post-production crew.

The full metadata is embedded in every single frame. Again, this is the correct way to do it. Compare that to XDCAM EX that uses separate BIM, PPM, and XML files that can easily get lost in the process.

Example: We have used MXFDump to display the IndexTableSegment of a Sony XAVC file in MXF format: The last component of each frame is Metadata. This means that we can retrieve Timecode, Aperture, and many other video parameters from every single video frame.

[ K = IndexTableSegment ( 000000000007fe00 )
06.0e.2b.34.02.53.01.01.0d.01.02.01.01.10.01.00, L =        175 (af) ]
         InstanceUID = 2c.cf.45.4a.89.a1.11.e3.a5.5f.08.00.46.6a.89.be
     Index Edit Rate = (         25 /          1 )
Index Start Position = 0000000000000000
      Index Duration = 0000000000000000
Edit Unit Byte Count = 00097400
            IndexSID = 00000001
             BodySID = 00000002
          SliceCount = 00
     DeltaEntryArray = [ Number of entries =         11
                         Entry size        =          6 ]
                 Pos Table  Slice     Element
                     Index              Delta
             0 :      0      0              0   System
             1 :      0      0            512   Picture
             2 :      0      0         569856   Sound #1
             3 :      0      0         576000   Sound #2
             4 :      0      0         582144   Sound #3
             5 :      0      0         588288   Sound #4
             6 :      0      0         594432   Sound #5
             7 :      0      0         600576   Sound #6
             8 :      0      0         606720   Sound #7
             9 :      0      0         612864   Sound #8
            10 :      0      0         619008   Metadata
     IndexEntryArray = [ Number of entries =          0
                         Entry size        =         15 ]

Open Standard

MXF is an open standard promoted by the video industry.
That’s a guarantee that a single vendor cannot and will not play dirty tricks. MXF is being adopted on its own merits, is being developed based on industry needs, and there is no risk of vendor lock-in like with proprietary formats.

Professionals should like this.

Future-Proof

Finally, I believe that this is what makes MXF a strong finisher and eventually a winner:
As Digital Cinema and 4K production are taking off, MXF is the natural choice from most software and hardware vendors because it’s a competent and extensible format.

To make a computing analogy, I would say that MXF is the 64 bits era format, ready for the next 20 years, whereas QuickTime MOV, designed in the late 80s, is the 32 bits era format that starts to show significant limitations.

Don’t Let Glitches Spoil your Recovered Videos

A few years ago, when we started repairing XDCAM EX videos, we noticed that output wasn’t clean. Every few frames there was a glitch that would affect both audio and video.

The good news for wedding videographers is that at Aero Quartet, we have collected statistical evidence correlating glitchy videos with happy marriages. Just kidding…

LOVE

HATE

When we examined the raw data inside the damaged videos, we noticed a pattern:

We found that those glitches occured where pages of “alien data” had been inserted in the footage.

In our jargon, a page is a block of data that starts at a remarkable address and has a remarkable length. For example, starting at 0x450000 and with a length of 0x10000 (those values with 0x prefix are hexadecimal numbers – that’s what computers use)
Note that those numbers end with five zeros: those are round numbers.

X-Rays

And those pages contain data this is really different from audio or video, and that makes them also easy to detect:
A page that always starts with AF 7E 12 EF and is almost filled with 00 bytes is in utter contrast with a page of XDCAM video (that looks like random numbers).

We have an internal tool called “X-Rays” that uses space-filling curves to visualize remarkable structures inside damaged video files. If you configure it to detect areas almost filled with 00 bytes, you get that:

XDCAM with pages of “alien data”

a normal XDCAM EX file

Areas filled with 00 bytes represented in BLACK.

Leaving aside lonesome pixels that are not representative, we have 18 “rough spots” in our damaged video, and none in a normal XDCAM file. Each spot corresponds to a page of alien data. If you repair the damaged video as is, the footage will have 18 glitches, because alien data is contaminating audio and video.

Usual Suspects

We quickly figured out that this “alien data” wasn’t really alien. Actually the XDCAM camcorders responsible for the damaged videos all had something in common: They were recording metadata files in parallel with the main video file.

The typical XDCAM EX folder looks like this, with a subfolder for each clip, and several files inside each:

The important one is the .MP4 file that contains the actual clip. Note that its size is orders of magnitude bigger than the rest. The rest with .BIM .PPN .SMI and .XML suffixes are metadata files that describe what the clip contains.

And this solved the mystery:

1. Damaged videos we are asked to repair are actually “recovered” files (using one of those data recovery utilities, or our own DeepMediaScan)
2. Recovery brings back raw data in the same sequence it was written to disk, regardless of the file it belongs to.
3. So if a small bit of a .BIM file was written every second, it ends up in the middle of the .MP4 stream
4. And this causes a visual glitch

Therefore, our “alien data” belongs to .BIM metadata files. (and some traces from the other ones as well).
To achieve a clean output, we must remove the corresponding pages and then repair.

Clean, Rinse, Repeat

For XDCAM EX customers confronted with this problem, we have developed special repair kits that have a “cleaner” built-in. It works like this:

• Our repair kit scans the file and flags all the pages containing alien data
• Then it scans again the file, carefully skipping the flagged pages, and extracts video and audio
• Last but not least, it builds the video and audio index.

For example, the log of the “cleaner” in our damaged file says:

No alien data found at beginning
Found block of alien data: SkipFrom 553000, SkipLength 20000
Found block of alien data: SkipFrom d5e800, SkipLength 10000
Found block of alien data: SkipFrom 1553000, SkipLength 20000
Found block of alien data: SkipFrom 1d53000, SkipLength 20000
Found block of alien data: SkipFrom 2553000, SkipLength 20000
Found block of alien data: SkipFrom 2d53000, SkipLength 20000
Found block of alien data: SkipFrom 3553000, SkipLength 20000
Found block of alien data: SkipFrom 3d53000, SkipLength 20000
Found block of alien data: SkipFrom 4553000, SkipLength 20000
Found block of alien data: SkipFrom 4d53000, SkipLength 20000
Found block of alien data: SkipFrom 5563000, SkipLength 10000
Found block of alien data: SkipFrom 5d53000, SkipLength 20000
Found block of alien data: SkipFrom 6553000, SkipLength 20000
Found block of alien data: SkipFrom 6d53000, SkipLength 20000
Found block of alien data: SkipFrom 7553000, SkipLength 20000
Found block of alien data: SkipFrom 7d53000, SkipLength 20000
Found block of alien data: SkipFrom 8563000, SkipLength 10000
Found block of alien data: SkipFrom 8d53000, SkipLength 20000
Smallest continuous interval: 8257536

What has the cleaner done? It has removed 18 blocks, most of them of length 0x20000 bytes (128kb) and 3 of them of 0x10000. This corresponds exactly to what X-Rays had visualized above.

After cleaning, the repair kit can produce clean video, restoring original quality.

Not just XDCAM

Having several files written in parallel during video recording is what causes recovered XDCAM to be contaminated with alien data.
But other cameras also record by writing several files at once, and it causes similar problems:

• when MXF is recorded with audio and video in separate files
• when GoPro HERO records at once HD and LRV (low resolution) videos

We have also developed “cleaners” for those cases.

LOVE WINS OUT

4K Winners and Losers

Unless you have been living under a rock last year, you know that all major players in video industry have been playing hard to lead the 4K and Cinema revolution. Historical players like RED, SI-2K and ARRI now face a fierce competition.

When it comes to sales and adoption of those new technologies, we don’t have figures to provide.
But regarding repair requests, we have solid numbers that have historically correlated pretty well with real world use of the cameras:
If you use a camera a lot, you end having to deal with corrupt videos and that’s when we hear from you.

So who is leading the pack?

The leader is… clearly Sony with its F5 and F55 models that record in both XAVC 4K and Sony RAW formats.
It’s too early to talk about a winner in 4K, but I would bet on Sony with its XAVC format. Panasonic has yet to release its Varicam 4K marvel, so let’s wait and see…

The loser is Canon with its C500. We have no evidence that it’s currently being used for real 4K work. Or maybe it’s that the camera is so reliable that it never generates corrupt files? We seriously doubt it, because Canon has a track record with bad firmware.
Canon Cinema EOS 1D C, which looks more like an afterthought, marketing-wise, is actually doing much better than the C500.
Note that C300 is very successful, but sorry it’s not neither a Cinema (lacks RAW capability) nor a 4K camera.

Blackmagic is more of a mystery: The new cameras look pretty popular but we receive very few requests. Possible explanation: Blackmagic comes from a digital recording background, so it must be very strong about firmware and reliability, thus causing very few disasters.

Did I mention that Treasured v3.0 can now detect and preview footage from those new cameras?
OK, done.

Xmas Quadcopter: 5 Tips to Keep it Alive until 2014

Aerial Photography for the Rest of Us?

From what we have observed in the last days, the quadcopter will be one of the hottest present for this Xmas.
And from what we see in the videos theirs owners ask us to repair, we see three clear trends:

• Quadcopters look easy, but are hard to fly (and are not as reliable as one could expect from an expensive toy). You’re never more than five seconds away from a hard landing, a tree crash, or a total loss.
• Quadcopters are dangerously inspiring. Actually, the number of assholeries you can do with copters seems unlimited.
• Kids no longer wait until Dec, 25 to unpack their present. Or is this that fathers need to verify that their kid copter works fine to avoid a christmas incident? Controversial…

But the deadliest incident is clearly the splash landing. Hovering a swimming pool doing Top Gun-style manoeuvres looks cool.
Now see below how mighty a quadcopter looks like 2 seconds before its death:

5 Tips to Avoid Crashing your Copter on the First Day

• Don’t try cool tricks. Fly the quadcopter as if your kids were onboard.
  Or your copter will probably not make it into 2014.
• Stay away from obstacles and confined areas. This birch tree in your garden could well be the copter’s habitat for the years to come.
• Never mount an expensive camera on your copter. (expensive: what you cannot afford buying every week)
• Never power down the copter before stopping camera recording.
  In camcopters like DJI Phantom 2 Vision, this will cause the video to be corrupt. (and for us it will be a pleasure to repair it)
• People and animals approached by a quadcopter won’t be amused.
  Check what this crocodile has to say about it:

Be careful with your quadcopter and happy new year!

Video Professionals: the 5 Recipes for Disaster

Warning: Reading this post can reduce your chances of using our Video Repair Service this year tenfold.

For a Video Professional with a hard-earned reputation, it’s a really bad day when you have to explain to a customer that you won’t be able to deliver the job.

At Aero Quartet, we are dealing with disaster every day, and we have been rescuing people for 5 years now. We have done root cause analysis and distilled that knowledge into 5 main causes, that we share with you.

Basically, almost every customer has one or more of those 5 problems:

1. Leave your Summer Intern in Charge

On any camera, you’re never more than 2 clicks away from formatting the card. On any production workflow, you’re never more than 2 clicks away from changing this special setting that will silently cause the loss of days of work.

Even routine operations like copying files from card to computer can silently fail. Seasoned professionals will always verify that copied footage is correctly ingested by editing software. Interns will probably not.

With new hires or inexperienced interns, we recommend a very close supervision (Trust, but Verify) and to avoid critical tasks altogether. There’s always some B-Roll, documentation or chores to get them started.

2. Buy Cheap Gear

Cheap is not bad per se, but always remember that the chain will break on its weakest link.
Would you compromise a $50,000 system by using cheap cards or hard disks to save $50 or $100? You shouldn’t.

Price is usually a good proxy for quality.
If product A costs 2x as much as product B and has the same specifications, start asking yourself where cost savings may have gone. The Devil is in the details:

• A power supply shall last forever and withstand grid incidences handsomely.
• An enclosure shall be mechanically strong while preventing electronics from over-heating
• Top-quality connectors shall withstand 5000 connection cycles
• Hard-disks shall be server-grade in terms of speed and reliability

You take them for granted, but Product B probably has one or more weak components.

3. Work at the Limit

In any system, failure rate will skyrocket when you come close to the design and operating limits.
If your priority is reliability, you should always use over-dimensioned parts. Boat engines run only at 10% of their maximum power, at very low rpm, but they do so for decades, requiring very low maintenance.

A few examples:

Cards and Disks Speed

If camera manufacturer specifies minimum Class 6 SDHC card, buy quality and margin by purchasing Class 10 SDHC cards.
If you need 100 Mbit/s writing speed on your disk, buy a disk with 2x as much writing speed.

Storage Capacity

Failure often occurs when you run out of disk or card capacity, or when you get close to it. In addition, this make recovery much more difficult, if not impossible, due to fragmentation. Our advice: Always have 25% of storage capacity available (ideally: 50%)

Temperature

Electronic systems are extremely sensitive to temperature. Storing your cameras exposed to sunlight or intense heat sources, or having your NAS, server or computers in a poorly refrigerated cabinet, is not really a good idea.

Working at the limits sounds cools, but that’s not what you want for your business.

4. Tolerate Problems

That’s a really bad habit that ends up causing unrecoverable situations. Zero tolerance with problems.
Intermittent problems are the worse ones, because you can live with them. Or at least you believe you can.

At the first signal of failure, be it a disk reading error, a connector reliability, or software hanging or crashing, please stop. That’s your opportunity to avoid major problems. Troubleshoot, replace or quarantine the defective element. That’s for you own good.

Always have one spare hard disk ready to kick in if your RAID5 warns you or if you have the slightest doubt about a production disk.
For software, keep it simple. Problems often come from using non-standard or complex features. Do you really need this plug-in? Is Magic Lantern really my friend?

Special Mention to Canon EOS 2011 epidemic failure: Some people still haven’t upgraded their firmware to fix it. Unacceptable.

5. Backups are for the Wimpies

This one is obvious. So obvious that it’s still the top offender…

And don’t forget that one thing is worse than not doing backups:
Backups that don’t work when you need them.

A backup routine can easily become broken, in the sense that you start changing your workflow but you don’t adapt your backup to it, and that you don’t verify that you can recover your work environment out of your backup.

Keep it simple and automated, and verify it:
Every time you set-up a new equipment (storage, computer), do the set-up out of the backup, not out of the production equipment. By doing so, you detect possible shortcomings and validate your backup.

Have a nice year without disaster!

Those GoPro HERO Videos that Cannot Could Not be Repaired

Once upon a time, there was an action camera called GoPro. It quickly became the top seller of its category, with 9 millions cameras shipped since 2009, and made its CEO a mad billionaire.
And since action cameras have a rough life, with no surprise did we start receiving several GoPro repair requests every day.

GoPro corrupt footage was routinely repaired, using deep parsing and re-indexing techniques.

One Day…

One day, we received a damaged GoPro file that couldn’t be cleanly repaired. Re-indexing seemed to work fine, and repaired videos were playable, but they were full of glitches, and audio was repeated every couple of seconds, like a mad echo effect.

At first we didn’t pay a lot of attention to this, we just said to the customer that the video could not be cleanly repaired due to presence of “alien data” inside the file. Alien data is material that should not be in the damaged file, and that makes re-indexing more difficult or even impossible. This is a bit like waste recycling that can only be efficient if waste has been properly triaged beforehand.

We were not wrong about “alien data” contamination, but not for the expected reasons… A few days later we received one, then two more damaged videos from different customers that were having the exact same symptoms:

• Damaged files preview perfectly with Treasured.
• However, after repair, it was a mix of clean and garbled video, with audio repeating (like echo) every two or three seconds.

At this point I considered having discovered a new failure mode, and I guessed that it was probably a side effect of how new GoPro cameras operate.

Cracking New GoPro

The most intriguing fact about this failure mode is that audio repeats, so we started our investigation with audio.

After close examination, I noticed that not only audio was repeating, but video as well! The only difference is that secondary video wasn’t directly visible, indeed the glitches inside repaired video are our secondary video stream. So we have a camera that records two streams at once, both with same audio but incompatible video.

Does it sound familiar?

It didn’t for me at first, but when I read the list of changes in new GoPro firmware and cameras, everything became clear…

LRV for Low Res Version

It turns out that GoPro camera now records an LRV clip at the same time as the HD clip. The LRV file is a lower resolution version of the video recording used for preview purposes on the phone and tablet GoPro applications (less data to transfer wirelessly to the device running the application, and less decoding raw power required as well).

During recording, your GoPro camera produces two streams of videos, one high resolution (represented by big orange images on illustration, and one low resolution, whose volume is one fourth to one tenth, hence the small blue images.

The Locker Room

Now let’s figure out what happens when the camera writes the two streams on the SD card. In computers, but also in phones and cameras, a program called “file system” is in charge of storing data and keep track of where it is stored to be able to read it later.

You can think of the SD card as a locker room, with thousands of lockers, all of the same size. People who want to store their stuff give it the person in charge, the file system, who will put it in one or several lockers and keep track of what goes where.

During recording, every second the GoPro guy comes to the locker room with new boxes that he hands over to the file system: 4 orange boxes, that fill 4 lockers, and 4 small blue boxes, that fit in one locker. To go faster, the file system fills the lockers in rows, so orange and blue boxes are interleaved.

Disaster Strikes

Unfortunately, before our file system could write in the registry where boxes have been stored, fire alarms suddenly start shrieking, and everyone leaves the building. A few minutes later, people is authorized to go back to work, but our file system no longer remember where video has been stored. Our GoPro is desperate… and calls SoS!

Cleaning up this Mess

Finding footage is easy, just open all the lockers, put all items together, and pray. This is what Data Recovery utilities do. Unfortunately, this won’t work because besides fragmentation, we have LRV data interleaved with HD video, that will cause glitches and echoing audio once fixed.

As you have guessed, we now have two problems:


• Separating LRV from HD video
• Re-indexing video

Triage

There’s no such thing as orange and blue data. Both streams contain H.264 video and AAC audio and are nearly indistinguishable. Yet we need to surgically separate them. Cutting just one byte off would generate glitches in several frames.

Fortunately, two things makes triage possible:

• Streams are mixed in locker-sized chunks. Cuts can only happen every 131072 bytes exactly, at discrete locations.
• Every H.264 frame starts with a pattern that informs of its exact bytes length, so we can verify whether it’s in one piece – or – whether 131072 bytes of “blue data” are contaminating the frame.

Combining those two hints, I have managed to remove LRV from the raw data. Now we are back to the normal situation of a corrupt video in need of re-indexing.

Amazing: Kayak Footage Restored!

The final step is to re-index the cleaned – yet still corrupt – raw file. Needless to say, when you have cracked a new failure mode like this one, this is a lot of excitement the first time your repair program finally cranks out clean video.

The first video that I have managed to fix shows some guys paddling in kayaks, so I have decided to call this feat: to pull a kayak

Frame from repaired clip, courtesy of Gordon Scott

If ever you face a corrupt video problem with your GoPro, even if it’s not a “kayak”, just send us a request through Treasured and we will do the paddling for you. 🙂

Just Reported: First Case of Corrupt Video from Canon EOS-1D C

I have finally managed to put my hands on corrupt footage coming from new flagship Canon camera.
Canon claims that its EOS-1D C is the first 4K resolution DSLR camera.

I have examined the damaged footage: video is actually encoded in Motion JPEG, 4096 x 2160 and audio is in Linear PCM 16 bits.
At almost 3 MB per frame, the camera has to write its 4K/24p footage at an astonishing speed of 500 Mbits/s. Needless to say, the fastest CompactFlash cards in the market can barely sustain this write speed, and actually the camera uses a Dual CF set-up to cope with that.

Freeze, Zoom, Focus … Read the Newspaper!

4K videos recorded by Canon EOS-1D C produces images so detailed and sharp that you can crop 1% of a frame and still read the newspaper! A Blade Runner-ish performance!

Highlighted image represents just 1% of the original video frame!

Repairing Corrupt 4K Videos

Such demanding bandwidth and storage specs will inevitably cause problems. Corrupt videos will likely be produced by “usual suspects”:

• lack of space in card
• lack of speed of card
• camera failure in hot conditions

At this point there is no evidence that firmware 1.0.2 is responsible of the failure. We will track the failures by firmware versions as we did during Canon 7D epidemic outbreak.

The damaged footage that I have examined comes in a QuickTime MOV file, but without the moov atom at the end since recording did not end gracefully. This missing moov atom should contain the media tables and index that are needed for the video to be playable.

Fortunately, Motion JPEG media is easy to parse, thanks to the markers used to encode the image, in particular FFD8 (start of image) and FFD9 (end of image). With a standard re-indexing technique, we have managed to make the video playable, restoring original quality.

Therefore, as of February 2013…

• Treasured cannot detect and preview corrupt Canon EOS-1D C videos yet.
  we will try to update Treasured in March with Canon 4K detection and preview.
• But we can repair Canon 4K videos if you send us a repair request with Treasured.

Why Recovered Videos won’t Play

Many users of Treasured and MP4repair.org are very surprised to discover that their videos, after being recovered or undeleted by data recovery tools, still cannot open or play!

After having burnt days trying several tools and spent money on the most promising one, their recovered video files are still corrupt and they finally come to us because only our diagnostics tools Treasured (or MP4repair.org) can preview them and give them some hope.

Why does it work for my PDFs but not for my videos?

Data recovery tools are usually good at undeleting small files, like PDFs, Word and Excel documents, or low-res images. But as file size increases, the chances of recovery plummet.
Even for JPEG pictures in the 2 – 5 MB range, about 15% of files appear missing or corrupted.

For video files, whose size is usually in the hundreds of MB, and even in the tens of gigabytes for post-production formats, the chances of recovery are almost zero.

The reason is FRAGMENTATION: In your disk, everything is stored in fixed-sized “blocks”, like the shelves in the picture below. But if the file is big, it will not fit in one shelf.

Video files occupy many shelves. If the file is really big, let say 20 shelves, it’s not even possible to store it in contiguous shelves. In picture below, it’s obvious that really big files will end up spread over different walls.

Why Fragmentation Kills Recovery

Fragmentated files are split into several pieces, each piece being stored at a different place. Data is no longer contiguous.

This is not a problem for the operating system, that knows where those pieces are stored. But after a file deletion or a disk failure, this “map” also disappears and the file is now lost in the middle of thousands of shelves.

All data recovery tools (except high-end forensics tools, more about that below) use simple file carving techniques to recover deleted data.

Most common technique is Header/Footer carving. It consists in identifying the beginning of a file of a certain type, using a signature or pattern. AVI files, for example, start with RIFF word. MP4 files have a movie atom at the end of the file (footer) that can easily be detected using pattern matching.

Once a header and a footer are found, the data recovery program assumes that all data in between is part of the video file. This will only be true is the file was stored in one piece, without fragmentation.

This rarely happens. Even in disks with plenty of space available, fragmentation often happens as a side effect of video recording. For example, XDCAM cameras write as the same time several files (MP4, BIN, PPN) and this causes a systematic fragmentation of the MP4 video file.

Alien Data

Therefore, any fragmented video file recovered will contain “alien data”.
This will cause the following problems, from less serious to more serious:

• Glitches in video and audio.
• Possible crashes during playback
• Missing footage, footage from different clip inserted in video
• File no longer opens due to inconsistency of video container
• File not even recovered if software doesn’t find footer

Recovery is not Repair

Data recovery tools are very good at restoring small files of many different formats, by identifying superficial features of the data (header signatures, footer patterns) and assuming that the file is not fragmented.

But this assumption is not true for video files. The only effective technique that can be used to recover fragmented files is called “deep carving”.

Tools implementing this technique are very few, very specialized, more expensive, slower, and are based on probabilistic algorithms specific to the video and audio formats to recover.

• Recovery tools achieve good results on small files, but not on videos, due to fragmentation.
• Recovery tools can’t actually repair fragments of video files, because they can only identify generic and superficial file features
• Video Repair tools are specialized. They are built to repair one type of videos only,for example Canon EOS 1080p24 or XDCAM 720p30.
• Video Repair tools identify video frames and audio blocks deep inside the file and re-index them into a playable video.

This is the reason why our repair service works when other generic recovery tools don’t: We build specialized tools that can assemble the puzzle, fit the parts of the disk together to restore a functional video file.

GoPro Freezing during Video Recording

GoPro has recently shipped its new line-up of HERO3 cameras with impressive specs.

Users have started reporting problems with the camera in November 2012: One of the most annoying failure mode is the dreaded “lock up during video recording” that leaves you with a corrupt video. We are now receiving new damaged clips to fix every day.

Affected Firmware Versions

Lock-up problems can be traced back to firmware versions that include Protune features. All GoPro models with Protune are affected.

As of January 2013:

• All GoPro HERO 3 cameras except White model (because it doesn’t include Protune)
• GoPro HERO 2 cameras with Firmware upgrade 8.12.198 (October 2012) that enables Protune

Two important points:

• Lock ups happen even if Protune mode is disabled.
• GoPro HERO 2 firmware can be downgraded to version 8.12.124, without Protune.

Tip: Save $20 on GoPro Repairs with Treasured and MP4repair.org

We have been busy in the last months catching up with new GoPro cameras. Current releases of Treasured (Mac app) and MP4repair.org (web app) can detect and preview corrupt GoPro videos, even from new models and formats.

However, GoPro special pricing is not always correctly applied. When video is identified as Media: H.264 instead of Brand: GoPro HERO, price quote will be higher. We will fix that by end of February.

In the meantime, just add a comment in your repair request or send us an email to notify that it’s GoPro footage and we will apply the correct price to your repair.

Corrupt Audio Files: How to Repair

Today we will provide a few tricks to recover corrupt audio files.

Whether you’re a professional using Apple’s Logic Studio or AVID Pro Tools, an extreme biker torturing the last GoPro camera or a video aficionado editing the last family gathering with Windows Movie Maker, you can end up fighting with a corrupt WAV, M4A, AIF or QuickTime MOV audio file.

Your fight is really unfair: Even if you are a skilled computer guy, you probably will spend hours on that, to no avail. Professional video and audio repair services exist for a reason. We have spent the last 5 years developing recovery tools for Mac and a web video repair app to give you predictable results.

You’ve been warned.

Do It Yourself: Linear PCM Audio Repair

Note that the techniques below only work for Linear PCM format and for containers mentioned below. We won’t be able to repair an .m4a file containing AAC with those methods, for example.

• How to repair a corrupt MOV file containing Linear PCM audio using “Transplant” technique
  Note that if you have a good audio file with exactly same settings than damaged one, and whose size is bigger, you can use it as placeholder and skip step 1.
• How to repair a corrupt AIFF file using “Container Correction” technique
• How to repair a corrupt WAV file using “Container Correction” technique

Unfortunately, repairing compressed audio formats like AAC or ALAC (Apple Lossless) is much more complex and we’ll leave it for future posts.

Canon 5D Mk III video files playing only first second of audio and black image

We are seeing a pattern with damaged Canon 5D Mark III videos:
Corrupt files are playable with usual tools (QuickTime Player, VLC) but video is black or gray and audio is OK during first second, then becomes white noise.

Note: With Treasured, our diagnostics app , you can preview those corrupt files and then repair them with through our Repair Service. This post is not about repairing videos, but about understanding why we see this pattern.

Let’s do a post-mortem of an unplayable video and figure out why.

Playing Detective

With the help of an hexadecimal editor, we look at the beginning of the file. Trained eyes will see the normal header of a QuickTime .MOV movie. It contains a moov atom, which usually is at the end of the file. Strange…

0000000: 0000 0018 6674 7970 7174 2020 2007 0900  ....ftypqt   ...
0000010: 7174 2020 4341 4550 0001 7fe8 6d6f 6f76  qt  CAEP....moov
0000020: 0001 004a 7564 7461 0000 0026 434e 4356  ...Judta...&CNCV
0000030: 4361 6e6f 6e41 5643 3030 3130 2f30 332e  CanonAVC0010/03.
0000040: 3030 2e30 302f 3030 2e30 302e 3030 0000  00.00/00.00.00..
0000050: 000c 434e 444d ffd8 ffd9 0001 0010 434e  ..CNDM........CN
0000060: 5448 0000 36e3 434e 4441 ffd8 ffe1 0e26  TH..6.CNDA.....&
0000070: 4578 6966 0000 4949 2a00 0800 0000 0900  Exif..II*.......

If we have a moov atom, let’s check if it is consistent and what is inside it. For this we use Dumpster or Atom Inspector, two free apps that you can download from developer.apple.com (free developer account required).

Surprise, our moov atom is a valid! Complete with 3 tracks (video, audio, timecode) and metadata.
This explains why the corrupt video file opens: the valid moov atom at the beginning of the file contains all the information required to configure the movie and the tracks, so QuickTime Player initializes and can start decoding media.

One Second of Audio

Now let’s look at our audio track. It corresponds to the second ‘trak’ atom. Tables ‘stco’ and ‘stsc’ tell us where audio media is fetched. First entry of tables gives us: at address 98312, 48048 samples.

Now let’s look at what we have at this address. Addr 98312 (hexadecimal 0x18008) is just after ‘mdat’ atom, and Linear PCM audio data starts here as expected.. Everything happens like in a valid movie file.

0018000: 0bd1 afd4 6d64 6174 0b0c 0b0c bc0b bc0b  ....mdat........
0018010: 780b 780b 9b0b 9b0b 390b 390b 190a 190a  x.x.....9.9.....
0018020: 0109 0109 0d08 0d08 3507 3507 e106 e106  ........5.5.....
0018030: 3507 3507 4d07 4d07 a506 a506 b006 b006  5.5.M.M.........
0018040: 8007 8007 9607 9607 f706 f706 9306 9306  ................
0018050: 4b06 4b06 8b05 8b05 cf04 cf04 3a04 3a04  K.K.........:.:.

Black Video

Now let’s looks at first video entry: Address is correct, but length is not. This explain the black image: H.264 cannot be correctly decoded. If you dig deeper, you see that audio and video tables don’t correspond to actual data inside the file. Maybe those tables refer to the last recorded video.

If you look at more corrupt videos, the same facts hold true. Everything can finally be explained:

• A fixed-length ‘moov’ atom is written at the beginning of the file. This is why it can be open.
• Audio and Video tables are wrong. This is why video is corrupt.
• But since first Audio block is always at the same address, and has same length, it’s correctly reproduced.

What it tells about Canon 5D Mark III firmware

Canon Mk III firmware writes a placeholder ‘moov’ atom at the beginning at the file, then writes media data (video and audio) while camera is recording. This placeholder is not empty, it could actually be the same ‘moov’ as in the last recorded video.

When recording ends, the camera overwrites the placeholder with real ‘moov’ atom containing good media tables.

If this last operation doesn’t complete, the video is corrupt.

Cameras usually write the ‘moov’ atom at the end of the file: Data is written in the same order as it is produced. But Canon 5D Mk3 firmware does it different as we have seen. This is possible because camera limitations dictate a maximum ‘moov’ atom size, so the corresponding space can be “booked” at the beginning of the file (placeholder) and then overwritten.

I can only see one advantage to this: Since ‘moov’ atom is at the beginning of the file, the file can be streamed. We talk of fast-start, streamable videos. This is important for Internet applications, but honestly I don’t see how this matters for Canon 5D workflows.

Apple’s new Fusion Drive and Video Repair

One of the most exciting announcement during Apple Oct. 23 event was the “Fusion Drive”: A mix of solid state drive (SSD) and mechanical hard disk, that brings the best of the two worlds: High-speeds of SSDs and high-capacity of HDDs.

How the “magic” works is explained here.

The Catch

But if you read Apple’s technical note, you will see that this doesn’t work great for high-speed video capture.

My understanding is that high-speed, high-volume writing operations have a serious bottleneck, because when the 4 GB SSD write cache is full, it has to be moved to the hard disk, and this causes some latency that probably drops frames during video capture.

Video Repair?

How does it affect video repair?

Video capture limitations will probably keep video professionals away from those Fusion Drives, at least for video capture. High-throughput Thunderbolt drives are a better fit.

But Fusion Drives will become mainstream among casual users. I fear that this will make our video repair job harder, in particular for DeepMediaScan jobs. (DeepMediaScan is used when footage is not found inside files, but scanned directly on disk)

With traditional disks, footage is written sequentially as the disk fills up, so it’s possible to for DeepMediaScan to extract it consistently.
In a Fusion Drive, it’s more complex. Footage is written to an SSD cache and then, maybe it’s transferred to the hard disk, or maybe not. In the end, there is more fragmentation, and footage recovery is more challenging.

As soon as I get get my hands on a Mac with a Fusion Drive, I will carry out repair tests and provide advice about Fusion Drive usage.

Stop Whining and Start Measuring

Last month, I have started to collect data about how visitors are browsing our website. Just standard analytics at work here.

First surprise: 40% of Treasured downloads are from Windows users, who will obviously never be able to run our Mac video recovery app.

Despite clearly stating on the website that this is a Mac app, a lot of Windows people is reaching the big Download button. This is not cluelessness, just the standard practice in a “we don’t read, we scan” world.

Second surprise: Among Mac people with a fresh copy of Treasured just downloaded, only 60% perform a diagnostics on a damaged video. In other words, 40% never get any value from the software.

I could give more examples. Every time I have measured a new metric related to our repair service use, I have discovered that it was grossly inefficient.

The most amazing thing: Those problems are usually very easy to fix. You can just add a warning for Windows guys trying to download Treasured and redirect them to MP4repair.org, for example.

This is a real eye-opener. Don’t start working on hard problems until you get the basic stuff right.

The Secret: Measuring