Don’t Let Glitches Spoil your Recovered Videos

A few years ago, when we started repairing XDCAM EX videos, we noticed that output wasn’t clean. Every few frames there was a glitch that would affect both audio and video.

The good news for wedding videographers is that at Aero Quartet, we have collected statistical evidence correlating glitchy videos with happy marriages. Just kidding…

When we examined the raw data inside the damaged videos, we noticed a pattern:

We found that those glitches occured where pages of “alien data” had been inserted in the footage.

In our jargon, a page is a block of data that starts at a remarkable address and has a remarkable length. For example, starting at 0x450000 and with a length of 0x10000 (those values with 0x prefix are hexadecimal numbers – that’s what computers use)
Note that those numbers end with five zeros: those are round numbers.

X-Rays

And those pages contain data this is really different from audio or video, and that makes them also easy to detect:
A page that always starts with AF 7E 12 EF and is almost filled with 00 bytes is in utter contrast with a page of XDCAM video (that looks like random numbers).

We have an internal tool called “X-Rays” that uses space-filling curves to visualize remarkable structures inside damaged video files. If you configure it to detect areas almost filled with 00 bytes, you get that:

Leaving aside lonesome pixels that are not representative, we have 18 “rough spots” in our damaged video, and none in a normal XDCAM file. Each spot corresponds to a page of alien data. If you repair the damaged video as is, the footage will have 18 glitches, because alien data is contaminating audio and video.

Usual Suspects

We quickly figured out that this “alien data” wasn’t really alien. Actually the XDCAM camcorders responsible for the damaged videos all had something in common: They were recording metadata files in parallel with the main video file.

The typical XDCAM EX folder looks like this, with a subfolder for each clip, and several files inside each:

The important one is the .MP4 file that contains the actual clip. Note that its size is orders of magnitude bigger than the rest. The rest with .BIM .PPN .SMI and .XML suffixes are metadata files that describe what the clip contains.

And this solved the mystery:

1. Damaged videos we are asked to repair are actually “recovered” files (using one of those data recovery utilities, or our own DeepMediaScan)
2. Recovery brings back raw data in the same sequence it was written to disk, regardless of the file it belongs to.
3. So if a small bit of a .BIM file was written every second, it ends up in the middle of the .MP4 stream
4. And this causes a visual glitch

Therefore, our “alien data” belongs to .BIM metadata files. (and some traces from the other ones as well).
To achieve a clean output, we must remove the corresponding pages and then repair.

Clean, Rinse, Repeat

For XDCAM EX customers confronted with this problem, we have developed special repair kits that have a “cleaner” built-in. It works like this:

• Our repair kit scans the file and flags all the pages containing alien data
• Then it scans again the file, carefully skipping the flagged pages, and extracts video and audio
• Last but not least, it builds the video and audio index.

For example, the log of the “cleaner” in our damaged file says:

No alien data found at beginning
Found block of alien data: SkipFrom 553000, SkipLength 20000
Found block of alien data: SkipFrom d5e800, SkipLength 10000
Found block of alien data: SkipFrom 1553000, SkipLength 20000
Found block of alien data: SkipFrom 1d53000, SkipLength 20000
Found block of alien data: SkipFrom 2553000, SkipLength 20000
Found block of alien data: SkipFrom 2d53000, SkipLength 20000
Found block of alien data: SkipFrom 3553000, SkipLength 20000
Found block of alien data: SkipFrom 3d53000, SkipLength 20000
Found block of alien data: SkipFrom 4553000, SkipLength 20000
Found block of alien data: SkipFrom 4d53000, SkipLength 20000
Found block of alien data: SkipFrom 5563000, SkipLength 10000
Found block of alien data: SkipFrom 5d53000, SkipLength 20000
Found block of alien data: SkipFrom 6553000, SkipLength 20000
Found block of alien data: SkipFrom 6d53000, SkipLength 20000
Found block of alien data: SkipFrom 7553000, SkipLength 20000
Found block of alien data: SkipFrom 7d53000, SkipLength 20000
Found block of alien data: SkipFrom 8563000, SkipLength 10000
Found block of alien data: SkipFrom 8d53000, SkipLength 20000
Smallest continuous interval: 8257536

What has the cleaner done? It has removed 18 blocks, most of them of length 0x20000 bytes (128kb) and 3 of them of 0x10000. This corresponds exactly to what X-Rays had visualized above.

After cleaning, the repair kit can produce clean video, restoring original quality.

Not just XDCAM

Having several files written in parallel during video recording is what causes recovered XDCAM to be contaminated with alien data.
But other cameras also record by writing several files at once, and it causes similar problems:

• when MXF is recorded with audio and video in separate files
• when GoPro HERO records at once HD and LRV (low resolution) videos

We have also developed “cleaners” for those cases.