SpaceX Falcon First Stage Landing (Part II)

This post is the second of a serie, explaining how at Aero Quartet we managed to recover the only existing pictures of the Falcon rocket “landing” on the Atlantic Ocean, first published on Elon Musk (SpaceX CEO) twitter on April 28.

Why the video stream can’t be recovered

First of all, I will explain why it’s pointless to try to recover the video, and why one must focus on recovering cherry-picked frames.

The rocket on-board camera has transmitted the video stream by radio to a plane flying in expected splash-down area. Unfortunately, the radio transmission was bad, probably due to distance and bad weather conditions.

The stream file has now been made public by SpaceX (download), and it’s identical to the file that we had access to a few days prior to public release: It’s a short (4 MB) file containing transport stream headers and footage encoded in MPEG4 Video format at 704 x 480.

I’m a fan of SpaceX, so when I first watched the video I was devastated: There’s not a single good frame in there, you can only guess some movement and boosters firing towards the end.

MPEG4 Video encoding is not redundant, so any error, even a single bit missing or flipped, will cause decoding to fail, leaving us with a garbled frame. Even worse, since MPEG4 Video uses temporal compression, a fault in a keyframe will cause the following 15 frames to be garbled as well.

I’ve been repairing videos full time for the last 7 years, so I immediately knew that the bitstream was corrupt and that at best we could hope to partially fix some frames. Maybe a 50% chance of recovering one or two frames.

Based on previous “edge-of-impossible” repairs, I know that the error rate must be close to zero. A 20kb frame, for example, contains 160,000 bits. If more than 2 or 3 of them are wrong, it becomes impossible to repair it.

This gives us a maximum error rate of 1/50,000.

Using Transport Stream Headers as Estimators of Error Rate

Transport Stream headers (TSH) are quite predictable: Every 188 bytes (this can also be 192 bytes when error correction is used) we have 4 bytes that follow this pattern:

47 03 E8 10
47 03 E8 11
47 03 E8 12
47 03 E8 13
47 03 E8 14
47 03 E8 15
47 03 E8 16
47 03 E8 17
47 03 E8 18
47 03 E8 19
47 03 E8 1A
47 03 E8 1B
47 03 E8 1C
47 03 E8 1D
47 03 E8 1E
47 03 E8 1F
47 03 E8 10 ... and repeating 11, 12, 13, ...
(note that this has been simplified)
(there are also special headers for padding and for beginning of frames)

TSHs, because they follow this pattern, are perfect to estimate the error rate in the stream: Assuming that bit errors happen randomly, if we find errors in the TSHs, then we also have errors in the payload and repair becomes impossible.
But if we find a segment of the stream without any error in the TSHs, we can have good confidence that the segment is almost error-free, hence maybe it can be repaired.

So let’s plot the good and bad areas of the file. For this, I’m using a tool called X-Rays that uses space-filling curves to represent properties of large files graphically. Don’t search for X-Rays on the Internet. It’s a proprietary tool used at Aero Quartet only. We may release it to the public in the future, but that’s another story.

In this particular case, I have programmed a custom filter that displays a black pixel where TSHs are corrupts or missing, and a white pixel where it looks healthy:

Necessary but not Sufficient

At this point, you may say that over 80% of the image is white, so the situation isn’t that bad.

But remember that this is encoded video with temporal compression, so in order to decode something, we need large areas free of black pixels, and such areas must contain at least a keyframe. That leaves us with very few possible spots that could be repaired.

Below I have plotted the file again with X-Rays, but with a filter that shows only keyframes.
Each Tetris-shaped area corresponds to a keyframe. Pixels in white correspond to good TSHs inside the keyframe, and pixels in grey to bad ones.

As you can see, most keyframes are severely “contaminated” by bad TSHs.

I have seen in some forums that people is trying to improve the video by fixing the TSHs.
This doesn’t hurt, but it will not improve the video, because while TSHs are easy to fix, a bad TSH indicates that probably several bits in the neighborhood are also bad. And those are the critical ones, and video won’t improve until you fix them all.

As mentioned previously, an error rate higher than 1/50,000 is impossible to fix. And if you have bad TSHs you probably have an order of magnitude more errors than that in the neighborhood.

Cherry Picking Frames

Now it’s time to cherry-pick the frames that we will try to repair.
When confronted to such a hard problem, I always start with the easiest case. And here, there’s clearly a winner among the keyframes: F199
F199 is almost entirely error-free. Second comes F215, then the rest is probably not worth looking at.

F199 stands for frame #199 of re-wrapped video.

Indeed, during the rest of the recovery I’m working on a re-wrapped video file.
I’ve used ffmpeg to rewrap TS in MOV. This operation doesn’t affect encoding, it only removes TSHs and gets rid of areas where TSHs are corrupt:
ffmpeg -i raw.ts -vcodec copy raw.mov

raw.mov is smaller than raw.ts, but contains the same information. What has been removed corresponds to black pixels in first X-Rays image.

In the raw video, F199 is unsurprisingly the best frame, from a visual point of view. The upper third of the picture is perfectly decoded. The rest is the picture looks like smear applied vertically, which is what is displayed when the MPEG4 Video decoder gives up.

In the next post I will explain how I managed to go from this to that:

Read next post