#2 DJI drones

During this 2015 we did a lot of drone footage repairs. A lot. Different drone models, brands, settings, but all of them share some common points. A worried customer and some non working files.
Crashed Drone
Back in 2013 and 2014, when all the drones fever started, we used to receive a lot of repair requests with, more or less, same content. Rude flight footage ending with a hard landing.

Fortunately, seems that our 5 Tips to Avoid Crashing your Copter on the First Day were taking into account and the skills of all of those pilots improved during this time.

But not only man skills were improved, but also the drone capabilities (Worry-free autopilot, Built-in practice flight simulator, etc). Now the copters are able to flight longer, faster, higher, farther and, what’s more interesting, they’re able to broadcast the live footage that is being recorded (Live HD view).

[UPDATE]: Seems DJI removed human pilot skills from the equation in their latest released model, the DJI Phantom 4.
Those factors changed our view about the drone’s damaged files. From a basic repair approach we evolved to a recovery one. Now the customer finds that sometimes their files are simply gone. Only a blank, usually mistakenly, formatted card is left behind. Data should be found and afterwards repaired.

That’s where our magic begins…

What could be done when a whole card, containing the aerial takes of your Thailand trip, is formatted before all clips had been backed up on your computer’s hard drive? Reshoot seems not an option but thanks to Treasured and it’s DeepMediaScan feature we can bring back the footage.

Thailand Aerial View

This particular case was involving a DJI Phantom 3 drone and 1080p at 50 fps video only (no Audio track). Despite DeepMediaScan results were promising, there was something weird within the data that caused any Video player closes when trying to play the clips. Seemed clear we must go the extra mile.

After some data struggling, we finally find the clue. There were some blocks of alien data (garbage data from any other source but not the footage we’re trying to repair) of 0x100000 bytes each every 4 seconds of Video.

That required a custom tailored cleaner to accurately remove them and, finally, repair the real footage data. Those blocks of data, as after some more study revealed, were in fact, parts of the proxy video that is being broadcasted from the drone to the remote control.

Looks similar to the good known Kayak problem with the GoPro cameras, but this time, those blocks of data are formatted in raw H264 instead of LRV (Low Resolution Video). Lesson learnt and repair flow updated.

Nothing compares with the satisfaction of the positive feedback from the customers when they get back their clips repaired. That make us feel that the effort put in every request is worthy.

Please fly your drone in the safest way possible, improve your pilot skills, and use top quality memory cards to record your footage. And obviously, have fun!

When everything else fails, we’ll be here ready to help you.

#3 XAVC-S

Thanks to the success of Sony A7 series and Sony AX series, XAVC-S is today one of the most common format that we are requested to repair, and so it deserves its number 3 ranking in my list of Tops and Flops of 2015.

In Sony XAVC family, the “S” flavor is the mainstream one, used by a wide range of devices, from DSLR cameras to cheap consumer camcorders.

In its novel “The Leopard”, Tandredi has the famous quote:

If we want things to stay as they are, things will have to change.

This illustrates perfectly what XAVC-S is in essence: A new name for our usual suspect, AVC/H.264

Back in 2013 when XAVC was unveiled, there wasn’t a lot of choice as far as video compression is concerned:
Once discarded the future technology (HAVC/H.265) that wasn’t ready, Sony could only pick either 20-years old MPEG-2 or 10-years old AVC/H.264.

H.264 scales to 4K, has better compression and quality, so the technical decision was easy.
But the marketing guys were not happy with that, because

you know, everybody else is doing H.264, we are Sony and for our new 4K products we need differentiation. We will use H.264 but it must be with another fancy new name.

OK let’s call it XAVC.

And the marketing guy to add: “and we need different brand names for cinematography-grade, professional and consumer H.264″ and so were invented the names XAVC-I, XAVC-L and XAVC-S.

Therefore, XAVC-S is nothing more than regular AVC/H.264 wrapped in a standard MP4 container.

Change Management

Let me tell you my first contact with XAVC-S in spring 2014.
Ironically, this was a corrupt recording of a conference on “Change Management”…

I was expecting this to be business as usual, a boring video (change management, anybody?) recorded in standard H.264 in MP4 container.
Apparently nothing new under the sun.. but I was wrong.

What first grabbed my attention was Treasured insistence in reporting “MXF”. In a video allegedly wrapped in MP4, the last thing you expect to detect is precisely MXF.

Treasured sometimes has “false positives”, so I fired my hex editor to check what was going on.
Upon searching for the MXF header (060e 2b34), I quickly found lots of matches inside the file.


00000c0: 0008 0100 00ad 034b 060e 2b34 0253 0101 .......K..+4.S..
00000d0: 0c02 0101 0101 0000 8300 000c 8000 0002 ................
00000e0: b3e3 8001 0002 51ad 060e 2b34 0253 0101 ......Q...+4.S..
00000f0: 0c02 0101 0201 0000 8300 0036 8100 0010 ...........6....
0000100: 060e 2b34 0401 010b 0510 0101 0101 0000 ..+4............
0000110: 8101 0001 0181 0900 0800 0000 0100 0000 ................
0000120: 6481 0a00 0200 0081 0c00 0200 6481 0d00 d...........d...
0000130: 0101 060e 2b34 0253 0101 0c02 0101 7f01 ....+4.S........
0000140: 0000 8300 002f e000 0010 9669 0800 4678 ...../.....i..Fx
0000150: 031c 2051 0000 f0c0 1181 e300 0001 00e3 .. Q............
0000160: 0200 0100 e303 0001 ffe3 0400 0844 2015 .............D .
0000170: 0625 0321 5100 0000 0000 0000 0000 0000 .%.!Q...........

Obviously Treasured was right, the Sony engineers had inserted MXF stuff inside their XAVC-S MP4 file. WTF?

I asked Mike (my “Change Management” customer) to send me a playable XAVC-S file, so I could investigate this in depth.

What were they smoking?

What I discovered was even more puzzling: MXF data is indeed part of a bigger, more byzantine thing.

The bits of MXF are inside samples of a real time metadata track of the MP4.
To every video frame corresponds one of such metadata sample of 1024 bytes.

Metadata samples look like the diagram below, and are padded with 00 bytes until filling the 1024 bytes size.
Others samples also include a mysterious a “kkad” atom at the end (more on this below).

Header was easy to understand:
0008 0100 00ad 034b
It reports the length in bytes of the components of the sample:

  • 8 bytes for the header itself
  • 0xAD bytes for the MXF KLV (Key Length Value)
  • 0×34B bytes for padding or kkad data.

And of course, the sum is 1024.

MXF KLV itself contains several items, but most are encoded in a proprietary format, and I haven’t managed to understand everything.
In any case, it is not very important and I can prove it:

This metadata is not even necessary.
For two years, I have been repairing XAVC-S without bothering to include the metadata inside the repaired videos, and no customer has reported any problem!

Therefore, Sony engineers have managed a real tour de force:
They have created a video format whose unique “new” feature (besides the shiny XAVC-S name) is to stuff some metadata that is not even being used!

To really appreciate the feat, let’s take a look at this mysterious kkad stuff:

“kkad” blocks include tables with addresses, lengths, and time code information of every video sample.

And with this we close the circle of stupidity:
This information would be useful if it wasn’t already included in the standard MP4 video track.

Yes, Sony engineers have added a metadata track with a proprietary and byzantine format that not only isn’t used, but contains information already present elsewhere in the MP4 container.

And there’s more strange stuff throughout our XAVC-S file…
For example, inside the H.264 stream we have NAL objects of type 6 with more metadata “smoke”.
Or the unnecessary file header below:


0000000: 0000 001c 6674 7970 5841 5643 0100 1fff ....ftypXAVC....
0000010: 5841 5643 6d70 3432 6973 6f32 0000 0094 XAVCmp42iso2....
0000020: 7575 6964 5052 4f46 21d2 4fce bb88 695c uuidPROF!.O...i\
0000030: fac9 c740 0000 0000 0000 0003 0000 0014 ...@............
0000040: 4650 5246 0000 0000 2000 0000 0000 0000 FPRF.... .......
0000050: 0000 002c 4150 5246 0000 0000 0000 0002 ...,APRF........
0000060: 7477 6f73 0000 0000 0000 0000 0000 0600 twos............
0000070: 0000 0600 0000 bb80 0000 0002 0000 0034 ...............4
0000080: 5650 5246 0000 0000 0000 0001 6176 6331 VPRF........avc1
0000090: 0164 002a 0003 0002 0000 c350 0000 c350 .d.*.......P...P
00000a0: 0032 0000 0032 0000 0780 0438 0001 0001 .2...2.....8....

Besides the funny “XAVC”, we have here a description of the tracks:
Audio “twos” at 48000Hz (0xbb80)
Video “avc1″ with resolution 1920 x 720 (0×780 0×438)

Again, all this information was already present in the MP4 tracks, so why duplicate it in other places?

XAVC-S: Engineered with love by Sony

If you are in software or firmware development and you have been working in a large organization, the reasons behind all this oddity should now be clear to you.

This is what I call “Frankenstein Engineering”, and it works like this:

  • Marketing decides that a new product with features A, B and C shall be developed
  • Management green lights the project, and gives the R&D departments a short schedule and a low budget
  • Every department has no choice but picking existing, off-the-shelf components and shoe-horn them into the product
  • Extra points if the different departments are spread across continents, have incompatible road maps, and don’t talk to each other anyway
  • When you put hardware, electronics, firmware and software together it’s a disaster, but it can be fixed
  • Eventually the company manages to ship a decent product, but internally the scar tissue is still visible

Therefore, all the odd things I have discovered inside those XAVC-S files was just that:

Scar tissue of the rushed development.

But let’s give credit to Sony: XAVC-S is very successful, and it delivers what Sony promised despite being a Frankenstein monster made with parts that barely fit together and ignore each others.

And the last word about odd XAVC-S again to Tancredi: “A house of which one knew every room wasn’t worth living in.”

#4 Sony XAVC

XAVC comes at #4 and #3 in my list of Tops and Flops of 2015.

You already know that XAVC has 3 variants: (bonus: wiki link for those who want a refresher)


XAVC-I is intra-frame, the highest quality variant
XAVC-L is long GOP variant
XAVC-S is consumer-oriented and will be the subject of next post: #3 Sony XAVC-S

But did you know that’s there a fourth variant and Sony is not even aware of it?

“XAVC Mark II” by Canon…

A few weeks ago, I have received a repair request for a format that Treasured could not detect.
So I parsed the damaged clip manually, and quickly convinced myself that it was a new kind of XAVC-I:
Same MXF container, same data layout, same H.264/MPEG-4 AVC video and Linear PCM audio channels.

And I also found an explanation for Treasured not detecting it as XAVC-I: H.264 encoding settings were slightly different than those I had trained Treasured for.

I thought it was just another flavor of XAVC-I, but found strange that after two years of dealing with XAVC-I, new flavors would still appear.
Nevermind, I did some tweaks to an existing XAVC-I repair kit, and I could repair the sample.

When I sent the XAVC-I preview to the customer, to my surprise he answered that preview was great, but that I got the format wrong: It was not XAVC-I, but rather Canon XF-AVC from the new C300 Mark II.

Rebuttal, the hard way.

What? I could hardly believe it, but evidence was damning:

  1. Google: Canon XF-AVC is the new 4K format in town and everybody is talking about the Canon C300 Mk II. How I had missed that remains a mystery.
  2. A quick MXFDump on the damaged file was saying it loud and clear:
  3. [ K = MXFIdentification ( 0000000000000a10 )
      [ k = CompanyName
      3c.01, l =    10 (000a) ]
           0  00 43 00 41 00 4e 00 4f 00 4e                      .C.A.N.O.N
      [ k = ProductName
      3c.02, l =    64 (0040) ]
           0  00 45 00 4f 00 53 00 20 00 43 00 33 00 30 00 30    .E.O.S. .C.3.0.0
          10  00 20 00 4d 00 61 00 72 00 6b 00 20 00 49 00 49    . .M.a.r.k. .I.I
    

    and the final nail in the coffin:

  4. X-Rays on the first 20 MB shows clear differences (left is Sony XAVC-I, right Canon XF-AVC)


X-Rays is an internal tool using space-filling curves to visualize remarkable structures inside damaged video files.

So what is actually going on?
On one side, Canon XF-AVC uses that exact same ingredients as Sony XAVC for its new 4K format, on the other side there are small implementation differences that make the two formats distinguishable if you have the right tools.

Technical false twins, if you will.

The XAVC Marketing Stunt

Canon has pulled exactly the same marketing stunt as did Sony two years ago:
Rebrand H.264/AVC as Canon XF-AVC to address 4K with a “new” format.

And there’s a good reason Canon did the same: because it worked so well for Sony with XAVC!

The advantages of coining a nice format name are manifold:

  • “New” is a magic word. Every marketer knows that.
  • Under XAVC umbrella you can encompass a variety of specs that otherwise would create confusion:
  • XAVC-S for H.264 in MP4 container
    XAVC-I for raw H.264 in MXF container, intra frame level 5.2
    and future variants can also use the XAVC branding

  • It is easier to communicate on XAVC compatibility, to certify or license XAVC third-party solutions, than to communicate on complicated H.264/AVC profiles and specifications.

The XAVC marketing trick has given a two years head start to Sony over Canon in the 4K transition.
In next post, we will explore a few interesting properties of XAVC and how Canon XF-AVC approach tends to differ.

#5 External Recorders

External Recorders come at #5 in my list of Tops of 2015.

A few years ago, the video camera was the most important device used in production. It included several functions: optics, sensor, encoding and storage. But the trend is to do encoding and storage outside of the camera, in an external video recorder.

Solid state drives (SSD) and high-speed connectivity through HD-SDI and Thunderbolt enable live recording of RAW and uncompressed feeds from 2K and 4K cameras. ProRes and DNxHD formats, previously only used in post-production, are now productions formats.


Camera <----------------> Aja Ki Pro Quad

Where we had a closed device, now we have an open system that is more versatile, but also more fragile.

Lessons of Redundancy

A few weeks ago, Mark contacted me to recover a 50 GB file containing footage from a concert.
The event was recorded in DNxHD format by an Atomos Ninja 2 recorder.

Mark is a seasoned professional, he always run two recorders for redundancy, prime and backup.
The prime recorder stopped and no one noticed until the end of the program.
This situation is not frequent, but it can happen, this is why a backup recorder must always be running when recording a live event.

Unfortunately, the backup recorder also had its share of problems, it crashed as well!

The statistics say that if a system has a failure rate of 1%, when you run two systems for redundancy the chances of having the two systems fail at the same time will be extremely low:

1% of 1%, or 1 out of 10000

That’s why redundancy matters if you are to keep your reputation as a professional:
1 out of 100 will happen almost every year, but 1 out of 10000 is more like doing a hole in one in golf: It may happen once in your carreer, and it may never happen at all.

But that’s the theory.

In the real world, I’m seeing far more failures in redundant recordings than the statistics predict.
Let me explain the three pitfalls of redundant recording and how to avoid them.

Redundancy Pitfall #1: Prime vs Backup

Mark had two recorders set-up, but had they the same chance of success?

Probably not. My guess is that the prime recording set-up was first rate whereas the backup set-up was a disaster waiting to happen.
For the prime recorder, Mark had used the gear in best condition, a very reliable recorder with encoding power in excess and fast storage.
For the backup set-up, Mark had used the old recorder, a bit underpowered for the job, and a memory card that had never given problems in the past, but that is 80% full (of footage from a previous job…)

Does it sound familiar? Mark had a redundant set-up, but the backup recorder had a failure rate probably around 20% instead of theorical 1%.
Unfortunately, this was one of those bad days for Mark: With the backup recorder struggling with encoding, some frames started dropping, and as the card was getting filled, fragmentation caused the write speed to plummet, finally causing the recorder to crash.

Therefore, a disaster like this is not due to a single failure, but to an accumulation of problems.
Redundancy is only safe if the two recorders are working in optimal conditions. Or to put it in another way: Your reputation as a video professional is in the hands of the less reliable of your two recorders, usually the backup system.

The mere labelling of the recorders as “prime” and “backup” is the first breach of redundancy. The two are equally important, and should rather be labelled “prime even” and “prime odd” and you would use the footage from the even recorder on even days, from the odd recorder on odd days.
That would protect you against the “broken backup” syndrome, more on that in pitfall #3.

Redundancy Pitfall #2: Independence

In statistics, one of the most important concept is variable independence:
An independent variable is a variable whose variation does not depend on that of another.

The theorical “1 out of 10000″ rate is based on the assumption of independent failures.
It means that a common cause can never produce a failure in the two systems.

In the real world, this assumption is often false!
For example, if the recorders are connected to the same power source, or if they are writing to the same storage device, it’s clear that we don’t have redundancy, only the illusion of it.

So the solution is to have two identical systems (to avoid pitfall #1) but that are fully independent? Nope.

Identical systems are not independent due to latent flaws. That’s more subtle, but can be as devastating:
Imagine two identical recorders: same model, same firmware, same recording settings… If this model of recorder starts to fail when temperature is over 35 degrees, the two recorders will be at risk because they are exposed to the same temperature and workload. The second recorder will probably fail a few minutes after the first one.
Same for firmware bugs: if a recorder crashes under a certain condition, the other one will also probably crash.

In the Space Shuttle, there were 5 computers running in parallel, but one of them running software written independently. This was the safeguard against a bug affecting all computers.

Therefore, to build two truly independent systems is harder than it seems.
You only operate a redundant system when both prime and backup are equally reliable and truly independent.

But there’s a last problem…

Redundancy Pitfall #3: Supervision

Redundancy is its own enemy.
We are humans, and when we know that something is almost 100% safe, we tend to take success for granted and to stop worrying.

Lack of supervision is what can finally put your redundant system at risk.

If you watch your two recorders, you will immediately notice when one of them stops, and take action.
On the other extreme, if you never check the results of your backup recorder (because prime recorder works), you can overlook a systematic failure present in your backup procedure, and you are no longer protected. That’s the “broken backup” syndrome.

RAID 5 is a redundant storage system where 4 hard disks work together, so that if any of the 4 disks fails, it can be replaced by a new disk, and without stopping the system the remaining 3 healthy disks “rebuild” the content of the failed disk, and redundancy is restored after a few minutes.
In theory, RAID 5 systems are almost 100% reliable, because the data is only vulnerable during the few minutes where rebuilding takes place. The rest of the time, the system is redundant.

However, the #1 failure mode of RAID 5 systems is lack of supervision: Nobody detects that one of the disk has failed, and the RAID5 continues to work in degraded mode for weeks or months. But one day, a second disk starts having errors, and unfortunately it’s too late to rebuild the system, with two failed disks the system has become unrecoverable.

Therefore, please add this to your new year resolutions list:

I will verify my redundant recording procedure and I will:

1. Label my recorders “Even” and “Odd” and, based on the day of the month, use one or the other as “Prime”
2. Verify that my two systems are truly independent, that a single cause cannot produce a failure in the two systems
3. Watch my recorders and investigate any issue detected.

Happy new year 2016!