#4 Sony XAVC

XAVC comes at #4 and #3 in my list of Tops and Flops of 2015.

You already know that XAVC has 3 variants: (bonus: wiki link for those who want a refresher)

XAVC-I is intra-frame, the highest quality variant
XAVC-L is long GOP variant
XAVC-S is consumer-oriented and will be the subject of next post: #3 Sony XAVC-S

But did you know that’s there a fourth variant and Sony is not even aware of it?

“XAVC Mark II” by Canon…

A few weeks ago, I have received a repair request for a format that Treasured could not detect.
So I parsed the damaged clip manually, and quickly convinced myself that it was a new kind of XAVC-I:
Same MXF container, same data layout, same H.264/MPEG-4 AVC video and Linear PCM audio channels.

And I also found an explanation for Treasured not detecting it as XAVC-I: H.264 encoding settings were slightly different than those I had trained Treasured for.

I thought it was just another flavor of XAVC-I, but found strange that after two years of dealing with XAVC-I, new flavors would still appear.
Nevermind, I did some tweaks to an existing XAVC-I repair kit, and I could repair the sample.

When I sent the XAVC-I preview to the customer, to my surprise he answered that preview was great, but that I got the format wrong: It was not XAVC-I, but rather Canon XF-AVC from the new C300 Mark II.

Rebuttal, the hard way.

What? I could hardly believe it, but evidence was damning:

  1. Google: Canon XF-AVC is the new 4K format in town and everybody is talking about the Canon C300 Mk II. How I had missed that remains a mystery.
  2. A quick MXFDump on the damaged file was saying it loud and clear:
  3. [ K = MXFIdentification ( 0000000000000a10 )
      [ k = CompanyName
      3c.01, l =    10 (000a) ]
           0  00 43 00 41 00 4e 00 4f 00 4e                      .C.A.N.O.N
      [ k = ProductName
      3c.02, l =    64 (0040) ]
           0  00 45 00 4f 00 53 00 20 00 43 00 33 00 30 00 30    .E.O.S. .C.3.0.0
          10  00 20 00 4d 00 61 00 72 00 6b 00 20 00 49 00 49    . .M.a.r.k. .I.I

    and the final nail in the coffin:

  4. X-Rays on the first 20 MB shows clear differences (left is Sony XAVC-I, right Canon XF-AVC)

X-Rays is an internal tool using space-filling curves to visualize remarkable structures inside damaged video files.

So what is actually going on?
On one side, Canon XF-AVC uses that exact same ingredients as Sony XAVC for its new 4K format, on the other side there are small implementation differences that make the two formats distinguishable if you have the right tools.

Technical false twins, if you will.

The XAVC Marketing Stunt

Canon has pulled exactly the same marketing stunt as did Sony two years ago:
Rebrand H.264/AVC as Canon XF-AVC to address 4K with a “new” format.

And there’s a good reason Canon did the same: because it worked so well for Sony with XAVC!

The advantages of coining a nice format name are manifold:

  • “New” is a magic word. Every marketer knows that.
  • Under XAVC umbrella you can encompass a variety of specs that otherwise would create confusion:
  • XAVC-S for H.264 in MP4 container
    XAVC-I for raw H.264 in MXF container, intra frame level 5.2
    and future variants can also use the XAVC branding

  • It is easier to communicate on XAVC compatibility, to certify or license XAVC third-party solutions, than to communicate on complicated H.264/AVC profiles and specifications.

The XAVC marketing trick has given a two years head start to Sony over Canon in the 4K transition.
In next post, we will explore a few interesting properties of XAVC and how Canon XF-AVC approach tends to differ.

#5 External Recorders

External Recorders come at #5 in my list of Tops of 2015.

A few years ago, the video camera was the most important device used in production. It included several functions: optics, sensor, encoding and storage. But the trend is to do encoding and storage outside of the camera, in an external video recorder.

Solid state drives (SSD) and high-speed connectivity through HD-SDI and Thunderbolt enable live recording of RAW and uncompressed feeds from 2K and 4K cameras. ProRes and DNxHD formats, previously only used in post-production, are now productions formats.

Camera <----------------> Aja Ki Pro Quad

Where we had a closed device, now we have an open system that is more versatile, but also more fragile.

Lessons of Redundancy

A few weeks ago, Mark contacted me to recover a 50 GB file containing footage from a concert.
The event was recorded in DNxHD format by an Atomos Ninja 2 recorder.

Mark is a seasoned professional, he always run two recorders for redundancy, prime and backup.
The prime recorder stopped and no one noticed until the end of the program.
This situation is not frequent, but it can happen, this is why a backup recorder must always be running when recording a live event.

Unfortunately, the backup recorder also had its share of problems, it crashed as well!

The statistics say that if a system has a failure rate of 1%, when you run two systems for redundancy the chances of having the two systems fail at the same time will be extremely low:

1% of 1%, or 1 out of 10000

That’s why redundancy matters if you are to keep your reputation as a professional:
1 out of 100 will happen almost every year, but 1 out of 10000 is more like doing a hole in one in golf: It may happen once in your carreer, and it may never happen at all.

But that’s the theory.

In the real world, I’m seeing far more failures in redundant recordings than the statistics predict.
Let me explain the three pitfalls of redundant recording and how to avoid them.

Redundancy Pitfall #1: Prime vs Backup

Mark had two recorders set-up, but had they the same chance of success?

Probably not. My guess is that the prime recording set-up was first rate whereas the backup set-up was a disaster waiting to happen.
For the prime recorder, Mark had used the gear in best condition, a very reliable recorder with encoding power in excess and fast storage.
For the backup set-up, Mark had used the old recorder, a bit underpowered for the job, and a memory card that had never given problems in the past, but that is 80% full (of footage from a previous job…)

Does it sound familiar? Mark had a redundant set-up, but the backup recorder had a failure rate probably around 20% instead of theorical 1%.
Unfortunately, this was one of those bad days for Mark: With the backup recorder struggling with encoding, some frames started dropping, and as the card was getting filled, fragmentation caused the write speed to plummet, finally causing the recorder to crash.

Therefore, a disaster like this is not due to a single failure, but to an accumulation of problems.
Redundancy is only safe if the two recorders are working in optimal conditions. Or to put it in another way: Your reputation as a video professional is in the hands of the less reliable of your two recorders, usually the backup system.

The mere labelling of the recorders as “prime” and “backup” is the first breach of redundancy. The two are equally important, and should rather be labelled “prime even” and “prime odd” and you would use the footage from the even recorder on even days, from the odd recorder on odd days.
That would protect you against the “broken backup” syndrome, more on that in pitfall #3.

Redundancy Pitfall #2: Independence

In statistics, one of the most important concept is variable independence:
An independent variable is a variable whose variation does not depend on that of another.

The theorical “1 out of 10000” rate is based on the assumption of independent failures.
It means that a common cause can never produce a failure in the two systems.

In the real world, this assumption is often false!
For example, if the recorders are connected to the same power source, or if they are writing to the same storage device, it’s clear that we don’t have redundancy, only the illusion of it.

So the solution is to have two identical systems (to avoid pitfall #1) but that are fully independent? Nope.

Identical systems are not independent due to latent flaws. That’s more subtle, but can be as devastating:
Imagine two identical recorders: same model, same firmware, same recording settings… If this model of recorder starts to fail when temperature is over 35 degrees, the two recorders will be at risk because they are exposed to the same temperature and workload. The second recorder will probably fail a few minutes after the first one.
Same for firmware bugs: if a recorder crashes under a certain condition, the other one will also probably crash.

In the Space Shuttle, there were 5 computers running in parallel, but one of them running software written independently. This was the safeguard against a bug affecting all computers.

Therefore, to build two truly independent systems is harder than it seems.
You only operate a redundant system when both prime and backup are equally reliable and truly independent.

But there’s a last problem…

Redundancy Pitfall #3: Supervision

Redundancy is its own enemy.
We are humans, and when we know that something is almost 100% safe, we tend to take success for granted and to stop worrying.

Lack of supervision is what can finally put your redundant system at risk.

If you watch your two recorders, you will immediately notice when one of them stops, and take action.
On the other extreme, if you never check the results of your backup recorder (because prime recorder works), you can overlook a systematic failure present in your backup procedure, and you are no longer protected. That’s the “broken backup” syndrome.

RAID 5 is a redundant storage system where 4 hard disks work together, so that if any of the 4 disks fails, it can be replaced by a new disk, and without stopping the system the remaining 3 healthy disks “rebuild” the content of the failed disk, and redundancy is restored after a few minutes.
In theory, RAID 5 systems are almost 100% reliable, because the data is only vulnerable during the few minutes where rebuilding takes place. The rest of the time, the system is redundant.

However, the #1 failure mode of RAID 5 systems is lack of supervision: Nobody detects that one of the disk has failed, and the RAID5 continues to work in degraded mode for weeks or months. But one day, a second disk starts having errors, and unfortunately it’s too late to rebuild the system, with two failed disks the system has become unrecoverable.

Therefore, please add this to your new year resolutions list:

I will verify my redundant recording procedure and I will:

1. Label my recorders “Even” and “Odd” and, based on the day of the month, use one or the other as “Prime”
2. Verify that my two systems are truly independent, that a single cause cannot produce a failure in the two systems
3. Watch my recorders and investigate any issue detected.

Happy new year 2016!

Tops and Flops of 2015

Time is flying, year 2015 is now almost behind us, so we can start looking backwards and analyze the trends.
In an industry like video production, that reinvents itself every few years, it’s possible to see significant changes on a yearly basis.

Three years that changed the Industry

  • 2013 has seen offering of 4K and Digital Cinematography EXPLODE
  • I remember spending the last months of 2013 adding support to Treasured for a slew of new formats and cameras: ARRIRAW, Sony RAW in CineAlta series, Sony XAVC, Canon Cinema RAW, Blackmagic RAW
    It seemed like every vendor had to release the new stuff before end of year. Some products were clearly not ready for prime time, and have been panned by reviewers and early adopters.

  • In 2014 this first wave of new products went from testing to field use.
  • RAW video functionality, a major new feature present in most of those products, turns out to be rarely used. Over the last 2 years, we have not seen any Canon Cinema RAW user requesting our services…

    Canon Cinema RAW frame
    Treasured preview

    Canon RAW is so rarely used that we haven’t bothered to provide color previews in Treasured…

    Winners and losers of the first round quickly emerged:
    Sony presented a coherent and comprehensive solution called XAVC (coherent at least from marketing stand point, in follow-up posts I will explain why technically XAVC is a clusterf…) and won, Canon with its baroque offering confused everybody and lost.

  • In 2015 came the second wave of 4K products
  • After 2013 products rush and 2014 triage, the manufacturers have now a better understanding of what video professionals need and expect.
    In 2015 we are seeing more mature products, and volumes that indicate that they are becoming mainstream and replacing the previous generation:
    Panasonic Lumix GH4, Sony AX, and Sony alpha 7 are replacing older Canon 7D, Canon C300 and other XDCAM products.
    Canon is also the main loser of 2015, its Canon 7D Mark II is not interesting a lot of people. But 2016 could be their year, as the new Canon C300 Mark II looks like a winner.

My 2015 ranking

The trends I have observed through 2015 can be summarized by this ranking:

#5 External Recorders
#4 Sony XAVC-I (Sony F5/F55) and -L cameras (Sony PXW)
#3 XAVC-S Sony A7S, Sony AX series
#2 DJI drones
#1 Panasonic Lumix GH4

In next posts I will go through the ranking, item by item, and explain surprising things about each…
For video file internals often reveals how things were engineered, what the plans initially were, and what the future allows for.

Help Wanted

Update Apr 2015: Position has been filled. Welcome on board, Marcel!

As we head into 2015, a new opportunity to join the Aero Quartet team has arisen. Love video? Love problem solving? And love making people feel good by helping?

We’ve been looking for you.

We are seeking front-line Video Repair Technicians to help our clients with their corrupt videos.

Ideally, you’ve got:

  • Excellent problem-solving, and ability to “read between the lines” of customer emails
  • Substancial Mac OS X, video and internet experience
  • A professional, courteous, and personable email disposition (Fluent English, spoken and written, is a must)
  • A pleasant personality, patience, and sense of humor

Bonus Points for:

  • Engineering / computer science experience
  • French, German, Russian or Portuguese languages
  • Familiarity with professional film making

You must live in Barcelona, Spain or be willing to relocate to the area. It’s honestly a very nice place.

In addition to base salary, Aero Quartet offers:

  • Indefinite contract (Spanish law)
  • Private medical insurance (in addition to Spanish public insurance)
  • Flexible hours, working from home in select days

Sound good? E-mail your resume to us and if we’re interested, we will send you additional details and possibly schedule an interview.

Contact: benoit@aeroquartet.com