TL;DR

• In iOS 12, Voice Memos app has a feature called Replace to edit an original recording.
  Replace is not reliable and can corrupt the original recording.
• Corrupted recordings can be repaired, but are significantly shorter in duration than originals.
  The missing duration is lost forever.

What I Have Observed

I have pulled data from recent Voice Memos requests to see what they have in common.
A pattern quickly emerged:

• All the failures happened on phones running iOS 12, and in consequence, the redesigned Voice Memos application.
  The iPhone model doesn’t matter.
• Failed recordings are always truncated, ie shorter in duration than what the user reports to have recorded, and also shorter than what the phone displays.
• No way to reproduce the recording on the phone or on other devices



I have plotted here a dozen of requests:
The green line is where the dots should be. Unfortunately, all the corrupted recordings are significantly shorter than the original, this is why the dots are below the line.
On average, only one quarter to one third of the audio is still present in the damaged file.

Reproducing the Issue

I made a few experiments to reproduce the problem. On my iPhone 8 and iOS 12, the Voice Memos app has two recording settings: Compressed and Uncompressed.

The samples recorded with each setting have the following audio profiles:
Compressed: AAC format (mono 48000Hz), bitrate 64 kbit/s
Uncompressed: Apple Lossless format (ALAC mono 48000Hz), bitrate 720 kbit/s

When I compare this to the damaged samples that I am receiving, the first thing that strikes me is that format and bitrate are different.
All damaged files come in AAC format, but with a frequency of 44100Hz and a bitrate around 225 kbit/s.

After playing a bit with the app, I finally managed to produce samples with those settings when using the Edit Recording function.
Whenever you use the Replace button to alter the original recording, Voice Memos performs an audio transcoding operation in the background, which converts the format into 44100Hz.

This operation is necessary, because replacing audio disrupts the organization of encoded media. In other words, it cannot be done directly on encoded media, as replacement involves a second audio track, and the two tracks need to be “flattened”, to use the technical word, into a new encoded audio track.

The first take-away is that the failure only occurs when Replace feature is used.

Mechanism of Failure

Now, why does the failure happen? Although I don’t have have access to the source code, I can guess how and why the problem occurs.

The first hint is the disconnect between the user interface and the reality:
Whereas the app gives the impression that Replace is a real-time thing, flattening the recording takes some time to complete.
Even if you replace a few seconds in a 2 hours long recording, your iPhone will need several seconds, maybe even minutes for very long recordings, to produce the modified recording.
This operation happens in the background, but Voice Memos doesn’t reflect this. Instead of showing a progress bar, and labelling the recording as “in process” or “unavailable”, it makes it look like the change is done and the recording is available when in reality it’s not.

From here I can guess at several possible flaws in the Voice Memos software:

• If flattening fails or does not complete, the recording being flattened overwrites the original, as is. This explains why it’s truncated.
• The user interface lets you initiate operations (like a new Edit) on a recording being flattened and this makes the app crash.
• Flattening and replacing the recording occurs as a single-step operation, without proper exception handling. Second step, replacement, should never happen if first step hasn’t been completed and verified first.
• Instead, the software should do it in two steps: First, flattening would make the original clip unavailable and a progress indicator would show on the screen that it’s because it’s being processed. Second, in case of failure, the software would show an error message and revert the state of the recording. Finally, in case of success, the modified clip would replace the original one.

There’s never a good excuse for the sloppy implementation of a feature like Replace.
I suspect that Apple engineers wanted first and foremost to make Voice Memos look cool and to give the illusion of direct manipulation and instant effects. But technology doesn’t work that way, flattening takes time, and not reflecting it in the user interface ended, I believe, in a flawed design.

Letting the user do a new edit while flattening takes place, is when user interface and processing enter in conflict. When the app crashes, and probably when corrupted recordings are generated.

How to Avoid it

Having understood the failure mechanism, I can now give some advice to avoid it:

1. Avoid using the “Edit Recording” as it seems that it’s not a reliable feature

If you have to use it, take some precautions:

2. Always do “Duplicate” and create a back-up of your recording before using “Edit Recording”
3. After editing a Voice Memo, give enough time to your phone to perform the “flattening”.
   In particular, don’t do a second “Edit Recording” or a “Share…” immediately after an edit.

What Our Service Can Do

I often tell my customers that we do magic, but no miracles!
In the Voice Memos case, the magic is to fix the corrupted recording, and the miracle would be to restore the full duration, when only part of it is present in the damaged file.

If you have a corrupted Voice Memo (or any unplayable audio file, from any device), you can use our Treasured service to determine the true duration of audio present in the file, and then to repair it. Unfortunately, for the reasons I have explained above, in Voice Memos the duration is usually significantly shorter than expected, but many customers still consider that for important material, it’s better to recover one third of it than to get nothing.

Cachu Hwch *

* It’s all gone wrong

Connecting the Dots

Fast forward to end of 2016: As I was putting the final touches on Treasured 4, I was a bit annoyed.

The rewritten “engine”, powered by libavcodec, was really shining. I could finally display those high-end Sony XAVC and Canon XF-AVC frames in all their 12-bit glory.

However, my redesign had also a few drawbacks: Some video formats would no longer “render”, due to lack of libavcodec equivalent to some proprietary QuickTime codec.

REDCODE and Sony HDCAM-SR fell into this category, and that was really annoying.
I quickly made some statistics of last years requests using those formats, and decided it wouldn’t be too significant. HDCAM-SR numbers indicated that the format was about to disappear, after decreasing for several years in a row.

Still, we had Sony at both ends of the spectrum: On the head, Sony XAVC, strongly leading the 4K pack; on the tail the dwindling Sony HDCAM-SR.

I couldn’t help but remember what had happened 5 years earlier, the flooded factories, the production lines stopped for months, and I made the connection:

Think a moment about Shiva, the Hindu god of creation and destruction, who creates, protects and transforms the universe… Tragedies always have a flip side – and don’t get me wrong, with over 20,000 casualties we’re better off without tragedy – and for Sony the flooded factories may have been a catalyst, that accelerated transition to new technologies.

But before we analyze the consequences over the years, let me tell you first the inside story of that spring and summer of 2011.

The Twilight of the Tapes

When production of HDCAM-SR tapes suddenly stopped in March 2011, professional video tapes were the medium of choice for prime time, episodic TV production.

Although Sony communicated that they expected a three months downtime, the folks in Hollywood, with only two weeks of HDCAM-SR tape supply and a shooting season about to begin, immediately saw the Sendai factory flooding as an existential threat, and started to look for alternatives.

Some of them were already engaged in a tape HDCAM workflow and had no choice but to frantically purchase all existences of HDCAM-SR tape on sale, to secure production until end of season. Tape prices on gray market surged and despite efforts by Sony to alleviate distribution bottlenecks, stocks evaporated in a few weeks.

For many other productions, there was still a plan B: Accelerate the move to tapeless workflow.

Others have explained in detail the immediate impact on the industry of HDCAM-SR tape shortage. To summarize, the shock wave travelled from production down to post-production and delivery:

• Japan tsunami accelerated a trend that was already happening: the use of tapeless cameras.
• Forced to embrace the new ARRI, RED, and other tapeless cameras, the production houses had their fears quickly dispelled: The result was flawless, technology was ready!
• Post production houses were now receiving files instead of tapes. Despite huge investments in SR decks, they were forced to adapt.
• For delivery of masters to broadcasters, HDCAM-SR tape, the standardized format, was suddenly questioned. If most deliveries were still done in tape, master file transfers through fiber optic lines became common. The future was here.
• Archiving, on the other end, moves at a slower pace. Transition from film to HDCAM-SR was still recent in 2011, but shortage raised the question.

And with all that, the summer passed quickly…

Gloomy Future

In October, seven months after the disaster, Sony finally managed to restart production and to resume shipments of HDCAM-SR tapes.
But so much effort would hardly be rewarded.

Productions that had done the switch during summer would no longer need the tapes. Those who had purchased a supply of tapes in the aftermath of the disaster would not need them either, and had already planned to switch to tapeless the next season.

In years to come, Sony would continue to manufacture tapes, but the writing was now on the wall.
With the sudden decline of a very profitable business that otherwise would have enjoyed several more years of bonanza, and the inevitable loss of dominance in professional video industry, Sony was facing a gloomy future.

Of course, any attempt to regain a dominant position had not only to address the 2011 events, but also to envision the future of professional video, and deliver the new end-to-end solution, a kind of HDCAM-SR for 2015-2020 if you will.

In response to that challenge, Sony correctly identified the opportunities born out of crisis:

• What will replace HDCAM-SR as a standard for post and delivery? Getting rid of HDCAM has opened the eyes of the industry on the importance of a standard medium. Let’s design a solution that can become the new standard!
• 4K is about to happen. Let’s embrace it!
• Professional file-based workflows have specific needs (quality, bitrate, grading, metadata, interoperability, …) Let’s use the appropriate technology: AVC/H.264 and MXF containers

2011 is reaching an end, and now Sony has a plan…

Resurrection

On October 30, 2012, Sony unveiled the XAVC recording format!

In the past 12 months the tape industry had been disrupted and chaos was looming.
Still, someone had to be the first to make a compelling offer, and start building a new order.
Sony had just done it!

Let’s speculate a bit:
In what position would Sony be today in the professional video markets if not for 2011 flooding?

I believe that they wouldn’t be enjoying a two years lead over Canon, healthy profits, and such a dominant position.
Profitable HDCAM line would have slowed them, the risk of cannibalizing their tape business would have discouraged innovation.

Sure, they would have moved to 4K AVC, but too bland too late. They wouldn’t lead.

A corollary to the Innovator’s dilemma is that companies with biggest investments in old and current technologies are the least inclined to invent the next generation of products.
In 2011, Sony tape business was probably in the middle of the “milking” phase. Sony was definitively not the natural innovator, but the tsunami changed everything.

Sony video division had to face the perfect storm, but with the proper timing, vision and resolve, against all odds, they managed an impressive turnabout. A good lesson for the competitors!

Treasured is the cornerstone of our Movie Repair Service, it provides diagnostics and shows preview of the contents of unplayable video files.
A lot of work has gone into Treasured since its first release in 2008, and we are about to release the most important redesign ever.

Our development roadmap follows a tik-tok schedule, where in even version numbers we make changes to the bowels of the application, changes that are not always visible to the user. And in odd version number we make changes to the user interface.

Tik

In January, we have putting the final touches on this “tik” release: We have rewritten almost everything under the hood, yet the on the surface this looks very similar to version 3.4 released in 2016.

The “engine” was redesigned to get rid of dependencies on deprecated parts of macOS (like 32-bit QuickTime and Carbon libraries) that will no longer work in the next years.

Treasured is now a 64-bit application, ready for the next 10 years. This redesign is the foundation upon which future developments will be built.

For the most technically inclined of you, we are now relying on libavcodec (of ffmpeg fame) to render the media found inside the damaged files, and this brings some advantages over last versions:

• H.265, the new high-performance codec, is now detected and previewed. (More about this below)
• Treasured will no longer ask you to install some QuickTime codecs to render images. Now everything comes bundled!
• Very fast selection of “Candidates” for H.264 format

And many more opportunities that we have identified, and that will progressively be deployed in future releases…
In summary, with this Treasured redesign we make a bold statement:
We want to be here for the next ten years.

H.265 aka HEVC (High Efficiency Video Coding)

And those next ten years will likely belong to H.265, the new high efficiency codec that promises to cut bitrates by 50% over H.264.

We have already repaired with success a dozen of H.265 videos last year, mostly from Samsung Gear cameras.
For 2017 and 2018, we are anticipating a surge, as H.265 encoding chips will become mainstream in high-end and DSLR equipment.

Treasured is not just capable to detect and preview H.265, there’s more…
We have figured out that repairing corrupt mov or mp4 files with H.265 encoding is not very different from what we have been doing with H.264 for over 8 years. In other words, all our experience in delivering high-quality, affordable repairs will be immediately available for new H.265 videos.

Tok

2017 will be a busy year, because we are also planning a huge “tok” release!

Treasured user interface is still surprisingly similar to the first version, that I unveiled in this blog in 2008.
This design has served us well, but is not adequate for the next decade:

• Mac desktop application user interface has progressed in the last years, influenced by smart phone apps and by macOS evolution. Customers deserve a high-quality, intuitive and beautiful user experience, and we are committed to make it happen.
• Progress in Movie Repair technology enables a profound redesign of interface. It’s not just about aesthetics, it will be about conceptual design, with new, intuitive and useful objects and features.

For the moment, we don’t have mock-ups of the “tok” Treasured to show you, but rest assured that this blog will be the first place where you’ll see them.
And you’ll love it!

Shortly after, a mathematician called Abraham Wald noted that the study only considered the aircraft that had survived their missions—the bombers that had been shot down were not present for the damage assessment. The holes in the returning aircraft, then, represented areas where a bomber could take damage and still return home safely.

Wald proposed that the Navy instead reinforce the areas where the returning aircraft were unscathed, since those were the areas that, if hit, would cause the plane to be lost.

And by following this advice, the Navy managed to minimize aircraft loss in the last years of World War II.

Focusing on “survivors” when analyzing some process leads to false conclusions, and I was about to make this exact mistake when I started writing this post.

A Nine Years Survival

A few weeks ago my business turned 9, and I asked myself what was the biggest mistake I had done during those years.

But when I started making a ranking of mistakes, I realized that this wouldn’t be very useful to publish such the list without first talking about the mistakes I didn’t make.

Just as an aircraft returning to base after a WW2 mission, my business had avoided the seminal mistakes that kill so many short-lived ventures.
Therefore, for a reader willing to take my advice, the pitfalls that I had managed to avoid would be the most important to consider.

I’m leaving this ranking of mistakes for my next post and I’m focusing in this post on things I got right.

1. Make Something People Want

The main reason I got a few things right when I created Treasured is that it wasn’t my first take at creating a product. In the previous years I had published a couple of “hobby products” and learnt a few lessons the hard way.

The first lesson is that without a painful, unsolved problem, that you can solve and charge money for, there is no future.

And that’s really counter-intuitive, because media coverage is always about twenty years old drop-outs that create an app that does a few trivial things, become an overnight planetary success, make no money but end up selling to Google or Facebook for a few hundred millions after a couple of years.

There’s no doubt that this is the story people wants to read, the success that startup founders want to experience. But for founders that’s like playing lottery and picking the winning ticket. For each such success they will be thousands of failures that nobody will talk about.

On the opposite, I prefer to bet on low risk ventures where my chances to win become acceptable. Granted, I won’t become super-rich, low risk brings lower reward. But the fact is that most innovative businesses that stick around for nine years have the same DNA: building a low key, niche, uncool product, but that solves a real, painful problem that people is willing to pay money for.

That’s true in the Internet age, and even now when free mobile apps are all the rage.

Now, how do you discover a painful, unsolved problem? Easy:
Just hang around in places where people with problems go, and ask them!

Forums in specialized websites are a perfect place to start hunting for a business idea.

One day in 2007 in a video forum, I stumbled upon a school teacher that needed to repair a surveillance video to figure out who had stolen the classroom computer. Luckily, the computer was recording live video to a hard disk in a locked case, that the thief had left behind. Classroom footage was available on the disk, but the MOV file been corrupted when the computer was taken by the thief.

That day, I repaired my first video (but unfortunately the thief was wearing a hoody and the teacher could not identify him) and decided that this could be a business idea worth investigating. The whole episode was captured in this blog a few days laters and is still a fresh read!

Once I knew that some people badly wanted to repair corrupt MOV files, the next step was to discover what the market would look like.
Through a market survey? No, by offering directly my expertise (which at this point was all but non-existent)

During a few weeks I requested corrupt MOV videos through my website, claiming that I would repair them for free.

I was contacted by around 20 persons, and discovered a surprising diversity of customers, situations, video formats and technical problems to solve. I managed to decently repair most of the videos, and the feedback was great.

This gave me confidence about feasibility and helped me take the seminal decisions about product and marketing, those that you can’t get wrong:

• Target Video Professionals, on Mac platform
• Deliver as a Service

Creating a “universal video repair program” was immediately ruled out: Diversity of situations, of formats, and quality required by professionals require specialized solutions and personalized support. Only a service can provide that.

The moment of truth is when I started to charge for the service.
Everyone is enthusiastic when you give away free repairs, but when there is money in the equation, everything changes.
You suddenly become liable. You quickly understand that unless you build trust and give solid proofs of repair, no customer will send you money.

Therefore, the service was shaped first and foremost to build confidence:

• Providing a diagnostics and price quote free of charge, using a sample of the damaged video provided by the customer
• Providing a short sample of repaired video
• By developing a customized Repair Kit that the customer can use on his Mac, to produce watermarked video out of the damaged MOV file and take the purchase decision
• Finally, give a refund guarantee in case expectations are not met

The first payment soon arrived. For sure, for the first months the service was still 100% craftsmanship, every customer being a new challenge, but the business was born. A minimal product (Treasured was still one year away) but a viable one.

To be continued…

Take the internal combustion engine used in automobiles.

For about a century the petrol and diesel motors in our cars have experienced incredible improvements, we can talk about 10x increase in energy efficiency, and don’t get me started with reliability, noise and air pollution.

Even in the last 15 years we have seen dramatic changes, like turbocharged diesel engines in sport cars, common rail direct fuel injection, with electronics and firmware keeping the fuel consumption always optimized.

Diminishing Returns

Yet it seems that reducing fuel consumption further, even a measly 5% or 10%, will require enormous efforts.

In high-end cars we start to see byzantine solutions, like reducing floor clearance at highway speeds to improve aerodynamics, closing partially the front grille to reduce drag while having enough engine cooling, “smart” management of pumps and air conditioning, and so on…

We are clearly in “diminishing returns” phase of the innovation cycle. Actually the engine system itself is already fully optimized; today fuel efficiency improvements come from the rest of car.

We are getting close to the hard limit. Thermodynamic cycle of combustion engines allow for a fuel efficiency of 25% to 30% at most. End of story.

Electric cars, on the other side, will soon be mainstream starting at 80% efficiency. (Producing electricity efficiently is another story, but let’s stay focused on our topic)

The State of Art for H.264 video in DSLR Cameras

I can almost sense that H.264 video encoding is at the same point in its technological cycle as combustion engine.

Today we are pulling almost 100% of its potential.

Just as combustion engine efficiency is limited by the laws of thermodynamics, H.264 encoding efficiency is limited by a standard approved in 2003. It has received amendments over the last decade, but the efficiency of the codec can’t improve significantly unless you redesign it.

First H.264 encoding chips didn’t have all the bells and whistles that Advanced Video Coding standard allows for.
For example, CABAC entropy encoding is about 10% more efficient than CAVLC, but makes chip design more complex.

This improvement in H.264 is the equivalent of common rail direct injection.

source: ephotozine – Joshua Waller

Then some manufacturers started to use different H.264 encoding settings depending on light, for example 3 possible settings for low light, interior and outdoor.

Canon introduced this optimization in cameras with DIGIC 5+ chip, starting with EOS 5D Mark III in 2012.

It consists in offering 3 different H.264 encoder configurations (called Picture Parameters Sets or PPS), each with a different value for a parameter called pic_init_qp_minus26.

The H.264 specification doesn’t shed too much of a light on what this parameters stands for:

pic_init_qp_minus26 specifies the initial value minus 26 of SliceQPY for each slice. The initial value is modified at the slice layer when a non-zero value of slice_qp_delta is decoded, and is modified further when a non-zero value of mb_qp_delta is decoded at the macroblock layer. The value of pic_init_qp_minus26 shall be in the range of -(26 + QpBdOffsetY ) to +25, inclusive.

In layman terms, this tells the encoder to encode with more detail the shades of color that dominate the picture, and thus avoid color banding: Image quality is slightly improved while keeping same bitrate.

This is like fine tuning the automatic transmission to optimize fuel efficiency.

Lumix GH4 seeking optimal H.264 video encoding

But there’s still a problem: at the time you start recording, the camera has to pick one of the possible H.264 encoding settings and stick with it until you push STOP. This is not optimal if you walk from interior to outdoor while camera is recording!

This is where Panasonic engineers made a smart contribution:
GH4 cameras also use several PPS, but they dynamically change the active PPS frame by frame to always use the settings that will yield optimal image quality.

Not only that: instead of playing with pic_init_qp_minus26, they use a more fine-grained method to get optimal quality called Picture Scaling Matrix.

Lumix GH4 engineers have used all the tricks in the H.264 encoding playbook to achieve optimal video quality.
There’s no much room for improvement for future H.264 cameras.

The Lumix GH4 is a serious contender to be remembered as a top DSLR camera of the H.264 era, just as the Leica M6 for rangeviewer film cameras.

Disruption Ahead: H.265

HEVC (H.265), the new standard for video encoding, promises a 50% economy in bitrate at constant quality over H.264.
Let’s see how the transition plays out in the next months. The whole industry has yet to start adopting the new standard.

If this is any indication of what is coming, last month we have received our first corrupt H.265 video from a Samsung Gear 360 camera.

Our tools are not yet ready to detect and repair H.265 routinely, but we already have some prototypes running in-house to address such repair requests and we will be ready for when the H.265 deluge starts.

For the moment, H.265 is only present in specialized markets (surveillance, IP cameras) and in new Samsung products (Gear 360, NX series).

You already know that XAVC has 3 variants: (bonus: wiki link for those who want a refresher)

XAVC-I is intra-frame, the highest quality variant
XAVC-L is long GOP variant
XAVC-S is consumer-oriented and will be the subject of next post: #3 Sony XAVC-S

But did you know that’s there a fourth variant and Sony is not even aware of it?

“XAVC Mark II” by Canon…

A few weeks ago, I have received a repair request for a format that Treasured could not detect.
So I parsed the damaged clip manually, and quickly convinced myself that it was a new kind of XAVC-I:
Same MXF container, same data layout, same H.264/MPEG-4 AVC video and Linear PCM audio channels.

And I also found an explanation for Treasured not detecting it as XAVC-I: H.264 encoding settings were slightly different than those I had trained Treasured for.

I thought it was just another flavor of XAVC-I, but found strange that after two years of dealing with XAVC-I, new flavors would still appear.
Nevermind, I did some tweaks to an existing XAVC-I repair kit, and I could repair the sample.

When I sent the XAVC-I preview to the customer, to my surprise he answered that preview was great, but that I got the format wrong: It was not XAVC-I, but rather Canon XF-AVC from the new C300 Mark II.

Rebuttal, the hard way.

What? I could hardly believe it, but evidence was damning:

1. Google: Canon XF-AVC is the new 4K format in town and everybody is talking about the Canon C300 Mk II. How I had missed that remains a mystery.
2. A quick MXFDump on the damaged file was saying it loud and clear:

[ K = MXFIdentification ( 0000000000000a10 )
  [ k = CompanyName
  3c.01, l =    10 (000a) ]
       0  00 43 00 41 00 4e 00 4f 00 4e                      .C.A.N.O.N
  [ k = ProductName
  3c.02, l =    64 (0040) ]
       0  00 45 00 4f 00 53 00 20 00 43 00 33 00 30 00 30    .E.O.S. .C.3.0.0
      10  00 20 00 4d 00 61 00 72 00 6b 00 20 00 49 00 49    . .M.a.r.k. .I.I

and the final nail in the coffin:

3. X-Rays on the first 20 MB shows clear differences (left is Sony XAVC-I, right Canon XF-AVC)

X-Rays is an internal tool using space-filling curves to visualize remarkable structures inside damaged video files.

So what is actually going on?
On one side, Canon XF-AVC uses that exact same ingredients as Sony XAVC for its new 4K format, on the other side there are small implementation differences that make the two formats distinguishable if you have the right tools.

Technical false twins, if you will.

The XAVC Marketing Stunt

Canon has pulled exactly the same marketing stunt as did Sony two years ago:
Rebrand H.264/AVC as Canon XF-AVC to address 4K with a “new” format.

And there’s a good reason Canon did the same: because it worked so well for Sony with XAVC!

The advantages of coining a nice format name are manifold:

• “New” is a magic word. Every marketer knows that.
• Under XAVC umbrella you can encompass a variety of specs that otherwise would create confusion:

XAVC-S for H.264 in MP4 container
XAVC-I for raw H.264 in MXF container, intra frame level 5.2
and future variants can also use the XAVC branding

• It is easier to communicate on XAVC compatibility, to certify or license XAVC third-party solutions, than to communicate on complicated H.264/AVC profiles and specifications.

The XAVC marketing trick has given a two years head start to Sony over Canon in the 4K transition.
In next post, we will explore a few interesting properties of XAVC and how Canon XF-AVC approach tends to differ.

Solid state drives (SSD) and high-speed connectivity through HD-SDI and Thunderbolt enable live recording of RAW and uncompressed feeds from 2K and 4K cameras. ProRes and DNxHD formats, previously only used in post-production, are now productions formats.

Camera <----------------> Aja Ki Pro Quad

Where we had a closed device, now we have an open system that is more versatile, but also more fragile.

Lessons of Redundancy

A few weeks ago, Mark contacted me to recover a 50 GB file containing footage from a concert.
The event was recorded in DNxHD format by an Atomos Ninja 2 recorder.

Mark is a seasoned professional, he always run two recorders for redundancy, prime and backup.
The prime recorder stopped and no one noticed until the end of the program.
This situation is not frequent, but it can happen, this is why a backup recorder must always be running when recording a live event.

Unfortunately, the backup recorder also had its share of problems, it crashed as well!

Redundancy Pitfall #1: Prime vs Backup

Mark had two recorders set-up, but had they the same chance of success?

Probably not. My guess is that the prime recording set-up was first rate whereas the backup set-up was a disaster waiting to happen.
For the prime recorder, Mark had used the gear in best condition, a very reliable recorder with encoding power in excess and fast storage.
For the backup set-up, Mark had used the old recorder, a bit underpowered for the job, and a memory card that had never given problems in the past, but that is 80% full (of footage from a previous job…)

Does it sound familiar? Mark had a redundant set-up, but the backup recorder had a failure rate probably around 20% instead of theorical 1%.
Unfortunately, this was one of those bad days for Mark: With the backup recorder struggling with encoding, some frames started dropping, and as the card was getting filled, fragmentation caused the write speed to plummet, finally causing the recorder to crash.

Therefore, a disaster like this is not due to a single failure, but to an accumulation of problems.
Redundancy is only safe if the two recorders are working in optimal conditions. Or to put it in another way: Your reputation as a video professional is in the hands of the less reliable of your two recorders, usually the backup system.

The mere labelling of the recorders as “prime” and “backup” is the first breach of redundancy. The two are equally important, and should rather be labelled “prime even” and “prime odd” and you would use the footage from the even recorder on even days, from the odd recorder on odd days.
That would protect you against the “broken backup” syndrome, more on that in pitfall #3.

Redundancy Pitfall #2: Independence

In statistics, one of the most important concept is variable independence:
An independent variable is a variable whose variation does not depend on that of another.

The theorical “1 out of 10000” rate is based on the assumption of independent failures.
It means that a common cause can never produce a failure in the two systems.

In the real world, this assumption is often false!
For example, if the recorders are connected to the same power source, or if they are writing to the same storage device, it’s clear that we don’t have redundancy, only the illusion of it.

So the solution is to have two identical systems (to avoid pitfall #1) but that are fully independent? Nope.

Identical systems are not independent due to latent flaws. That’s more subtle, but can be as devastating:
Imagine two identical recorders: same model, same firmware, same recording settings… If this model of recorder starts to fail when temperature is over 35 degrees, the two recorders will be at risk because they are exposed to the same temperature and workload. The second recorder will probably fail a few minutes after the first one.
Same for firmware bugs: if a recorder crashes under a certain condition, the other one will also probably crash.

In the Space Shuttle, there were 5 computers running in parallel, but one of them running software written independently. This was the safeguard against a bug affecting all computers.

Therefore, to build two truly independent systems is harder than it seems.
You only operate a redundant system when both prime and backup are equally reliable and truly independent.

But there’s a last problem…

Redundancy Pitfall #3: Supervision

Redundancy is its own enemy.
We are humans, and when we know that something is almost 100% safe, we tend to take success for granted and to stop worrying.

Lack of supervision is what can finally put your redundant system at risk.

If you watch your two recorders, you will immediately notice when one of them stops, and take action.
On the other extreme, if you never check the results of your backup recorder (because prime recorder works), you can overlook a systematic failure present in your backup procedure, and you are no longer protected. That’s the “broken backup” syndrome.

RAID 5 is a redundant storage system where 4 hard disks work together, so that if any of the 4 disks fails, it can be replaced by a new disk, and without stopping the system the remaining 3 healthy disks “rebuild” the content of the failed disk, and redundancy is restored after a few minutes.
In theory, RAID 5 systems are almost 100% reliable, because the data is only vulnerable during the few minutes where rebuilding takes place. The rest of the time, the system is redundant.

However, the #1 failure mode of RAID 5 systems is lack of supervision: Nobody detects that one of the disk has failed, and the RAID5 continues to work in degraded mode for weeks or months. But one day, a second disk starts having errors, and unfortunately it’s too late to rebuild the system, with two failed disks the system has become unrecoverable.

Therefore, please add this to your new year resolutions list:

I will verify my redundant recording procedure and I will:

1. Label my recorders “Even” and “Odd” and, based on the day of the month, use one or the other as “Prime”
2. Verify that my two systems are truly independent, that a single cause cannot produce a failure in the two systems
3. Watch my recorders and investigate any issue detected.

Happy new year 2016!