Where is my footage? (part III)

Last week, I have explained how DeepMediaScan helps you visually map the footage on the disk and make repair operations much easier, faster and accurate.

But a few questions remain unanswered:

• How do you tell your repair kits what regions of the disk map should be repaired?
• What happens with disputed cells like A6?
• Which repair kit use?

DeepMediaScan “Clips”

Your Disk Map is a cell-based representation of the contents of the disk. Now we want to group cells together into “Clips”.

Looking at the Disk Map images, it’s easy to determine that biking footage occupies cells A1 to A6.
Similarly, birthday footage is in A7-A8 and B6-B7. As A6 and B5 are disputed, let’s include them also here: A5-A8 and B5-B7.
And we do the same for party and concert images:

Therefore, you just have to define 5 clips. For each clip, just select the images and drag them into Treasured.

Birthday footage is split over Clip2-A and Clip2-B, because the file was recorded fragmented. We cannot define it in one clip, because Treasured wants the images inside a clip to be consecutive. We will see later how this affects repair.

Repair Kits

Once clips are defined, we want to repair them.
As explained in part II, the clips have been recorded in 2 formats and need 2 different repair kits.

We know that Clip1 and Clip3 are in HD format, because the corresponding Disk Map cells have an HD resolution of 1280×720. The other clips are in SD, their display resolution in 640×480.

This means that we know exactly what kit to apply to what clip.

The diagram below shows that this time, we can recover all the footage.

• Once repaired with HD kit, Clip1 and Clip3 contain 100% of the original footage for biking and party.
• Once repaired, Clip2-A and Clip2-B contain 100% of the original footage for birthday. The two files can be merged together with a video editor.
• 100% or concert footage is recovered from Clip4 with SD kit. Note that it’s done in one shot, the “line-break” in the Disk Map does not divide it into two clips.

Conclusion

We have tools to recover every bit of footage present in the present, even if there is no visible file, or if files have funny names and end up not containing what is expected.

If DeepMediaScan doesn’t find it, it doesn’t exist.
If DeepMediaScan finds it, we can repair it.

Where is my footage? (part II)

Last week, I have explained why recovered video files are often not playable, and why even once repaired, they do not always contain all the footage that you expected.

This week, we will see how we can improve and try to salvage all the footage anyway.

SD and HD

Let’s go back to our disk “disaster”: We have used a data recovery utility and we have files but they do not contain what we expect:

On top of that, the files were recorded in two formats:

• Biking and Party in high definition (HD) and mono-aural audio,
• Birthday and Concert in standard definition (SD) with stereo audio.

And a different repair kit is needed for SD and for HD, but the real problem is that our recovered files can contain both formats mixed, like “Party”.

What will happen?

You will have to try each file with each kit:

• SD kit on an HD file will produce an empty file, or garbage video.
• HD kit on an SD file will do the same.
• On files with mixed footage, the kits will only repair the corresponding segments of the file. “Party” will give two repaired files, an HD file with some biking and an SD file with part of the birthday.
• “Birthday” file will give empty files with both kits
• None of the repaired files with contain an event end-to-end, due to recovered files not matching the media data. No party footage will be recovered at all (“Party” file contains biking and birthday)

Frustration

Bottom line: You will spend a lot of time repairing every file with every kit and you will end up with a mess:

• 80% of biking but in two separate files (one of them named Party-0.mov)
• Only 50% of the birthday recovered (in a file named Party-1.mov)
• No party footage found
• Only 95% of concert recovered. A couple of songs missing

Imagine if you have to repair a 2TB hard disk with 1000 files and 5 kits!

DeepMediaScan can do better than data recovery

Enter DeepMediaScan, a free feature inside Treasured.

The problem we have is to map correctly the data regions inside the disk, a task at which the data recovery utility failed.
DeepMediaScan helps you do that by creating a Disk Map.

Just like archeologists to put a grid on the ground area they are searching, DeepMediaScan puts a grid on your disk, and divides it into tens or hundreds or cells:

DeepMediaScan lets you specify the number of cells. Sometimes a fine-grained map is needed, in particular if disk is big and files are relatively small (needles in a hay stack).

But the idea is simple:
For every cell, DeepMediaScan will generate a preview picture, that tells you the format, resolution, and visually indicates what clip it belong (biking, partying, birthday and concert images are easily distinguishable).

The benefits are clear:

• Now you know where you footage is stored, with the resolution of a cell
• Thanks to format and resolution information, you know what kit is needed for a given cell or group of cells
• On big disks with hundreds of clips, you can focus just on the project that you need to recover, you don’t have to blindly try all files on all kits as above

All in all, with the visibility brought by DeepMediaScan, a big recovery becomes manageable.

What about A6?

Some cells, like A6 or B5, contain media from 2 different clips and formats.
In the Disk Map, however, you only get one preview image, corresponding to the left-most clip/format: A6 will show biking and HD format.

To minimize loss, you can either increase the scan resolution (smaller cells), or consider the A6 cell as disputed between biking and birthday (since A7 coming just after is a birthday image), and schedule A6 to be repaired in two different clips (biking and birthday).

In the third and last part of this story, I will explain how to do it and more…

Where is my footage? (part I)

My customers often get confused when the file they repair…

• doesn’t contain stuff corresponding to the file name or extension,
• contains footage from several clips,
• or contains large swaths without footage, or are completely empty

And in this case, your confusion is legitimate, because after a storage failure, we have in a twilight zone as far as data storage is concerned: Normal assumptions are no longer valid.

In this post, I will try to explain what is going on “behind the scenes” and how the repair results can be optimized.

Disk, Data and Files

Through this serie of posts, I will use the terms “disk”, “data” and “files” very often, so let’s start defining what I mean and how I will represent them in the diagrams:

disk stands for physical storage, it can be your computer hard disk, an external hard disk, an SD memory card, or and USB pen drive, for example. It is represented by an area whose surface is the capacity.

data is represented by a colored area. For example, green is used for the biking video. Note that the dark green region at the right side represents the index of the video, whereas the light green is video media. Both are necessary for the video to be playable.

file is represented by a boundary and a bubble with the file name.

In a normal situation, your disk contains several files and looks like the diagram below:

• We have recorded 3 video clips: Biking, Joke and Party
• File boundaries match exactly the corresponding data
• Disk is getting filled from left to right, in rows from top to bottom.

Joke, not so funny

Now imagine that we don’t find the joke very funny after all, and we delete the video file.

Note that only the file has disappeared, the data is still written on the disk, but unreferenced. This is why recovering deleted files is possible in the first place:

Later we record more clips called Birthday and Concert. Let’s observe the new diagram:

• Birthday file has overwritten Joke data
• Birthday is one file but occupies two regions: We call that fragmentation.
• Concert is stored in one region, because line-breaks do not count as boundary.

Bottom line: Unreferenced data will be overwritten when new files are added. So whenever you experience a storage failure or data loss, stop using the disk immediately. Reading doesn’t harm, but writing will end up wiping definitively your lost data. Now we can’t recover Joke because Birthday is written on top of it.

IMPORTANT: When repairing data or video from a disk, always operate from a different disk!
This avoids writing to the damaged disk.

Unplayable videos

To be playable, a video file must contain media data (video and audio), an index, and be coherent.

• Without media data, the file is not just unplayable, it’s not repairable either.
• A file with media but without index can be repaired.
• A file with media and incoherent index can be repaired, but is unplayable.

The most typical corrupt videos are those that you find after a recording failure. The media data is present, but unlike a well-formed video file, the index is not here:

This is because the index is written at the end of recording, and if we experience a failure (battery dead, camera overheating, …) this will never happen.
Note also that those unplayable files can also contain some empty space at the end, where data was about to be written when recording stopped.

Disaster strikes!

Now the dreaded storage failure suddenly happens. We no longer see any file in the disk (or we have files with funny names, or files that are now unplayable).

The first thing to do is to quarantine the disk, to avoid writing on it. Because the data is still here, waiting to be recovered, but it is also vulnerable since it is unreferenced.

At this point, many people use a free or commercial data recovery utility to “undelete” the files.
The result is usually disappointing, because video files are difficult to recover due to their diversity, complexity and huge size.
You often end up with a few good files, and plenty of bad ones: (see diagram below)

• File boundaries do not match exactly the data regions
• “Biking” contains only half of the footage and is unplayable because index is not present.
• “Party” now contains footage from Biking and Birthday, and is unplayable for being incoherent.
• “Concert” is unplayable, because the missing bit makes the whole file incoherent.
• “Birthday” is empty!

Note that I’m using quotes to differentiate the recovered files (“Party”) from the original ones (Party). Indeed data recovery often mixes file names and contents, so after recovery, file name should always be taken with a grain of salt.

File sizes, however, are always a good indicator of what you can expect. A 160 kb file is too small to contain footage. A 16 MB file can contain a few seconds (or can be empty). A 160 MB file can contain several minutes of footage, …

We can also find combinations of the examples above, like a recovered file with footage from 3 or more clips, a recovered file with large empty regions and footage, …

Repair will give limited results

Even if you invest time (and money) in repairing those files, you will probably not get a satisfactory result. As we have seen before, the recovered files do not contain all the missing footage, and can also contain footage from several clips:

• Once repaired, “Biking” will just contain half of the original video.
• Once repaired, “Party” will contain some biking and part of the birthday video!
• Once repaired, “Concert” will now have the first songs.
• “Birthday” will not repair, it contains nothing!
• Party cannot be repaired. Parts of other clips are still missing.

But there is hope…

To achieve optimal results, we would need to recover files that match better the data regions, and then repair them.
Fortunately, this is possible, using our advanced repair tools, and I will explain it in the second part of this post.

Unveiling our new logotype: “Grulli”

Aero Quartet is now in its teens, as far as we can use the age metaphor for a software company.
Childhood was the 4 years where Aero Quartet was a one-man-shop (I was the man!)

18 months ago I quit my day job at HP and went full-time, then a few months later I hired the first employee, then the second one, and so on.
As you can imagine, teen years are busy, fast-growth years, and our identity is building accordingly.

That’s why we have designed a new logotype:

This is a crane bird during migration. It let you imagine the company values that we want to convey through it.

The logotype will appear in products in website as soon as we find time to refresh them.