Teach a man to fish…

(Third installment of my “Movie Repair Guide“, where you’ll learn how a movie can be extracted from a larger file)

When you repair movies, you never know what you can find.
Yesterday I had an old shoe moment while looking for some mountain footage in a corrupt file.
Some obscure AVI Windows file containing this infamous “Download” animation surfaced when I least expected it:

Today I’ll teach you to fish, in other words I’ll explain how to rescue movies from inside larger files. It will only work if certain conditions are met, but it’s an exciting experience… that can end with an old shoe. You’ve been warned.

Container Structure Correction is a repair technique that acts upon the container structure data, leaving the media data and the index and tables untouched.
Unlike reindexing or other techniques that need to act on hundreds of audio or video frames, a structure correction is usually a question of few bytes to correct or a misplaced block of data. For this reason, it can be done manually.

If the same correction has to be applied on a collection of files, then maybe it’s worth spending time automating the structure correction.

Most common structure damages and how to fix them:

Full movie data, including container, is embedded in a larger file, for example after a data recovery.
In this case, the data can be extracted and saved with the container suffix and it usually works.

In this example, we have examined the file with an Hex editor and have found an intact RIFF structure in the middle of a file. Note that the length of the structure 0x49e4 is encoded just after RIFF in 32 bits, big-endian.
A quick check shows that after 0x49e4+8 bytes, there is suddenly no data, thus confirming that we are into something.

By copying this data to a new file and saving with an AVI suffix, we now have a valid movie.
Make sure you copy exactly the required bytes, starting from the R in RIFF, ending after idx structure, otherwise it won't work.

When you edit a movie in-place (ie without writing again all the file), some edition softwares just add a new moov container at the end of the file, without bothering to remove older ones. If the file has become corrupt, those old containers can still work.
The trick consists in finding the moov structures and redirect towards an older one.
You'll need an Hex editor and good knowledge of QuickTime file format to do that.

Files from a data recovery can present defects: contents not matching the file name, truncated files, mashed contents coming from several files, moved or missing blocks.
As data recovery acts at the lower filesystem level, the data is always presented in blocks, for example exactly 2048 bytes.
A movie can fail to open because some blocks have been moved, duplicated or missing. The fact have blocks always have a length of 2048 bytes helps to detect and correct those problems. It's like solving a puzzle and is usually complex.

Indie fever: more about pursuing creative interests, than making money

Interesting research paper about the small Mac developers who form a community with its own role models and non-written rules. (via Gruber)

I feel identified with this description of independent Mac developers. Hopefully I can join this ‘creative class’ soon.

Indie Mac developers appear to be prototypes of what Richard Florida (2002) calls the ‘creative class’. On average they do not make a sharp distinction between their personal and professional lives (Florida, 2002, p.14), they appreciate systems of meritocracy and subscribe less to the status of wealth, they seem more motivated by the respect of their peers than by money (idem, p.78) and rally around their professional identity as a software entrepreneur (idem p.80). Indies fall into the category of ‘free agents’ (idem, p.107): people whom willingly sacrifice the securities of a job to be able to pursue their own creative interests.

But you can only pursue creative interests full-time once you’ve solved the money problem. Thus making remarkable products that sell is the most difficult part of the equation.

More icon work in progress…

A few days ago, as I was brainstorming ideas for my new icon design, several concepts emerged.

The “Chest”, my preferred one, was featured in the previous post. But a second idea also looked interesting and I decided to develop it as well:

A common problem I find when I write or design stuff, either an user interface, an illustration or an icon, is how difficult it is to judge your own work.

Usually, I let pass a few days before I look at it again. While you’re designing, you’re trying hard to make a certain message emerge out of the drawing, so your perception is biased until you’ve evacuated it completely. You can discover that the design doesn’t work or conveys a different message, but only after a while.

That’s why I’m leaving those two icons in this “almost-done” stage, until my mind cools down and I decide that I like them enough to finish the job.

New icon, work in progress

As it’s just for a secondary application, that plays a small role in a bigger thing, I haven’t engaged the services of a professional designer.

I’ve still some rough edges and details to polish, but I’m pretty happy with that. It took me about 10 hours of work:
Brainstorming, selecting ideas, developing a few ideas, handmade sketches, mock-ups to see what works and what doesn’t, pick an idea, and finally do detailled drawing.

A few lessons learnt:

• You will end with something different than what you plan.
• Design at once Application Name + Icon + “Catchphrase”. Otherwise it’s too heavy for only one element to convey all the force of your idea. Make each element reinforce and complete the other two.
• No more than 2 or 3 visual elements in your icon. Beyond that, it will not be understood and will not work at small sizes.
August 18th, 2008 in General | Comments Off on New icon, work in progress

“Zip test”

(Second instalment of “Movie Repair Guide“)

When confronted with a corrupt movie file, there’s a quick way to check that it still contains the media data, which is a necessary condition to repair it: the “Zip test”

We create a compressed version of the file, which tells us the compression rate. For example, 50% means that the zip file is only half the size of the original one.

Audio and video data will give a low compression rate whereas other type of data will be around 50% or higher.

In Mac OS X, you can use the zip utility, either from the Finder, or from the Terminal:

zip --filename--

In Windows, you can use WinZip utility.

Here are aproximate compression rates you get:

• For data completly erased (file full of zeroes), 90 to 100%
• For alien data, 25 to 100% (Office documents: 50%, Applications: 40%)
• For audio integer 16, 21%
• For AIFF, 17%
• For DVCHD Pro, 11%
• For DV, 5 to 12%
• For other video codecs, DivX with mp3, Intermediate, MPEG2, H264, JPEG: 0 to 8%

There is a big gap between audio/video data and any other type of data. If you have a corrupt file, and the “Zip test” gives you a result over 25%, probably a complete repair is impossible because the data is partially or completely gone.

You can also refine the test by measuring on segments inside the file:

With a 500mb file, we split in 5 segments of 100mb and run the zip test on each segment with this two Mac OS X Terminal commands:

split -b 100m corrupt.mov
zip deleteme xaa xab xac xad xae

(The first one, split, cuts the file in chunks of 100mb each, called xaa, xab, xac, xad and so on.)
(The second creates a compressed archive called deleteme from xaa, xab, xac, xad and xae.)

This gives for example:

xaa (deflated 5%)
xab (deflated 5%)
xac (deflated 12%)
xad (deflated 100%)
xae (deflated 100%)

Which tells us that the media data in last two segments is probably lost. If a repair is possible, maybe 50 to 60% of the original file will be restored.

Expectations

(First instalment of my “Movie Repair Guide” serie)

Not all repairs can be equally successful at restoring original quality and integrity of footage.

It’s due to three factors:

• a. initial state of data
• b. redundancy of data
• c. ability of the repairman

a. initial state of data
The worse case is a file that even if it keeps the correct movie name and size, no longer contains data related to the movie.
It can either be “alien data”, random data, and it can affect a part or the whole file.
It’s clear that no repair is possible here. I don’t do miracles and I will just tell you that it cannot be repaired (free of charge, of course!)

The best case is a file that contains all the media data, cleanly arranged in frames, but that fails to open because some small detail is not correct. If you correct this detail, the file is repaired at original quality. This detail is usually the structure of the container and the index of the media inside the file.

In the middle, you find files that still contain 50% to 99% of the media data, but it’s completely mixed and disorganized. Furthermore, the file can contain some quantity of alien data. Those files usually come from a failure in the file/operating system or in the storage device (Hard disk, memory cards, …). Even with the best repair skills at work, one cannot expect in this case to recover the entire movie at original quality. The cost of such a repair will be much higher than a “best case” repair, because the techniques involved are more complex and need customization to each case, and even if the results is far from “broadcast quality”.

Finally, initial state of data can be misjudged when the repair is developed on a sample of the file. The sample can be clean, but on average the whole file can contain several “hot spots”, pattern breaks and segments full of alien data. This can lead to an increase between quotation given after diagnostic and final cost.

b. Redundancy of data
A rule of the thumb is that the more advanced codecs you use, the more efficient compression you get. But at the same time, the repair becomes more difficult.
The reason is that efficient compression leaves few hints to do a repair, as efficiency means no information “wasted” to make encoding redundant and movie structure easy to read. The extreme case would be a codec so efficient that the bitstream looks like random data, and that any random bitstream is decoded as valid.
AAC audio codec, for example, is quite close to this situation.

Identifying an audio or video frame in such cases is really challenging, and a certain failure rate is to be expected. If you have 5% failure rate repairing AAC, the result is barely audible. At 0.5% you have something decent, but not broadcast quality.
On the other hand, in codecs like DV, the failure rate is 0 because more than 1% of the data is predictable and redundant, so mistake is impossible.

c. Ability of the repairman
An experienced video hacker with the right set of tools: That’s what I’m trying to become. If you are able to detect the subtle data patterns inside the damaged file, you have the key for a quality repair.

Below, you see the same movie frame repaired by an experienced hacker (on the left) and by an inspired hacker (on the right, also me ;). This was a very tough repair, because we were on the bad case for initial state of data. This explains why a traditional repair leaves a lot of artifacts on the picture. But after more work, I discovered a more advanced technique, called de-spotting, that eliminates almost all the defects by trial and error. It takes a lot of computing time for each frame, but it cleans almost everything.

I try to give the right expectations to my customers. Difficult repairs can be more expensive and yet achieve only bad video quality. That’s a tough business.

Movie Repair Service, Year Two

After one year of activity, and over one hundred movies repaired, I can start drawing the first conclusions about this incipient business.

I’m amazed by the diversity of movies that have revived through my hands:

A submarine hockey game, an award delivery in name of the Queen of England, 2 house robberies caught on a spy cam, a few indie concerts, a dozen family events, from births to … burial, a dentistry congress, a Chernobyl nuclear plant tour, funny classroom stuff, a few documentaries and even a feature-length movie.

Diversity also in terms of codecs and formats:

Video: DVCHD Pro, DV, JPEG, RPZA, MPEG-4 Video, H264, Sorenson 3, Uncompressed, HDV, Apple Intermediate codec, ProRes422, XDCAM EX Pro.
Audio: AAC, Integer of all sort, uLaw

My customers are in general video professionals, but there’s also a number of hobbyists that value their lost movies enough to afford a repair. The two main causes of corruption are: technical problems during recording, and storage failure (hard disk, memory card).

First of all, let’s take a look at it from the problem solving prospective.
It’s definitively a problem that is useful and interesting to solve, thus meeting my top goals as an entrepreneur.
But, it’s not an easy business: it’s time consuming, needs a lot of interaction with customers, and doesn’t make a lot of money at the end.
So is it time to give up and move on? Well, not so fast. As any incipient business, it has to go through several phases.

The first months, it started as an experiment. At that moment, I did not really care about making money (but charging for the service was a reality check that you must put in place soon), nor about being able to repair any movie a customer would throw at me. The goal was to check that movie repair delivered as a service was possible. In other words, that the mechanics of the repair that I had envisioned, from the first contact with the customer, the transfer of data, diagnostic, delivery of a “repair kit”, iterations to achieve full satisfaction, until the payment, was working. With the first dozen of happy customers behind, the goal was considered achieved.

Once you know it’s possible, you want it to be reproducible, reliable and predictable: How will I advertise my repair service if I don’t know what can or cannot be repaired, what level of quality to expect? That’s what the second phase was about, and it took me the rest of the first year to fully characterize this.
With over one hundred of cases solved, I’ve developed all the tools and techniques, classified the cases from easy to impossible, the quality of repair from rough to broadcast quality: At the end I was able to repair almost anything, and to predict the quality that I would achieve.
This second phase required some sustained effort, so I wanted it to be profitable. I adjusted the price points accordingly.

Now it’s time to enter into the third phase. It’s about scaling up: to reach more customers, and increase productivity. I have a few ideas about how to reach them (more about this in a future post). I will continue standardizing and automating the repair techniques to be more productive. Ideally, I would end up using fully automated repair tools that could even be deployed as web services. I think that it’s possible in a few happy cases, and that for the rest I will continue having a mix of manual and automated processes. More productivity means also profitable at cheaper price points, thus potentially reaching a lot more customers.

All this is really interesting for me. I have serious technical and marketing challenges ahead, and this business can succeed if I shape it as a small industry. Each industry has its own rules, that reflect the balance between what the customers want and are willing to pay, and what technology can do.

I will succeed if I can discover those rules. I will fail if I play against them.

You can be an Entrepreneur Hobbyist too!

Many people talk about creating their own business but few will ever do it.

Ironically, if you have the right preparation to create your own venture, for the same reason you are probably installed in a comfortable corporate environment, with a well paid job and good career outlook. Few people will take risks if they are not pushed to do so, whereas persons in a compromised situation (after getting fired for example), or people with nothing to lose, will realize greater things with less preparation.

But do you really have to leave your job to start a company?
No, it’s not necessary but it seems hard to run both concurrently. By doing this, you trade speed for safety, but that’s probably what you want when you start.

Enters the Entrepreneur Hobbyist: a person that has gone too far into a business to be called a hobbyist, but that has not yet become a full time entrepreneur. He doesn’t depend on the money to live, he can stop the experiment at any time, but time is with him to eventually grow the business to the critical mass and go full-time.

Nowadays, thanks to technology, there is a lot of activities that can be started with little or no investment and a reasonable workload. It’s possible, but it’s not easy: such a low barrier of entry will inevitably create a lot of competition, and being focused and smart will definitively help to get out of the pack. And build something that people will actually want to pay money for, starting from zero, is hard.

Fortunately, nowadays a business doesn’t need to be big to be profitable. Internet and search engines make millions of niche markets available, where you can “play small” and terms of risk while still making (little) money.
As a hobbyist, to become a millionaire is not a priority (not yet!) and a 5 digits business is enough to be entertained if it’s profitable. After a few years, you’ll probably want to “play bigger” and go full-time, and perhaps you’ll be in position to do it safely because you’ve laid the grounds for it.

What does a hobbyist company look like?
Take Aero Quartet, it’s three years old, has 3 products available, a few thousand customers, it is profitable, yet not big enough to sustain a family. It takes a few hours per day to run, and I run it as seriously as a normal job.
Aero Quartet is a one-guy company, I am the boss and also the lead developer, the accounting department, customer support, public relation, marketing manager and so on. I do everything, I take all credit for success and all blame for failure.

Hundreds of hours per year for a hobby, isn’t it too much?
Nobody questions the fact that we waste years of our lives watching TV. It’s common for people to spend 5 hours or more per day in front of TV, and it’s a socially accepted behavior. Is TV really worth this amount of our time? So why should I have to justify spending my spare time on a hobby? Even if as a business it never takes off.
(I have nothing against people watching TV, ok?)

Where does the motivation come from?
If it were for money, then probably I would be doing something else. I would be a broker, a lawyer or a notary.
But as a vocational engineer, what I really like is to build stuff. Excitement comes from creating something new out of nothing, from solving problems, from making a contribution.
I like to read stories about small things becoming important, about the power of individuals to change the world. Developing a start-up is your opportunity to do this. You will experience moments of doubts, despair, intense work, but also a big personal achievement when you get the damn thing to work, when you get the first sale, the first month with 4 digits revenues, the first press review. It’s probably the hardest thing that you will ever do in your profesionnal career, but also the most rewarding one.
If you’re good at building stuff and you like it, then you have the motivation. The rest can be learnt.

Why bother about selling? If it’s a hobby, why do you want to make money?
Indeed selling something immediately creates liabilities. The first dollar of revenue implies to set-up a legal business, providing support, paying taxes, and so on. But on the other side, even if it’s a small revenue, it will sustain the cost of building.
But first and foremost, charging for something raises the bar, and with real customers come real challenges. Poor ideas will never definitively fail if not through the filter of money.

Take my Movie Repair Service for example. While I was doing it for free, I repaired dozens of movies, and everybody was happy. I eventually learnt enough about repair techniques to start the business. And it suddenly becomes a lot harder. Just having the know how to repair movies is not enough. Now the determining factor is to reach the customers – Video professionals – that really see a value in this service, that are willing to pay $129 for it.

So you are confronted with a choice: Either kill it, make it a more appealing product, or keep it as it is.
You feel like an entrepreneur when you face those kind of decisions. And unless you charge for your products or services, you’ll never live the full experience.

My 4 advices for would-be Entrepreneur Hobbyists:

1. Don’t be obsessed with finding a good business idea. Just cultivate the habit of curiosity and of building stuff. The rest will come naturally.

2. Spending time in a product doesn’t make it better. Some products will never work, no matter how many hours you invest in improving them. Be courageous and kill them before it’s too late.

3. Do what your bigger competitors cannot do: Play smart and risky where your competitors cannot. Because you’re nothing, you can do anything. Not them.

4. Play safe: You don’t want to take legal or financial risks for a hobby, right?

And finally: If luck is preparation meeting opportunity, success for Entrepreneur Hobbyists is probably just a question of time.

Advice to prevent movie corruption

Over the last few months, I’ve been in contact with a few dozen people that submitted damaged movies for repair.

I can now share my experience (in fact their experiences, here I’m just an aggregator) and come up with a few recommendations to prevent movies from getting corrupt.
I’ve also recently started my Movie Repair Service, so you may say I’m giving advices that go against my business. But would you trust your doctor if he didn’t not recommend you healthy habits? Probably not.

First and foremost, back-up your movies when possible. This advice is good for any kind of data, but particularly for movies. Due to their huge size, movie files have a higher probability to get damaged, not to mention that long writing operations on an almost full hard disk are more disaster-prone.
Don’t trust storage of any kind. Hard disks die, memory cards are cheesy, USB sticks are fragile, and RAM is … volatile, to say the least.

Half of the repairs come from a storage problem.

Second, always check thoroughly the outcome of your work.
Imagine a simple workflow: Capture in DV, editing with iMovie, Export to iDVD, Authoring of DVD.
Rule of the thumb: Keep all files generated in earlier workflow steps until you have double checked the final product.

By checking I mean watching every second of the movie, on the device it’s intended to go (if it’s DVD, use your living room DVD player, not computer DVD software).
Sometimes a step fails silently and you don’t notice until it’s too late. For instance, it’s common to loose audio or to experiment video freeze several minutes into your DVD. Just because something went wrong during the conversion and iDVD didn’t show any alert message. If you have kept your files, you are safe and you can retry the failed operation. If not, everything is lost.

Third, be careful with live recording. Live recording, ranging from poor man’s webcam to high-end professional, can go wrong. One day it’s battery, the other day it’s software that hangs, or a cable that gets accidently unplugged, or hard disk choosing the worst possible moment to fill, whatever. You end up with a truncated file, with the index missing. Because index is stored in memory, waiting end of recording to be written, it is lost.
Unless someone comes up with a safe recording software (ie writing the index in real time) this will keep happening. This is a nice product idea, by the way.

Finally, the usual suspect: the Panasonic HVX-200 / Firestore combo. I have no statistical evidence here, but I can tell you that I’ve already repaired many movies recorded with this set-up. Take precautions, handle everything with care, or you will soon need my services.

From Idea to Product in 2 months

I’m not a marketing guy, and have never received related training. My background is engineering and science, so I have a strong bias and what I know about marketing techniques come from casual observation and corporate exposure.

What I’ll explain here is as much about marketing as about pursuing a business idea:
I’m exploring opening a movie repair service. Often something goes wrong with a video capture or a hard disk or a memory card, and you loose precious data. For a fixed fee, I would repair the data and retrieve as much footage as possible.

Businesses usually get born out of an idea. I came up with the movie repair idea because I know enough about video encoding and movie formats to be able to repair them by hand. So the idea itself comes from a technological “can do”.

Rule #1: It’s not because you can do it that they will need it.

It is important to do basic checks before building anything. Movie repair sounds like a great idea, like most ideas (in particular when it’s YOUR idea), but will people shell out one hundred bucks for it?

A Google search on “Movie Repair Service“, “Corrupt movie” or similar terms doesn’t bring significant results. You find people asking in forums if someone can help them repair their movies, or if some software actually does it. But nobody is advertising a solution (except data recovery, which is a different business).

With over one billion people recording video and a few hundred thousand people seeking business opportunities, it’s hard to think that I’m the first to have this idea. No contender on the market is definitively a bad sign.

Rule #2: First player on a market = void market?

Let’s find a few possible reasons why there’s nobody doing it:

• No practical way to do it at an interesting cost for customers.
• No guarantee of  success can be provided upfront.
• Every case is different, no standard method/process.
• People don’t want to pay before seeing the repaired movie.
• Data transfer of several gigabytes is not practical over the Internet.

But also a few possible reasons to be optimistic:

• Nobody bothers for such a small market (Great! As an entrepreneurial hobbyist, this is just what I’m looking for! Right Focus + Internet = You can reach any market, no matter how small, at virtually no cost)
• People have probably tried, but with the wrong approach (Software instead of Service, Wrong target, Wrong pricing, …)

You really need to spend a lot of effort to understand your potential customers. This is when marketing enters into the game.

Rule #3: Marketing = Help customers choose your product

As I said before, I’m not a marketing guy, but what guides me is thinking of marketing in terms of the service that I provide to my customers:
The first service I provide is pristine information. A good product, well presented, speaks to the customer. They immediately understand whether it fits their needs, how it works, and what they can expect. I don’t want to misspend their time with vague information, uncertainty or irrelevant “features”.

Marketing homework is well done when the product’s value is self-evident, and it makes sense to them. Doing this is very hard, because this goes against one’s intuition. You might think that more features means more sales, But instead, a simple pristine approach targeted to the specific needs of a customer is far more effective.

Therefore, I decided to go after potential customers and learn by providing a “free service” for a few weeks. With good Google ranking on relevant keywords, you will get a flow of spontaneous customers coming. For free.

In two months, I was contacted by around 20 persons, and discovered a surprising diversity:

First of all, many false positives: Movies that don’t need to be repaired. Either they don’t work because a codec is missing or they need some editing to remove bad frames. I also found movies that contain no data at all or full of zeros, and thus cannot be repaired.
Lesson learned: I need to detect those cases very early, maybe an automated test hosted on the Internet can perform this.

Then, there are two main movie corruption causes: Live capture problem and Storage problem (Hard disk, Memory card). Every repair is different and it’s very unlikely that an algorithm can be effective for every situation. Manual video hacking is required, thus pricing cannot be cheap.

Another important factor is volume of data. At over a few hundreds megabytes, data transfer over the internet becomes impractical; therefore I need to work on a sample of the whole movie. In this case, I will develop customized software that repairs the movie and then ask the customer to run it on their computer. I call it remote repair. And obviously it’s more expensive because several iterations may be required to get it to work.

Finally, I’ve identified three groups of customers: Hobbyists (Kids footage, home videos), Video Professionals (Producer, Technician), and Detectives (Yes, investigating trashed files).

After repairing their movies for free, I invited those customers to take a survey to provide me more specific information. This was time well spent, because I could take the defining decisions that will eventually, I hope, make this business successful.

Rule #4: Choose your battlefield
Rule #5: Reduce your target niche until you fully understand it

First decision: Only target Video Professionals.

It’s always counter-intuitive to limit oneself to a fraction of a market, but most of the time it’s the right decision.
Let me explain the rationale:

Hobbyists have a limited budget, but repairing their movies is not easier. Time invested in repairing a hobbyist’s movie will often have a higher cost than what their willing to pay. The volume is here, but not the sweet spot.

Video professionals will be more demanding in terms of quality of repair and deadline, but have a good reason to pay for your service: Their footage cannot be re-shot, and they need to deliver their product to their customers.

Detectives don’t need good quality (lost frames, frames with artifacts, and scratching audio is still ok), but they will only pay for results that support their investigation. Furthermore, I don’t really like the idea of having incriminating evidence on my hard disk, even less to send it over the Internet.

So it’s clear that Video Professionals are the customers I want to target first. They may represent only a few percent of total customers, but this is where my business is.

The advantage of a smaller target is homogeneity: These people know each other, they use the same tools, workflows and formats, have the same skills. This is what you want when marketing a service or product: Homogeneity. It will make everything easier, from communication to delivery.

Later, if you want to grow, you can always widen your target to adjacent segments. But not the other way around: If you start too big, your service will be a jack-of-all-trades that sucks for everybody.

Once you’ve decided, you must do a reality check. Advertising a free service can dangerously distort the feedback. This is why I applied rule #6 even before launching the service.

Rule #6: Start charging your customers as soon as you deliver a decent service.

Several customers have already paid and enjoyed the service. This is a good proof point and I can now continue working on this service with relative confidence.
I’ll launch it soon, and I’ll keep you posted.

Playing Detective: Repair a corrupt movie to identify a thief

UPDATE
I have started Movie Repair as a business in July 2007 and I no longer repair movies for free.

ORIGINAL POST
A few months ago, I read a forum post about a school teacher requesting help to identify the thief of a computer in the classroom during the night. It turns out that a webcam was hooked to the Mac mini, which recorded the theft to an external hard disk.
The Mac was stolen, but the hard disk, inside a locked enclosure, was left behind.
Unfortunately, as the recording was interrupted abruptly, the movie file was corrupt and could not be open.

I knew that in such cases, the video data was probably intact, but as the table of content and structure of movie was missing, it could not be accessed. At least not without some hacking…
I felt really excited by the prospect of repairing a file that would reveal a crime. I finally offered some assistance to the teacher. He sent me the file, and it took me a few hours to find out a way to repair it. First I managed to decode one frame. Then another.
It involved some byte hacking, using an hexeditor and writing a short program to extract some information. I also had to record a placeholder movie with exactly the same settings as the corrupt one. I transferred the video data to the placeholder, and, like magic, the movie was suddenly readable.

The movie shows how the teacher turns on the webcam before leaving, switches the lights off, then it’s completly dark in the classroom. The webcam software only records when there is some movement, so the next scene directly depicts the theft. You see a shadow approaching, holding a lamp. The lamp moves in front of the camera as the thief opens the cabinet and takes the computer, but unfortunately you never see his or her face, and suddenly the movie ends.

Too bad, the movie would not reveal the identity of the thief.
But I felt I had stumbled upon something. Every day, hundreds of corrupt QuickTime movies are “produced”. And I can probably repair most of them.

I want your corrupt Quicktime movies.
Stuff happens. After hours of rendering, the spinning ball of death appears and your favorite application goes west. You’ve just lost your work and you’ll miss that important deadline. If ever you could get it repaired!
As the process was interrupted abruptly, the half-baked QuickTime movie file cannot be open.
You know it contains the video and the audio, mostly intact, with maybe a few corrupt frames, but as the table of content and the structure of the movie is missing or uncomplete, there’s no way to repair it.

Or at least until today.
I’m trying to develop a software or a service, or a mix of both, that will repair QuickTime files.
At this stage, I’m just learning the most efficient techniques to repair QuickTime movies, that’s why I need to collect many corrupt movies. I will work for fun and for free. I don’t guarantee success either, sometimes it’s just not possible.

If you have such a file, please contact with me by email: info at aeroquartet.com

My software doesn’t get enough pirate’s attention.

Today I stumbled on an article by Daniel Jalkut about piracy. Daniel is also a Mac software developer.
He claims that piracy is not such a bad thing, because pirates will become our customers tomorrow.

A short summary of his article could be:
Granted, “Pirates are not paying our bills today”, but they contribute to word-of-mouth marketing, they provide some peer-to-peer support, and by choosing your product they help fighting your competitors. Eventually, they’ll “grow a moral backbone” and legitimately pay for your products.
He concludes that piracy should be ostracized as a bad social behavior, rather than fought like crime.

At first, I found his reasoning very interesting and mostly agreed with him. But when I tried to apply it to SimpleMovieX, my main product, I discovered a more complex reality.

Like every software worth two pennies, SimpleMovieX has been cracked for a long time. My protection scheme is extremely simple, and I guess that any cracker with the right tools can break it in five minutes. Making it more difficult would be a loss of time, it would be cracked anyway. So this protection scheme is just aimed at reminding users of demo version that it’s a 30$ product, that they can buy it, receive my eternal gratitude, and enjoy a world-class technical support.

So as I’m not fighting piracy seriously, does it make me a supporter of Daniel’s reasoning? Not so fast!

SimpleMovieX is not pirated enough to see the benefits that Daniel mentions. Word-of-mouth marketing, peer-to-peer support, fighting competitors, and finally getting future customers, all this depends on the pirates massively adopting your software.
You will say: Pirates don’t choose it because it’s a bad quality product with too few features. Wrong! They don’t choose it because it’s competing in a completly distorted “market”.

I’ll take an example: If you like cars, and you’re a thief, you’ll probably pick the most exclusive cars. A Porsche is not more difficult to steal as a Ford, but it’s much more rewarding, and the cost is the same: Nothing.
With software piracy, the same happens: If you can get cracked versions of Final Cut Pro, Adobe Premiere, and SimpleMovieX, which one would you use? High-end video editing or entry-level video editing? The fact that SimpleMovieX costs 30$ versus over 500$ for the other ones, doesn’t matter for the pirate crowd. Not to mention freeware products, iMovie, and QuickTime Pro with hundreds of registration keys “in the wild”.

But it’s not an excuse. I’m willing to fight for SimpleMovieX anyway. I’ll continue improving it, differentiating it, and I’ll try to turn this distorted market to my advantage.
It’s clear that a certain type of entry-level software, often high-quality products, developed by small developers, is getting hurt harder. If the positioning is: offer less features but be affordable, then you get little pirate’s attention.

Like it or not, freeware and piracy are two forces that are shaping the software market. They decide to what extent a product can be successful, or said in other words, they decide what products get developed.
I’ll learn the lesson for my next product, and one of design goals (not the first one!) will be to get maximum pirate’s attention.
A market shaped by free products and future customers cannot but be a passionate one!

There’s nothing like a hard deadline

Version 3.0 is finally here, today is a great day. But bigger than the D-day, is the “Deadline”, yet to come.

Let me explain what is the Deadline:
When I first started planning and developing this version, I was overly optimistic about the schedule. As always, the software is quickly 80% done, but the remaining 20% is what takes more time. Also, it’s hard to resist the temptation to add more and more features. What was to last 2 months stretched to 10 months.

Then arrived the September’06 breaking news: my wife was expecting our second kid.

Babies take 9 months, no matter what. The Deadline was set to April’07. Needless to say, in the last few weeks, I was under pressure. The baby could arrive at any moment, and the software was not ready.

Finally, Lady Nature was wise enough to let me finish my SimpleMovieX. The result is a fairly good 3.0 release, the most stable I’ve seen in years. Now I’m preparing for another great Day.
But that’s another story…

Google giveth, Google taketh away

Last week, suddenly my site disappeared from Google search results.

Nowadays, for a company like Aero Quartet, search engines can generate a big part of website traffic, and also be responsible for a similar chunk of the sales pie. So it’s a big loss that can even break a company if one gets too reliant on search engine traffic.

The same day, I read an article in reddit about the demise of a small software company due to sudden loss of search engine traffic.

I started to be really concerned after checking how my weekly visits from search engine had plummeted:

• First week of March: 224
• Third week of March: 239
• First week of April: 14

What could I have done to deserve the wrath of *Googlebot?

I did a round-up of related internet pages, read some FAQ pages from Google, and checked again my webpages. No clue.

As far as I remembered, I had just uploaded my redesigned website by mid-march and cancelled my Adwords account a couple of weeks ago.

But wait! Isn’t this just looking for problems?
What if Google had “punished” me for cancelling my account and drown my site into search results oblivion? I’m resisting to believe it, it would be “evil”, and violate Google’s sixth corporate commandment.

It’s most likely a side effect of uploading a whole new site, and I hope that after a transitory black-out, the flow of visitors coming from Google, Yahoo and MSN will recover to March levels. If within one or two weeks, it hasn’t happened, then I switch to DefCon 3.
With a Google account, you can use the Webmaster tools and consult some statistics about how Google is indexing your website. To my surprise, only a fraction of my current pages are here, and Google still has a lot of old, now defunct pages, listed. I’ll track it in the next days.

Another theory: Googlebot doesn’t see all my pages because I’m using redirects and invalid XHTML content. What made me think this is a visit of my site through a web spider’s eyes. I’ve used Poodle Predictor and it was not able to see most of my pages content. I quickly corrected all my pages by validating them against strict XHTML rules, and using 301 Permanent redirects instead of refresh redirects.
Now let’s see what happens in the next days.

UPDATE: It took several weeks to recover, now everything is back to normal. My guess is that the website redesign caused this problem.

* Googlebot is the name of the computer or group of computers at Google that every few days visit all websites in the world to collect information.

Making a new website

When, like me, you’re not a web or a graphic artist, and you redesign your website with higher standards, you always learn a lot and find that things are more complex that they seem.
You don’t redesign you site from scratch every day. I did it last week. It was long overdue, and the two years-old site design it replaces was not difficult to improve upon.
So let me explain you the process I’ve followed:
— Disclaimer: I’m not a professional, I’ve just tried to get as good a website a non-professional, self-taught, can —
When I started redesign, I had already for a few months taken bits of notes from pages, sites, or ideas collected during casual browsing. I also did some benchmarking by visiting and analyzing the design of half a dozen of sites related to my business.
You end up with a long list of items that you like and may want to copy. A short extract of mine:

• Clean layout, nice graphics, excellent presentation of products, this website is an example to follow.
• Create hi-res pictures of main features. If screenshots are used, give them life with perspective effect, …
• Present product by features but also through “success stories”. People identify better with a short story than a list of tech specs.
• Pages will be indexed by search engines and people will land on a page directly after a search. Make sure the content is adapted.
• Search button can easily be implemented by posting a form to a Google address.
• Nice layout to present a list of features

You’ll have to filter the list and retain maybe 1 out of 5 items because you cannot implement everything you like in just one design. That’s when you need to figure out the style, layout, structure and technologies.

Let’s go one by one:

Technologies were in my case a constraint. Remember that I’m not a web professional. I opted for the simplest possible: Static pages served by Apache server. No Flash, ActiveX or Javascript involved. Just plain HTML with CSS style sheets.

What I call the structure is a hand-drawn sketch where a put a box for every page and I draw lines to represent navigation between pages. I tried to keep the number of pages low while using one page per topic. What shall be avoided is a page that contains ten different topics, or a topic scattered in ten different pages. You want to make it readable and clear to the visitor, but also indexable by search-engines.

For layout, I picked a template from a web designer, many very good templates are available for free on the Internet. I wanted something simple and elegant, with several possible layouts (2 or 3 columns, feature box, …) to have some variety between pages.

Finally, style is the most difficult aspect. It’s a bit how your website smells and has to reflect the relationship you try to establish with your reader. If it’s plain, descriptive style, no emotion is transmitted, we are in pure technical communication. If you want to feel “cool”, you’ll probably get the opposite effect. I’ve tried to be personal and honest. I think it corresponds to the way I’m running the business and behaving with my customers.

So the result is certainly not perfect, but way better than before.

4 months of development, 8 months of polish

After more than one year of development, SimpleMovieX version 3 is finally ready. Software takes a lot of time to make, and if you want a good product, most of the time will be dedicated to what I call polish, or Mac polish to use carpeagua words. Take the Timeline for example. It’s a critical element of user interface that must give clear information that you can understand in a fraction of a second, and also a space for mouse interaction. And it has to be nice looking. You can’t come up with something radical or new, as the user has already worked with other similar applications. You have to ensure “backward compatibility” with what the users bring, what they take for granted.
So, you may ask, why do I persist in using a 2-knobs interface when QuickTime Pro and iMovie use a 3-knobs one?

This was one of the first design decision taken, two years ago, and maybe it was taken for a wrong reason: At that time, SimpleMovieX was not doing much, and I couldn’t just copy the QuickTime timeline without trying to “improve” it. So I decided to simplify it by dropping the Playhead knob. After all, when you are editing a movie, what is the point of playing it back if not locating one of the 2 knobs at the playhead? So let merge the 2 functions.
The other reason is that as a developer I use the keyboard a lot to test and use my software, so I tend to optimize the design for me. And driving the 2 knobs interface with the keyboard make SimpleMovieX fly. The power-user that invests a bit of time in learning the 2 knobs interface and the keyboard shortcuts gets a huge reward. But a good user interface is also learnable and with the lowest possible entry cost. I may have turned away a lot of people with my “improvement”.
Still, I can reconsider this choice in a future version of SimpleMovieX. If I switch now to a 3 knobs interface, I take the risk of infuriating the user base that is happy with the current one. I can always make it configurable, with a little switch in the Preferences, but nothing is free: I would probably end up with a fragmented user experience, some actions working better with 2 knobs, other with 3. Divide and conquer is rarely the best option when you’re short on resources like me.
What do you think about my choices?