arthur richards

First of all, ***n3rd alert***. This post is technical and jargony - so unless you care about video codecs, video editing, storage, etc, you probably don't want to read this :) But if you know anyone who knows about these things, please pass this on!

I'm facing some technical challenges attempting to shoot a documentary (potentially hundreds of hours of footage) on a camera using the AVCHD codec. The primary issues are storage space and coming up with a decent workflow to handle footage in Final Cut Express 4.2. I've got some thoughts on the matter, but really could use some advice!

I'm relatively new to the world of digital video - especially high definition digital video. If you know me or have been following the blog, you know that my partner Sarah and I have embarked on a documentary and social media project on water scarcity in Guadalajara - Adapting to Scarcity. After a lot of research, we wound up buying the Canon HF-S100 camcorder. It shoots in Hi-Def onto an SD flash card using the AVCHD video codec.

This presents us with some interesting challenges. We've been planning on editing our footage in Final Cut Express 4.2 - which we understood work just fine with AVCHD. However, Final Cut Express does not natively work with AVCHD. Rather, upon import, it transcodes your video into AIC. This winds up increasing the video file size by about 3x. This is a mind boggling prospect when 3 hours of maximum-quality footage on our camera takes 32GB - after importing to Final Cut, the footage takes up over 90GB.

On the one hand, this is a bit of a blessing - apparently the transcoding takes some of the stress of dealing with AVCHD off of the computer's processor. However, it obviously puts a lot of stress on storage space. We're shooting footage in Guadalajara, Mexico - we've got limited gear and getting more is more expensive here (than in the US) and not particularly easy to do. Even if we ran out and bought hella hard drives, we'd need hella terrabytes worth to deal with the footage. Possible, but unpleasant. So what do we do, how do we deal?!

my thoughts

1) Make a compressed, archival copy of the SD card when it's full or otherwise needs to be erased and store it on some external hard drive(s). I like to make an exact copy as a disk image so it's easy to mount as it's own disk (emulating what happens when I stick the SD card in my laptop). This is easy in Linux or using the terminal in OS X:

For the uninitiated and curious, this uses 'dd' to make a copy of the device located /dev/sdb1. That copy is piped to GZip for compression. The '-9' means 'use the highest level of compression' (this is a bit slow, you can leave this out to use the slightly speedier default). Then, send the output to a file called '0001.img.gz' buried in some folders on my external hard drive.

2) Review the footage, and store detailed meta-information about the disk and its clips. I'm almost done putting together a handy media log using Drupal 6, Views and CCK. I've created some content types (eg 'disk' and 'clip') with some custom fields (with CCK) to get info like archived file path, tags, description, location, date shot, etc. Using views, I'm putting together some ways to flexibly and powerfully browse/search the stored clips, including a 'related clips' feature. More on this when I actually finish :)

3) When it comes time to actually edit raw footage, make a plan. Come up with at least vague ideas for the desired material.

4) Using the media log, search for pertinent/useful footage. When found, mount the archived disk image. Review actual footage and select portions of the footage that are actually desirable/usable.

5) Use log/transfer feature in Final Cut Express 4.2 to import desired footage. Store project/imported footage to a ph/fat array of large, fast hard drives (ideally connected to editing machine via eSATA).

When I've worked on video editing in the past, I've just logged/transferred all footage into Final Cut so I'd always have it at my disposal. Clearly, this is not a viable option in our current case. The 'flow I've outlined above seems to be our best option at the moment. Plus, it has the advantages of STARTING with backups of the footage as well as keeping detailed meta-information about it (I'm sure there are a bagillion different ways this will come in handy!), and then only transcoding footage that might actually be pertinent.

Is this the best way to handle all of this? If you know about this stuff and have any thoughts/suggestions/advice, I would L-O-V-E to hear it.