Ensuring trouble-free backups from your Mac to not-a-Mac

My current project is to enable network backups of my Mac and my wife’s PC over the internet, so that we have an off-site backup of last resort, should our house burn down, fall over, and then sink into the swamp.

Anyone who’s used a Mac for a long time knows that transferring Mac-native files over the internet is fraught with peril. You risk losing type and creator codes and resource forks, as well as a number of other forms of metadata introduced with MacOS X. So my first step was to determine how I could safely encode my files so that they could make the trip to a foreign server (which would either by a Linux-type box on my web host, or an Amazon S3 account) and then back again, with the file intact for recovery.

A few months ago, the Plasticsfuture blog ran an excellent article comparing the capabilities of darn near every backup and restore program on the Mac. The results were disheartening: Only SuperDuper! could precisely back up and restore a file (although, in my own testing, I found that ChronoSync was updated and can now handle the job).

However, my needs are somewhat different. A network backup requires that I only update changed files, and furthermore, I won’t be accessing them on a filesystem. So SuperDuper’s use of a disk image to perform network backups is straight out. So, too, is ChronoSync, since it can only back up to a network filesystem.

No, I need something that can encode individual files, ideally via a script or some other automated method. Furthermore, I’m mostly just concerned about archiving documents (I don’t want to pay for online storage of my whole hard drive!), so if permissions, ACLs (which I don’t use), or BSD flags are munged, that’s all right. This is truly a last-resort backup, so if I can get resource forks and extended attributes to back up, I’m pretty happy.

Now, according to Apple, a number of command-line utilities have been updated to deal correctly with extended attributes and resource forks – these include tar, cpio, ditto, and zip. There are also a handful of third-party archivers/compressors out there which also claim Mac compatibility such as the x7z (a Mac version of the 7-zip compression algorithm) and StuffIt compression programs, the xar archiver, and Interarchy’s “backup” format. (Note: Interarchy’s backup page is currently 404’ing, but suffice to say it’s an open format that attempts to encode files in such a way as to store all of Apple’s fancy metadata.)

So I figured I’d do what any red-blooded geek would do, and test all these programs to see if they did what they claimed to do. My test was pretty simple: I took a text file, assigned a Finder label and some comments, and added a resource fork and some custom extended attributes. If all these things made it through intact, I’d consider the tool good enough for my purposes.

The results were interesting. First off, every program I tested successfully maintained the resource fork, which is really the most important part of the file to keep around. Additionally, every program except for tar managed to keep the Finder label. As for extended attributes, only tar, the Interarchy backup format and cpio successfully kept the custom extended attributes. This is especially baffling given that the resource fork in MacOS X 10.4 is nothing more than an extended attribute!

The most troubling thing for me was that none of the programs I tested managed to maintain the “Spotlight Comments” I’d added. I frequently use these comments as a way to tag my files for Spotlight searching, so their loss is somewhat problematic. It turns out that these spotlight comments are stored in the invisible .DS_Store file in the same folder as the file I was backing up. So provided I restored (and backed up) the whole folder, that wouldn’t be a problem. Still, it would be nice to see it handle all that.

Update: I hadn’t originally tested it, since Apple hadn’t listed it on their OS X pages as a utility that had been updated to work with resource forks, but some online discussions led me to believe that the pax archiver had also been updated. Indeed, it has, and it successfully maintains resource forks, extended attributes, and Finder labels, just like cpio. It does, however, seem to have bugs when used on systems with ACLs enabled, and like everything else, it loses Finder comments. I have updated the discussion, below, accordingly.

So this leaves me with three solid options: The Interarchy Backup format, pax, and cpio archives. The latter two archive formats are fully compatible with other unix-like systems and can even be expanded using the graphical BOMArchiveHelper on the Mac, so they may be the best choice. Both archivers permit me to compress files in the archive, which is a nice bonus for network archiving. Of the two, pax has some nice advantages, including a larger file size limit on archives and some interesting command-line tricks including the ability to write out to different archive types. Cpio, on the other hand, seems to work on systems with ACLs enabled.

However, I suspect (although I haven’t tested this) that the Interarchy Backup format captures more Mac-specific metadata, since it was designed with exactly that goal in mind, and Apple’s programs are, well, less than entirely consistent in supporting these filesystem features. I do wish Apple could at least provide tools that were consistent and reliable.

It is worthwhile to note that none of the graphical Mac compression utilities managed to maintain extended attributes, not even Apple’s customer zip archiver (which is the same tool used when you archive files in the Finder) nor StuffIt, which has long been a Mac-friendly standby compression program. This leaves Mac users with essentially no easy options for compressing files prior to emailing them or posting them to an FTP site other than disk images (which aren’t cross-platform).

I’m disappointed that the xar archiver, which was designed precisely to handle metadata schemes on different systems, didn’t perform better. It is still a work in progress, so I left some bug reports, and I remain hopeful they will properly support extended attributes and comments in future releases. Since xar can also handle compression as well as encryption, it would be a fantastic solution for off-site backups.

Soon I’ll post how my backup system develops and works over the long-term. If you have any questions or comments, please feel free to sound off, below!

Written on March 13, 2007

Hardening your Mac against hackers

Cocoia Blog hosts a great two-parter on how to make your Mac virtually impregnable to the common thief or hacker. While a lot of these measures will seriously impair your ability to easily use your Mac (such as turning of Bonjour networking entirely), they will also make you quite a bit safer.

If you have sensitive documents on your computer, these are things you should, at the very least, consider implementing.

Cocoia Blog Howto: A more secure OS X before Leopard

Written on March 13, 2007

Cringley says hardware encoding/decoding of video is coming to a Mac near you!

Robert X. Cringely found himself a very interesting rumor:

Now comes the rumor I have heard, that I believe to be a fact, that has simply yet to be confirmed. I have heard that Apple plans to add hardware video decoding to ALL of its new computers beginning fairly soon, certainly this year.

…and it doesn’t just do hardware H.264 decoding, it does hardware H.264 ENCODING, too.

What does this mean? Well, it means that a Mac Mini will be able to play back video like a god (box), and encode it in H.264 with similar performance.

So when you sit down and decide to, say, rip your (legally purchased, fully owned and operated by yourself and none other) DVDs to your iPod or your Mac, you’ll be able to do so in, say, minutes rather than hours.

Already, there’s some USB video encoders out there which use a similar principle, so that when you compress video, it doesn’t tax your CPU at all. Media center type PCs have video encoding cards that do exactly the same thing. So none of this is new technology.

And, from an iTunes point of view, it could certainly save Apple money. They can use much more aggressive (and therefore CPU-taxing) compression on the HD videos they sell through iTunes, confident that downstream Macs have dedicated hardware to play that video back. Additionally, with Apple’s strong presence in video, this gives every Mac user the benefits of a dedicated encoding card (possibly not a great one, but far better than nothing), which will make iMovie much less painful and give Final Cut a similar boost. This does nothing less than allow users to take a video from soup to nuts, in HD resolution, on a MacBook Pro.

Written on March 9, 2007

Amazon Unbox on TiVo: If I Unbox my TiVo, can I still send it back?

The long-awaited partnership between TiVo and Amazon.com’s “Unboxed” video service finally launched, after weeks and weeks of promises.

I’ve been really excited about this service. Ever since my daughter was born, I’ve pretty much fallen off the planet, Hollywood-wise. So I’m just stoked at the possibility of just jumping online and renting a movie on a whim when the kiddo happens to go to sleep early.

Anyhow, here’s how Unbox + TiVo works: I can go to Amazon.com, browse their selection of downloadable and rentable movies, pick the one(s) I want, and they’ll automagically download to my TiVo. Slick! Not only that, but they gave me a $15 credit to play with the service just for signing up! How nice!

I think this is a great model. I can browse the store from a rich interface (my laptop, which has a keyboard and easy links to IMDB) and pick my movies, and can even do that when I’m not at my house. So I can pick a movie while I’m at work, and watch it at night once I get home. That convenience is well worth the premium price that Amazon charges for these rentals. ($2-$4 per rental, or $15 to buy a movie.)

Now, buying movies this way doesn’t appeal to me too much. There’s DRM, so I can ONLY play them on my TiVo or on a PC (Amazon’s video player isn’t Mac compatible), and that’s not good. But for a rental, I couldn’t care less if there’s DRM. I just want to watch it and then be done with it.

But here I am browsing their selection, and I can’t seem to find anything that I’d like to download.

First I looked for “Borat,” a movie I really want to see. They have it, but I can’t rent it. I don’t want to watch it for $15, so that’s out. I search for “The Departed,” and I find that I can buy it or… um… I can not rent it. What the hell?

But hey, there’s TV shows! So I search for my favorites: Scrubs? Nope. House? Nope. Hmm… What about “Heroes,” I’ve been curious about that show, it’d be great to watch a couple episodes with my $15 credit! Err… nope. Not there.

By now, I’m already turned off by this service. I normally consider Amazon an excellent retailer. I have a Prime account with them, I buy groceries from them, and do about all my X-Mas shopping on Amazon.com as well. I love ‘em. But this download store… not so much. Not only is the selection hit-or-miss, but they make it hard for me to do what I want. I mean, they have hundreds of videos, but they’re all different, and not just by pricing. Some are purchase only, some are rental only, and some don’t even offer a TiVo download, it seems! And then there’s the huge quantity of videos that aren’t available and don’t have prices but are still, inexplicably, clogging up my search results.

So instead of paying a premium price for a hassle-free download, I get tons of hassle and I get to pay a hefty premium.

Nope. I’m going to Buffalo Video and I’m going to slap down a cool dollar for a rental. Sure, it’s an extra drive, but the selection’s bigger and I know the movie will play on my DVD player.

Written on March 7, 2007

Scribd: YouTube for eBooks

I just learned about Scribd - Home, a new service that serves up user-generated eBooks and other text-filled content. You can read the shared documents online, or download them in Word, Text, PDF or even MP3 (using pretty decent text-to-speech software).

The bar to self publish just became quite low indeed.

Written on March 7, 2007

Sleazy

Apparently Best Buy has been running a shadow copy of their website so that people looking for online prices while shopping in the brick and mortar stores will be unable to find the lower online prices.

>…even when one informs a salesperson of the Internet price, customers have been shown the intranet site, which looks identical to the Internet site, but does not always show the lowest price.

From The Hartford Courant

Written on March 3, 2007

Drupal quick-post bookmarklet

Perhaps I’m being dense, but I’ve spent about four hours today trying to get a decent quick-post bookmarklet set up for Drupal. (And no, this isn’t a version 5 problem, I’ve never had such a thing.)

While I found many sites that had advice on crafting an appropriate URL, none of them mentioned that I needed the prepopulate module in order for any of them to work.

Additionally, only one person mentioned that I needed to put “edit[body_filter][body]” into the URL string rather than the more commonly suggested “edit[body]”.

So, in interests of saving people time, those are the two tricks to getting this to work. I’ve attached my javascript as a text file (it assumes you’re using Markdown for formatting, adjust as necessary for HTML) since it’s rather ugly to paste it inline.

I hope this helps somebody. I know I’ve wasted way too much time on it already.

Written on March 2, 2007