This week, I’ve taken a very close look at how various Mac utilities deal with the plethora of metadata available in MacOS X. I took a look at file archiving and compression programs, backup and synchronization software, and also released a set of test files for other people to verify my own tests and run their own.

In addition to what I’ve published, I’ve corresponded with various closed and open source developers to learn more about what’s going on under the hood, and have come to a few conclusions which I think are worth sharing.

Transferring and backing up

As of the writing of this article, I am aware of only three utilities that will successfully maintain all your file’s information upon copy. The Finder, SuperDuper! and ChronoSync. No other utility that I’ve tested will maintain all your metadata, extended attributes, and resource forks when copying or transferring your files!

Furthermore, when it comes to compressing and archiving your documents on foreign file systems, your options become even more limited. Only one encoding format will keep your file intact when decoded, and that’s the Interarchy Backup Format. All other compression and archiving schemes, including many created specifically for the Mac, lose some measure of your files’ metadata.

The good news is that the Interarchy Backup Format is free and open-source. If you want your own copy (and don’t want to build it out of the source code), you can just download a demo of Interarchy, and you can find it in the application’s bundle in Interarchy/Contents/Resources/FileConverters/Backup and can move that wherever you like to use it for your other files.

I’d be very pleased to see some aspiring programmers give that format a little extra love and turn it into a more robust tool than it is today.

It is also worth noting that there are no real backup programs that can restore your files reliably and also track rolling archives, versions, etc. All we have are sync utilities.

Metadata is hard to keep track of, but vitally important to save

All of this metadata is very well hidden from the user. Aside from certain Finder flags, dates, and Spotlight Comments, the user can go about their day blithely unaware of the resource forks, BSD flags, extended attributes, and complex permission systems working under the surface.

However, when some of that data is lost, the results can range from highly confusing to catastrophic, especially for the active geek!

If you are accustomed to using smart folders, tagging your files with third party utilities, using the Spotlight Comments to further refine your searches and smart folder hierarchies, you will find that nearly everything you do to your files from the command-line will break your delicate system. Even less-geeky users will find thumbnail previews missing from some of their files if the resource fork is lost, or will be unable to find that article they wrote two years ago (because it now says it was created last week).

And third parties are already making heavy use of extended attributes to add an extra level of polish and ease of use to their applications. I’m sure we aren’t far off from seeing “real” information stored in those attributes, rather than just system-specific metadata.

Apple really screwed up, bad

The worst part is that Apple built an OS with this rich metadata layer, and they’re also responsible in large part for how fragile and therefore useless it is.

First off, there’s just too many ways to store this sort of information. Old-style resource forks, new style resource forks, invisible .DS_Store files, extended attributes, type/creator codes, and more. Apple really needs to pick one scheme for metadata and stick to it. (My choice would be Extended Attributes, as they’re by far the most flexible.)

Secondly, Apple needs to provide developers APIs to use to manipulate files that just plain work. As it is, not a single API nor command-line-utility that’s provided by Apple will successfully copy a file and keep its metadata intact. Apple brags that a whole host of command-line programs have been updated to support resource forks and other HFS+ goodies, but in truth, none of them work as well as a drag & drop copy in the Finder. And comments… don’t get me started! Apple hasn’t even provided an API to manage comments at all!

So those few developers who put in the effort to be respectful of our data and move/copy it with precision are forced to roll their own file copying algorithms. These algorithms are slower than a normal copy, and are necessarily idiosyncratic and have to be updated as Apple adds features and functionality to the operating system.

Conclusion

In the end, I’m putting this all on Apple’s doorstep (not that they’re listening to their customers on these points – rsync has been broken to the point of uselessness for the past 3 or 4 point updates of MacOS X). They need to set the standard and give us all the tools to reliably move our files and benefit from the functionality this metadata provides. It’s unreasonable to force users to choose among tools based on which one is most respectful of their files. This should be easy, and it isn’t.

Until then, I hope this guide is informative as you look to the right tools to do what you need.

Following up on my earlier test of Mac archiving software, I decided to test some popular file copying/synchronization software to see which of these programs kept metadata and other Mac/HFS+ attributes intact. Rather than do a comprehensive test, I tried some popular utilities which seem to cover the general breadth of the software and which are particularly popular or prevalent. I also wanted to catch programs which had been updated since this article was written a year ago.

If you want detail on other utilities, I recommend reading the article linked above, or doing your own tests if you have the time. (And please let us know what you find out!)

The Utilities

The Finder: I just did a simple copy from one disk image to another. Drag and drop. For what it’s worth, an AppleScript copy via The Finder has the exact same results.

Disk Utility: Disk Utility does a pretty good job when making an image from a device, but I was more curious about how it would do making an image from a folder. So I simply chose “New Image From Folder…” and let ‘er rip.

cp: The venerable copy command on the command line. Apple’s updated it to respect resource forks and other metadata.

hard link: This is simply using the ln command or the Gnu cp command to create a separate instance in the logical directory of a file. This is a way to make a file literally exist in two places, where neither version is a pointer or alias to the other. This can be done with the ln command or the Gnu cp command (part of the gnu coreutils).

rsync: Rsync is a great and lightning fast synchronization utility that’s at the core of many command-line backup systems. Apple’s updated it to support resource forks and whatnot, but unfortunately, they broke it in the process. I ran a patched version installed via Fink as the other version pretty much doesn’t work at all.

ditto: This is Apple’s answer to rsync, as best as I can tell, and it’s a nice way to quickly duplicate folders and files. It’s been resource-fork-aware from the start.

psync: psync is a Perl-based utility that does file synchronization similar to rsync. A few graphical clients (such as Deja Vu) use psync on the back end.

CCC 3.0 b5: Carbon Copy Cloner is a great utility for cloning your hard drive. The 3.0 branch is brand new, so I thought I’d give it a try and see how it did.

SuperDuper!: Shirt-Pocket Software’s utility is a fave among many folks who like having a bootable backup. While somewhat inflexible in how it works (it’s pretty much geared toward being a whole-drive duplicator), it’s easy to use and very reliable.

ChronoSync: ChronoSync is probably the most configurable and powerful graphical synchronization utility available for the Mac. It’s users rave about it, so I figured I’d include it in my tests.

Result of tests of file copy/synchronization software

  Tool
  inode number
  permissions
  ACL
  BSD flags
  resource fork
  extended attributes
  type
  creator
  creation date
  modification date
  lock
  stationary
  invisible
  label
comments


  Finder
  copy
  N
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y


  Disk
  Utility Image
  N
  Y
  N
  N
  Y
  N
  Y
  Y
  N
  Y
  N
  Y
  Y
  Y
  Y


  cp -r
  N
  N
  N
  N
  Y
  Y
  Y
  Y
  N
  N
  N
  Y
  Y
  Y
  Y



  hard link
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  N
  Y
  Y
  Y
  N


  rsync -aE
  N
  Y
  N
  N
  Y
  Y
  Y
  Y
  N
  Y
  N
  Y
  Y
  Y
  Y


  ditto
  N
  Y
  N
  N
  Y
  N
  Y
  Y
  N
  Y
  N
  Y
  Y
  Y
  Y


  psync
  N
  N
  N
  Y
  Y
  N
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y


  CCC 3.0
  b5
  N
  Y
  N
  N
  Y
  N
  Y
  Y
  N
  Y
  N
  Y
  Y
  Y
  Y


  SuperDuper!
  N
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y


  ChronoSync
  N
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y

Notes:

• Finder Copies: Comments are maintained even when the invisible .DS_Store file is not copied, unlike most other techniques
• Disk Utility Image: Attempting to build an image from a folder on a volume with ACLs enabled will fail. As a result, these tests were carried out on a folder without ACLs.
• hard link: Files with the uchg flag/locked files could not be linked and were skipped
• rsync: Attributes in bold were only successfully copied from a volume with ACLs disabled. With ACLs enabled, these attributes do not copy.
• psync: Psync didn’t copy extended attributes, but it did embed the file’s type/creator information into a new extended attribute. Very strange behavior.
• SuperDuper!: SuperDuper! maintained the “arch” BSD flag on the clone. While this is not technically correct, it is thorough, and may be desirable if you want a precise clone.
• ChronoSync: ChronoSync has an option to maintain Finder comments even when you aren’t copying invisible .DS_Store files

Conclusions

Clearly, The Finder is your only free and bulletproof solution to copying files. Every command-line option, despite Apple’s efforts to make them compatible with all the fancy Mac metadata, has serious failings.

If you’re willing to spend a little money, SuperDuper!, or ChronoSync is a good option. Carbon Copy Cloner comes close, but even with its lower price (it’s donationware), SuperDuper! does a much better job and is quite reasonably priced.

For this very special Nik’s Picks, I’m letting someone else shoulder all the work.

John C. Welch, that juggernaut of Macdom, has compiled a list of his 22 can’t-live-without programs for the Mac. Lots of good tools in there for the casual user, the hacker/geek types, and the IT administrators alike. Go check it out!

My current project is to enable network backups of my Mac and my wife’s PC over the internet, so that we have an off-site backup of last resort, should our house burn down, fall over, and then sink into the swamp.

Anyone who’s used a Mac for a long time knows that transferring Mac-native files over the internet is fraught with peril. You risk losing type and creator codes and resource forks, as well as a number of other forms of metadata introduced with MacOS X. So my first step was to determine how I could safely encode my files so that they could make the trip to a foreign server (which would either by a Linux-type box on my web host, or an Amazon S3 account) and then back again, with the file intact for recovery.

A few months ago, the Plasticsfuture blog ran an excellent article comparing the capabilities of darn near every backup and restore program on the Mac. The results were disheartening: Only SuperDuper! could precisely back up and restore a file (although, in my own testing, I found that ChronoSync was updated and can now handle the job).

However, my needs are somewhat different. A network backup requires that I only update changed files, and furthermore, I won’t be accessing them on a filesystem. So SuperDuper’s use of a disk image to perform network backups is straight out. So, too, is ChronoSync, since it can only back up to a network filesystem.

No, I need something that can encode individual files, ideally via a script or some other automated method. Furthermore, I’m mostly just concerned about archiving documents (I don’t want to pay for online storage of my whole hard drive!), so if permissions, ACLs (which I don’t use), or BSD flags are munged, that’s all right. This is truly a last-resort backup, so if I can get resource forks and extended attributes to back up, I’m pretty happy.

Now, according to Apple, a number of command-line utilities have been updated to deal correctly with extended attributes and resource forks – these include tar, cpio, ditto, and zip. There are also a handful of third-party archivers/compressors out there which also claim Mac compatibility such as the x7z (a Mac version of the 7-zip compression algorithm) and StuffIt compression programs, the xar archiver, and Interarchy’s “backup” format. (Note: Interarchy’s backup page is currently 404’ing, but suffice to say it’s an open format that attempts to encode files in such a way as to store all of Apple’s fancy metadata.)

So I figured I’d do what any red-blooded geek would do, and test all these programs to see if they did what they claimed to do. My test was pretty simple: I took a text file, assigned a Finder label and some comments, and added a resource fork and some custom extended attributes. If all these things made it through intact, I’d consider the tool good enough for my purposes.

The results were interesting. First off, every program I tested successfully maintained the resource fork, which is really the most important part of the file to keep around. Additionally, every program except for tar managed to keep the Finder label. As for extended attributes, only tar, the Interarchy backup format and cpio successfully kept the custom extended attributes. This is especially baffling given that the resource fork in MacOS X 10.4 is nothing more than an extended attribute!

The most troubling thing for me was that none of the programs I tested managed to maintain the “Spotlight Comments” I’d added. I frequently use these comments as a way to tag my files for Spotlight searching, so their loss is somewhat problematic. It turns out that these spotlight comments are stored in the invisible .DS_Store file in the same folder as the file I was backing up. So provided I restored (and backed up) the whole folder, that wouldn’t be a problem. Still, it would be nice to see it handle all that.

Update: I hadn’t originally tested it, since Apple hadn’t listed it on their OS X pages as a utility that had been updated to work with resource forks, but some online discussions led me to believe that the pax archiver had also been updated. Indeed, it has, and it successfully maintains resource forks, extended attributes, and Finder labels, just like cpio. It does, however, seem to have bugs when used on systems with ACLs enabled, and like everything else, it loses Finder comments. I have updated the discussion, below, accordingly.

So this leaves me with three solid options: The Interarchy Backup format, pax, and cpio archives. The latter two archive formats are fully compatible with other unix-like systems and can even be expanded using the graphical BOMArchiveHelper on the Mac, so they may be the best choice. Both archivers permit me to compress files in the archive, which is a nice bonus for network archiving. Of the two, pax has some nice advantages, including a larger file size limit on archives and some interesting command-line tricks including the ability to write out to different archive types. Cpio, on the other hand, seems to work on systems with ACLs enabled.

However, I suspect (although I haven’t tested this) that the Interarchy Backup format captures more Mac-specific metadata, since it was designed with exactly that goal in mind, and Apple’s programs are, well, less than entirely consistent in supporting these filesystem features. I do wish Apple could at least provide tools that were consistent and reliable.

It is worthwhile to note that none of the graphical Mac compression utilities managed to maintain extended attributes, not even Apple’s customer zip archiver (which is the same tool used when you archive files in the Finder) nor StuffIt, which has long been a Mac-friendly standby compression program. This leaves Mac users with essentially no easy options for compressing files prior to emailing them or posting them to an FTP site other than disk images (which aren’t cross-platform).

I’m disappointed that the xar archiver, which was designed precisely to handle metadata schemes on different systems, didn’t perform better. It is still a work in progress, so I left some bug reports, and I remain hopeful they will properly support extended attributes and comments in future releases. Since xar can also handle compression as well as encryption, it would be a fantastic solution for off-site backups.

Soon I’ll post how my backup system develops and works over the long-term. If you have any questions or comments, please feel free to sound off, below!

Cocoia Blog hosts a great two-parter on how to make your Mac virtually impregnable to the common thief or hacker. While a lot of these measures will seriously impair your ability to easily use your Mac (such as turning of Bonjour networking entirely), they will also make you quite a bit safer.

If you have sensitive documents on your computer, these are things you should, at the very least, consider implementing.

Cocoia Blog Howto: A more secure OS X before Leopard

Robert X. Cringely found himself a very interesting rumor:

Now comes the rumor I have heard, that I believe to be a fact, that has simply yet to be confirmed. I have heard that Apple plans to add hardware video decoding to ALL of its new computers beginning fairly soon, certainly this year.

…and it doesn’t just do hardware H.264 decoding, it does hardware H.264 ENCODING, too.

What does this mean? Well, it means that a Mac Mini will be able to play back video like a god (box), and encode it in H.264 with similar performance.

So when you sit down and decide to, say, rip your (legally purchased, fully owned and operated by yourself and none other) DVDs to your iPod or your Mac, you’ll be able to do so in, say, minutes rather than hours.

Already, there’s some USB video encoders out there which use a similar principle, so that when you compress video, it doesn’t tax your CPU at all. Media center type PCs have video encoding cards that do exactly the same thing. So none of this is new technology.

And, from an iTunes point of view, it could certainly save Apple money. They can use much more aggressive (and therefore CPU-taxing) compression on the HD videos they sell through iTunes, confident that downstream Macs have dedicated hardware to play that video back. Additionally, with Apple’s strong presence in video, this gives every Mac user the benefits of a dedicated encoding card (possibly not a great one, but far better than nothing), which will make iMovie much less painful and give Final Cut a similar boost. This does nothing less than allow users to take a video from soup to nuts, in HD resolution, on a MacBook Pro.

The long-awaited partnership between TiVo and Amazon.com’s “Unboxed” video service finally launched, after weeks and weeks of promises.

I’ve been really excited about this service. Ever since my daughter was born, I’ve pretty much fallen off the planet, Hollywood-wise. So I’m just stoked at the possibility of just jumping online and renting a movie on a whim when the kiddo happens to go to sleep early.

Anyhow, here’s how Unbox + TiVo works: I can go to Amazon.com, browse their selection of downloadable and rentable movies, pick the one(s) I want, and they’ll automagically download to my TiVo. Slick! Not only that, but they gave me a $15 credit to play with the service just for signing up! How nice!

I think this is a great model. I can browse the store from a rich interface (my laptop, which has a keyboard and easy links to IMDB) and pick my movies, and can even do that when I’m not at my house. So I can pick a movie while I’m at work, and watch it at night once I get home. That convenience is well worth the premium price that Amazon charges for these rentals. ($2-$4 per rental, or $15 to buy a movie.)

Now, buying movies this way doesn’t appeal to me too much. There’s DRM, so I can ONLY play them on my TiVo or on a PC (Amazon’s video player isn’t Mac compatible), and that’s not good. But for a rental, I couldn’t care less if there’s DRM. I just want to watch it and then be done with it.

But here I am browsing their selection, and I can’t seem to find anything that I’d like to download.

First I looked for “Borat,” a movie I really want to see. They have it, but I can’t rent it. I don’t want to watch it for $15, so that’s out. I search for “The Departed,” and I find that I can buy it or… um… I can not rent it. What the hell?

But hey, there’s TV shows! So I search for my favorites: Scrubs? Nope. House? Nope. Hmm… What about “Heroes,” I’ve been curious about that show, it’d be great to watch a couple episodes with my $15 credit! Err… nope. Not there.

By now, I’m already turned off by this service. I normally consider Amazon an excellent retailer. I have a Prime account with them, I buy groceries from them, and do about all my X-Mas shopping on Amazon.com as well. I love ‘em. But this download store… not so much. Not only is the selection hit-or-miss, but they make it hard for me to do what I want. I mean, they have hundreds of videos, but they’re all different, and not just by pricing. Some are purchase only, some are rental only, and some don’t even offer a TiVo download, it seems! And then there’s the huge quantity of videos that aren’t available and don’t have prices but are still, inexplicably, clogging up my search results.

So instead of paying a premium price for a hassle-free download, I get tons of hassle and I get to pay a hefty premium.

Nope. I’m going to Buffalo Video and I’m going to slap down a cool dollar for a rental. Sure, it’s an extra drive, but the selection’s bigger and I know the movie will play on my DVD player.

I just learned about Scribd - Home, a new service that serves up user-generated eBooks and other text-filled content. You can read the shared documents online, or download them in Word, Text, PDF or even MP3 (using pretty decent text-to-speech software).

The bar to self publish just became quite low indeed.

Apparently Best Buy has been running a shadow copy of their website so that people looking for online prices while shopping in the brick and mortar stores will be unable to find the lower online prices.

>…even when one informs a salesperson of the Internet price, customers have been shown the intranet site, which looks identical to the Internet site, but does not always show the lowest price.

From The Hartford Courant

>Testing shows a “These Come From Trees” sticker on a paper towel dispenser reduces paper towel consumption by ~15%

Pretty impressive for a sticker. And the right way to do activism. Encourage good behavior in the Big Guys (business owners) by offering to save them a few hundred bucks worth of paper towels every year. Meantime you give the potentially moral schlub a little reminder to see if you can get some better behavior.