Following up on my earlier test of Mac archiving software, I decided to test some popular file copying/synchronization software to see which of these programs kept metadata and other Mac/HFS+ attributes intact. Rather than do a comprehensive test, I tried some popular utilities which seem to cover the general breadth of the software and which are particularly popular or prevalent. I also wanted to catch programs which had been updated since this article was written a year ago.

If you want detail on other utilities, I recommend reading the article linked above, or doing your own tests if you have the time. (And please let us know what you find out!)

The Utilities

The Finder: I just did a simple copy from one disk image to another. Drag and drop. For what it’s worth, an AppleScript copy via The Finder has the exact same results.

Disk Utility: Disk Utility does a pretty good job when making an image from a device, but I was more curious about how it would do making an image from a folder. So I simply chose “New Image From Folder…” and let ‘er rip.

cp: The venerable copy command on the command line. Apple’s updated it to respect resource forks and other metadata.

hard link: This is simply using the ln command or the Gnu cp command to create a separate instance in the logical directory of a file. This is a way to make a file literally exist in two places, where neither version is a pointer or alias to the other. This can be done with the ln command or the Gnu cp command (part of the gnu coreutils).

rsync: Rsync is a great and lightning fast synchronization utility that’s at the core of many command-line backup systems. Apple’s updated it to support resource forks and whatnot, but unfortunately, they broke it in the process. I ran a patched version installed via Fink as the other version pretty much doesn’t work at all.

ditto: This is Apple’s answer to rsync, as best as I can tell, and it’s a nice way to quickly duplicate folders and files. It’s been resource-fork-aware from the start.

psync: psync is a Perl-based utility that does file synchronization similar to rsync. A few graphical clients (such as Deja Vu) use psync on the back end.

CCC 3.0 b5: Carbon Copy Cloner is a great utility for cloning your hard drive. The 3.0 branch is brand new, so I thought I’d give it a try and see how it did.

SuperDuper!: Shirt-Pocket Software’s utility is a fave among many folks who like having a bootable backup. While somewhat inflexible in how it works (it’s pretty much geared toward being a whole-drive duplicator), it’s easy to use and very reliable.

ChronoSync: ChronoSync is probably the most configurable and powerful graphical synchronization utility available for the Mac. It’s users rave about it, so I figured I’d include it in my tests.

Result of tests of file copy/synchronization software

  Tool
  inode number
  permissions
  ACL
  BSD flags
  resource fork
  extended attributes
  type
  creator
  creation date
  modification date
  lock
  stationary
  invisible
  label
comments


  Finder
  copy
  N
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y


  Disk
  Utility Image
  N
  Y
  N
  N
  Y
  N
  Y
  Y
  N
  Y
  N
  Y
  Y
  Y
  Y


  cp -r
  N
  N
  N
  N
  Y
  Y
  Y
  Y
  N
  N
  N
  Y
  Y
  Y
  Y



  hard link
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  N
  Y
  Y
  Y
  N


  rsync -aE
  N
  Y
  N
  N
  Y
  Y
  Y
  Y
  N
  Y
  N
  Y
  Y
  Y
  Y


  ditto
  N
  Y
  N
  N
  Y
  N
  Y
  Y
  N
  Y
  N
  Y
  Y
  Y
  Y


  psync
  N
  N
  N
  Y
  Y
  N
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y


  CCC 3.0
  b5
  N
  Y
  N
  N
  Y
  N
  Y
  Y
  N
  Y
  N
  Y
  Y
  Y
  Y


  SuperDuper!
  N
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y


  ChronoSync
  N
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y
  Y

Notes:

• Finder Copies: Comments are maintained even when the invisible .DS_Store file is not copied, unlike most other techniques
• Disk Utility Image: Attempting to build an image from a folder on a volume with ACLs enabled will fail. As a result, these tests were carried out on a folder without ACLs.
• hard link: Files with the uchg flag/locked files could not be linked and were skipped
• rsync: Attributes in bold were only successfully copied from a volume with ACLs disabled. With ACLs enabled, these attributes do not copy.
• psync: Psync didn’t copy extended attributes, but it did embed the file’s type/creator information into a new extended attribute. Very strange behavior.
• SuperDuper!: SuperDuper! maintained the “arch” BSD flag on the clone. While this is not technically correct, it is thorough, and may be desirable if you want a precise clone.
• ChronoSync: ChronoSync has an option to maintain Finder comments even when you aren’t copying invisible .DS_Store files

Conclusions

Clearly, The Finder is your only free and bulletproof solution to copying files. Every command-line option, despite Apple’s efforts to make them compatible with all the fancy Mac metadata, has serious failings.

If you’re willing to spend a little money, SuperDuper!, or ChronoSync is a good option. Carbon Copy Cloner comes close, but even with its lower price (it’s donationware), SuperDuper! does a much better job and is quite reasonably priced.

For this very special Nik’s Picks, I’m letting someone else shoulder all the work.

John C. Welch, that juggernaut of Macdom, has compiled a list of his 22 can’t-live-without programs for the Mac. Lots of good tools in there for the casual user, the hacker/geek types, and the IT administrators alike. Go check it out!

My current project is to enable network backups of my Mac and my wife’s PC over the internet, so that we have an off-site backup of last resort, should our house burn down, fall over, and then sink into the swamp.

Anyone who’s used a Mac for a long time knows that transferring Mac-native files over the internet is fraught with peril. You risk losing type and creator codes and resource forks, as well as a number of other forms of metadata introduced with MacOS X. So my first step was to determine how I could safely encode my files so that they could make the trip to a foreign server (which would either by a Linux-type box on my web host, or an Amazon S3 account) and then back again, with the file intact for recovery.

A few months ago, the Plasticsfuture blog ran an excellent article comparing the capabilities of darn near every backup and restore program on the Mac. The results were disheartening: Only SuperDuper! could precisely back up and restore a file (although, in my own testing, I found that ChronoSync was updated and can now handle the job).

However, my needs are somewhat different. A network backup requires that I only update changed files, and furthermore, I won’t be accessing them on a filesystem. So SuperDuper’s use of a disk image to perform network backups is straight out. So, too, is ChronoSync, since it can only back up to a network filesystem.

No, I need something that can encode individual files, ideally via a script or some other automated method. Furthermore, I’m mostly just concerned about archiving documents (I don’t want to pay for online storage of my whole hard drive!), so if permissions, ACLs (which I don’t use), or BSD flags are munged, that’s all right. This is truly a last-resort backup, so if I can get resource forks and extended attributes to back up, I’m pretty happy.

Now, according to Apple, a number of command-line utilities have been updated to deal correctly with extended attributes and resource forks – these include tar, cpio, ditto, and zip. There are also a handful of third-party archivers/compressors out there which also claim Mac compatibility such as the x7z (a Mac version of the 7-zip compression algorithm) and StuffIt compression programs, the xar archiver, and Interarchy’s “backup” format. (Note: Interarchy’s backup page is currently 404’ing, but suffice to say it’s an open format that attempts to encode files in such a way as to store all of Apple’s fancy metadata.)

So I figured I’d do what any red-blooded geek would do, and test all these programs to see if they did what they claimed to do. My test was pretty simple: I took a text file, assigned a Finder label and some comments, and added a resource fork and some custom extended attributes. If all these things made it through intact, I’d consider the tool good enough for my purposes.

The results were interesting. First off, every program I tested successfully maintained the resource fork, which is really the most important part of the file to keep around. Additionally, every program except for tar managed to keep the Finder label. As for extended attributes, only tar, the Interarchy backup format and cpio successfully kept the custom extended attributes. This is especially baffling given that the resource fork in MacOS X 10.4 is nothing more than an extended attribute!

The most troubling thing for me was that none of the programs I tested managed to maintain the “Spotlight Comments” I’d added. I frequently use these comments as a way to tag my files for Spotlight searching, so their loss is somewhat problematic. It turns out that these spotlight comments are stored in the invisible .DS_Store file in the same folder as the file I was backing up. So provided I restored (and backed up) the whole folder, that wouldn’t be a problem. Still, it would be nice to see it handle all that.

Update: I hadn’t originally tested it, since Apple hadn’t listed it on their OS X pages as a utility that had been updated to work with resource forks, but some online discussions led me to believe that the pax archiver had also been updated. Indeed, it has, and it successfully maintains resource forks, extended attributes, and Finder labels, just like cpio. It does, however, seem to have bugs when used on systems with ACLs enabled, and like everything else, it loses Finder comments. I have updated the discussion, below, accordingly.

So this leaves me with three solid options: The Interarchy Backup format, pax, and cpio archives. The latter two archive formats are fully compatible with other unix-like systems and can even be expanded using the graphical BOMArchiveHelper on the Mac, so they may be the best choice. Both archivers permit me to compress files in the archive, which is a nice bonus for network archiving. Of the two, pax has some nice advantages, including a larger file size limit on archives and some interesting command-line tricks including the ability to write out to different archive types. Cpio, on the other hand, seems to work on systems with ACLs enabled.

However, I suspect (although I haven’t tested this) that the Interarchy Backup format captures more Mac-specific metadata, since it was designed with exactly that goal in mind, and Apple’s programs are, well, less than entirely consistent in supporting these filesystem features. I do wish Apple could at least provide tools that were consistent and reliable.

It is worthwhile to note that none of the graphical Mac compression utilities managed to maintain extended attributes, not even Apple’s customer zip archiver (which is the same tool used when you archive files in the Finder) nor StuffIt, which has long been a Mac-friendly standby compression program. This leaves Mac users with essentially no easy options for compressing files prior to emailing them or posting them to an FTP site other than disk images (which aren’t cross-platform).

I’m disappointed that the xar archiver, which was designed precisely to handle metadata schemes on different systems, didn’t perform better. It is still a work in progress, so I left some bug reports, and I remain hopeful they will properly support extended attributes and comments in future releases. Since xar can also handle compression as well as encryption, it would be a fantastic solution for off-site backups.

Soon I’ll post how my backup system develops and works over the long-term. If you have any questions or comments, please feel free to sound off, below!

Cocoia Blog hosts a great two-parter on how to make your Mac virtually impregnable to the common thief or hacker. While a lot of these measures will seriously impair your ability to easily use your Mac (such as turning of Bonjour networking entirely), they will also make you quite a bit safer.

If you have sensitive documents on your computer, these are things you should, at the very least, consider implementing.

Cocoia Blog Howto: A more secure OS X before Leopard

Robert X. Cringely found himself a very interesting rumor:

Now comes the rumor I have heard, that I believe to be a fact, that has simply yet to be confirmed. I have heard that Apple plans to add hardware video decoding to ALL of its new computers beginning fairly soon, certainly this year.

…and it doesn’t just do hardware H.264 decoding, it does hardware H.264 ENCODING, too.

What does this mean? Well, it means that a Mac Mini will be able to play back video like a god (box), and encode it in H.264 with similar performance.

So when you sit down and decide to, say, rip your (legally purchased, fully owned and operated by yourself and none other) DVDs to your iPod or your Mac, you’ll be able to do so in, say, minutes rather than hours.

Already, there’s some USB video encoders out there which use a similar principle, so that when you compress video, it doesn’t tax your CPU at all. Media center type PCs have video encoding cards that do exactly the same thing. So none of this is new technology.

And, from an iTunes point of view, it could certainly save Apple money. They can use much more aggressive (and therefore CPU-taxing) compression on the HD videos they sell through iTunes, confident that downstream Macs have dedicated hardware to play that video back. Additionally, with Apple’s strong presence in video, this gives every Mac user the benefits of a dedicated encoding card (possibly not a great one, but far better than nothing), which will make iMovie much less painful and give Final Cut a similar boost. This does nothing less than allow users to take a video from soup to nuts, in HD resolution, on a MacBook Pro.

The long-awaited partnership between TiVo and Amazon.com’s “Unboxed” video service finally launched, after weeks and weeks of promises.

I’ve been really excited about this service. Ever since my daughter was born, I’ve pretty much fallen off the planet, Hollywood-wise. So I’m just stoked at the possibility of just jumping online and renting a movie on a whim when the kiddo happens to go to sleep early.

Anyhow, here’s how Unbox + TiVo works: I can go to Amazon.com, browse their selection of downloadable and rentable movies, pick the one(s) I want, and they’ll automagically download to my TiVo. Slick! Not only that, but they gave me a $15 credit to play with the service just for signing up! How nice!

I think this is a great model. I can browse the store from a rich interface (my laptop, which has a keyboard and easy links to IMDB) and pick my movies, and can even do that when I’m not at my house. So I can pick a movie while I’m at work, and watch it at night once I get home. That convenience is well worth the premium price that Amazon charges for these rentals. ($2-$4 per rental, or $15 to buy a movie.)

Now, buying movies this way doesn’t appeal to me too much. There’s DRM, so I can ONLY play them on my TiVo or on a PC (Amazon’s video player isn’t Mac compatible), and that’s not good. But for a rental, I couldn’t care less if there’s DRM. I just want to watch it and then be done with it.

But here I am browsing their selection, and I can’t seem to find anything that I’d like to download.

First I looked for “Borat,” a movie I really want to see. They have it, but I can’t rent it. I don’t want to watch it for $15, so that’s out. I search for “The Departed,” and I find that I can buy it or… um… I can not rent it. What the hell?

But hey, there’s TV shows! So I search for my favorites: Scrubs? Nope. House? Nope. Hmm… What about “Heroes,” I’ve been curious about that show, it’d be great to watch a couple episodes with my $15 credit! Err… nope. Not there.

By now, I’m already turned off by this service. I normally consider Amazon an excellent retailer. I have a Prime account with them, I buy groceries from them, and do about all my X-Mas shopping on Amazon.com as well. I love ‘em. But this download store… not so much. Not only is the selection hit-or-miss, but they make it hard for me to do what I want. I mean, they have hundreds of videos, but they’re all different, and not just by pricing. Some are purchase only, some are rental only, and some don’t even offer a TiVo download, it seems! And then there’s the huge quantity of videos that aren’t available and don’t have prices but are still, inexplicably, clogging up my search results.

So instead of paying a premium price for a hassle-free download, I get tons of hassle and I get to pay a hefty premium.

Nope. I’m going to Buffalo Video and I’m going to slap down a cool dollar for a rental. Sure, it’s an extra drive, but the selection’s bigger and I know the movie will play on my DVD player.

I just learned about Scribd - Home, a new service that serves up user-generated eBooks and other text-filled content. You can read the shared documents online, or download them in Word, Text, PDF or even MP3 (using pretty decent text-to-speech software).

The bar to self publish just became quite low indeed.

Apparently Best Buy has been running a shadow copy of their website so that people looking for online prices while shopping in the brick and mortar stores will be unable to find the lower online prices.

>…even when one informs a salesperson of the Internet price, customers have been shown the intranet site, which looks identical to the Internet site, but does not always show the lowest price.

From The Hartford Courant

>Testing shows a “These Come From Trees” sticker on a paper towel dispenser reduces paper towel consumption by ~15%

Pretty impressive for a sticker. And the right way to do activism. Encourage good behavior in the Big Guys (business owners) by offering to save them a few hundred bucks worth of paper towels every year. Meantime you give the potentially moral schlub a little reminder to see if you can get some better behavior.

Perhaps I’m being dense, but I’ve spent about four hours today trying to get a decent quick-post bookmarklet set up for Drupal. (And no, this isn’t a version 5 problem, I’ve never had such a thing.)

While I found many sites that had advice on crafting an appropriate URL, none of them mentioned that I needed the prepopulate module in order for any of them to work.

Additionally, only one person mentioned that I needed to put “edit[body_filter][body]” into the URL string rather than the more commonly suggested “edit[body]”.

So, in interests of saving people time, those are the two tricks to getting this to work. I’ve attached my javascript as a text file (it assumes you’re using Markdown for formatting, adjust as necessary for HTML) since it’s rather ugly to paste it inline.

I hope this helps somebody. I know I’ve wasted way too much time on it already.