I have a Western Digital 250Gb Passport drive for backing up my Windows laptop. I figured I would just use cygwin rsync to move data between the two drives. When I mount the Passport as drive Q, I use a command like this:

rsync -az --progress . /cygdrive/q/bak

This faithfully copies all the data, and avoids copying stuff that hasn't changed, mostly. But some files are copied every time I run the command, even when they haven't been changed, and in fact, a diff program shows that the source and the backup copy are identical. I figure it's some Windows characteristic of the file that is throwing off rsync and making it think the file needs to be copied, but I can't figure out what. Some of the files have capital letters in the filename, but not all of the afflicted files do. I never set permissions on files, so I don't think that could be it (but I could be wrong).

Anyone have any experience with this?

tagged: , » 18 reactions

Comments

[gravatar]
Rik Hemsley 8:11 AM on 9 Jan 2008

rsync isn't designed to understand Windows filesystems.

I think you might be better off with something like SyncToy.

[gravatar]
Nate 8:24 AM on 9 Jan 2008

Yeah, what he said. SyncToy:
http://www.microsoft.com/windowsxp/using/digitalphotography/prophoto/synctoy.mspx

[gravatar]
Chris 8:36 AM on 9 Jan 2008

You may want to try the --modify-window option to rsync. I was having the same problem backing some data up to a FAT drive until I used --modify-window.

[gravatar]
eric casteleijn 9:15 AM on 9 Jan 2008

The arguments I'm happy with for interaction with windows filesystems, is:

--size-only --delete --delete-excluded --exclude-from=exclude.txt -avz --no-group

although this grew over time, so some options might actually be redundant. I'd look at the man pages and search for no-group and --size-only particularly, to see if they are what you're looking for.

good luck,

eric

[gravatar]
Ned Batchelder 10:21 AM on 9 Jan 2008

I knew I'd get some good ideas! Of course, I neglected to bring the Passport to work with me so I could try them out right away. :(

The --modify-window suggestion led me to this page which has a number of suggestions that seem helpful.

[gravatar]
Matt 10:53 AM on 9 Jan 2008

On Windows I have had better luck with Unison (http://www.cis.upenn.edu/~bcpierce/unison). The aforementioned SyncToy also works well.

[gravatar]
susan senator 11:54 AM on 9 Jan 2008

I don't know about rsync, but I do know about nsync. Also, I do know alot about the kitchensink, and also, that syncing feeling. Glad to help anytime, just aks.

[gravatar]
Kelly 12:45 PM on 9 Jan 2008

Try an MD5SUM on both to make sure the files had not changed. Windows (Especially MS Office) does not change the datetime stamp and or archive bit when it changes the user who last printed a file.

[gravatar]
Arkady 3:01 PM on 9 Jan 2008

I had the same exasperating experience with rsync. And with many of the always-copied files being rather large RAW photos and such, I was driven to abandon it in favor of a more hacky homebrew Python concoction that compared a manifest of files from both locations to decide what to copy. I will try SyncToy next.

[gravatar]
ephemient 3:10 PM on 9 Jan 2008

IIRC, the problem is that the granularity on file timestamps in Windows is too low -- something like 2s. So --modify-window works by telling rsync, "times differing by ____ are equivalent".

[gravatar]
Joel Hockey 4:57 PM on 9 Jan 2008

I've had the same problem with rsync. I found the --size-only option fixed it. I also get strange windows permission problems from time to time on the backup drive. I fix them with a 'chmod -R 777 /cygdrive/f/'. It is a little extreme to set everything to 777, but for the purpose of my backup, it works OK.

[gravatar]
Colin Pretorius 5:17 PM on 9 Jan 2008

FAT32 has a 2 sec timestamp granularity, and --modify-window=1 solves that problem. That shouldn't be a issue if you're backing up FAT32 to FAT32 though.

The other problem is that FAT32 loses ownership permissons. The -a switch is equivalent to -rltpgoD, where the o and g tell rsync to check owners and groups. Try rsync -rltpD (that's what I use for backups on my FAT32 laptop).

Also, the z switch is most useful across the wire, not disk to disk, and is probably just wasting cycles if you're backing up a lot of compressed data and media (which is usually what's taking up the most space, anyway).

[gravatar]
Ned Batchelder 7:53 PM on 9 Jan 2008

You guys are the best! It turns out --modify-window=2 did the trick! Thanks for all the suggestions!

[gravatar]
Adam 10:30 PM on 12 Jan 2008

Perhaps a bit intensive but worth it for anything you really care about....

-c

Adam

[gravatar]
Marcelo Oliveira 4:16 PM on 22 Feb 2008

Tks,
these tips were very useful to me.

My 2 cents below:

I was trying to sync my homedir with a pendrive (fat32), kubuntu 7.10.

To solve the problem I have to:

1) manually remount the pendrive with option shortname=mixed (default is lower).
2) --modify-window=2
3) remove the --perms option, actually I'm using -rltODvz

[gravatar]
Ward Christensen 8:44 AM on 9 Nov 2010

Hey great blog, great tips!!

If I have timezone trouble, I'd use --modify-window=3600 or even 7200 I presume?

I use rsync between windows XP and a wonderful NAS box called a Netgear NV+ (now I think replaced by a ?? Pro or something like that). If you can afford close to $1K for your small biz or family backups, it is really really nice! Even has telnet shell option.

[gravatar]
Matt 6:02 AM on 10 Dec 2011

Reading through this old thread, im surprised no-one has mentioned robocopy, rsync does not make much sense in this instance http://en.wikipedia.org/wiki/Robocopy.

For windows rsync (over wan), i use yintersync http://www.yinter.net.

[gravatar]
Das.Flaigsi 2:11 PM on 23 Dec 2012

Hi, great comments. I had problems with permissions and so on. Repalcing -a with -rltD solved this. Additional I added --modify-window=2 to prevent further problems ;)

Thanks a lot.

Add a comment:

name
email
Ignore this:
not displayed and no spam.
Leave this empty:
www
not searched.
 
Name and either email or www are required.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.