Any way to compare files from 2 directories?

Here to save you all
Donator V6.0
TheGameguru's picture
Location: Cinemaction!

Somewhere copying about 65GB of mp3's from an external HD to a NAS drive the copying died on some illegal file..

Now this is the second of 2 external drives filled with MP3's is there an easy way to see what files from the external made it to the NAS or do I just start over again?

Lots of duplicate artists folders exist.. just with different albums under them.

Pain in the ass

Aint nothing new about the world order..it's been playing since the day they put George Washington on a quarter

85's face the truth you're too dumb.

http://www.myspace.com/armyofthepharaohs

Off With My Head!
Donator V2.0
LiquidMantis's picture
Location: Rocky Mtn. Foothills

Same tree structure? I use SyncBackSE to compare and copy up to terabyte volumes. It's really cheap for home use to: $25 for a 5 computer license. The free demo will do what you need too.

Vice Versa is another one I've successfully used in production but SyncBackSE has more active development and the developers have an active forum.

Live: LiquidmantisGWJ | PSN: LiquidmantisGWJ
Web: Mantis on the Mountain
--
Women can't be in the same room with me without abandoning men forever - rabbit

Office Linebacker
Brannil's picture

Get a copy of windiff.exe. Open up a command prompt and navigate to the directory you have your collection in (ie. "My Music").
Type "dir *.* /s > drive1.txt"
Now type "dir X:\*.* /s >drive2.txt" (X being the external drive letter)
Then type "windiff drive1.txt drive2.txt"
That should bring up a directory structure list between both drives and compare them to each other. I'm sure someone else can explain the process better than I can though.

Coffee Grinder

Any diff tool will work (I use the one from http://unxutils.sourceforge.net/ since I'm on a Windows machine).

diff -qr dir1 dir2

should output something like:

Only in dir2: a.txt
Only in dir1: b.txt

Off With My Head!
Donator V2.0
LiquidMantis's picture
Location: Rocky Mtn. Foothills

Sure these diff tools will help but that's still a lot of hands on interaction and should it fail again then you have to start over. The sync tools I mentioned are all-in-one diff/copy (either using time & date stamp or CRC comparison) utilities with shadow copy support and are absolutely painless to use.

Live: LiquidmantisGWJ | PSN: LiquidmantisGWJ
Web: Mantis on the Mountain
--
Women can't be in the same room with me without abandoning men forever - rabbit

Pimpin' Ain't Eezy
Donator V5.0
Eezy_Bordone's picture
Location: Western Washington

I guess I'm confused, are you trying to consolidate the artist folders while copying? If not and you're just doing a straight up copy then read on.

I run two scripts, one uses Robocopy to do the actual copy. There are switches to copy just changed files after the original copy for use when doing a migration from one server to another and other neat stuff (like retry to copy failed files). I have it dump the results to a log file that looks like this.

I then run a script that uses Diruse.exe (a resource kit tool) that spits out a .csv with just summary info of the source and destination sizes of the directories I back up. Using the logs from the robocopy I can find any files that might be off.

Let me know if you want copies of the scripts.

Do you ever walk alone like a drifter in the dark?

Here to save you all
Donator V6.0
TheGameguru's picture
Location: Cinemaction!

Let me be a bit clearer..

I have 2 external Hard drives filled with MP3 (and FLAC's but thats another story) I'm trying to consolidate the 2 external drives into a single 1TB NAS drive.

I copied the contents of one drives MP3 folder which looks like this /artist name/album name/song title.mp3 into the NAS no problem..

The second external drive during the copy failed somewhere.. in my bleary pre Latte phase I didnt write down where it failed.

since there are many of the same artists on both drives its confusing to exactly figure out where it left off and failed.

Aint nothing new about the world order..it's been playing since the day they put George Washington on a quarter

85's face the truth you're too dumb.

http://www.myspace.com/armyofthepharaohs

Pimpin' Ain't Eezy
Donator V5.0
Eezy_Bordone's picture
Location: Western Washington

TheGameguru wrote:
Let me be a bit clearer..

I have 2 external Hard drives filled with MP3 (and FLAC's but thats another story) I'm trying to consolidate the 2 external drives into a single 1TB NAS drive.

I copied the contents of one drives MP3 folder which looks like this /artist name/album name/song title.mp3 into the NAS no problem..

The second external drive during the copy failed somewhere.. in my bleary pre Latte phase I didnt write down where it failed.

since there are many of the same artists on both drives its confusing to exactly figure out where it left off and failed.

In that case I would start a robocopy from the second drive to the nas.

I use these options with the xp/2k3 version (XP010) of robocopy:
/E - copies all sub directories, even if empty.
/NP - No progress indicator.
/FFT - Assumes FAT, helps with some file types on NTFS volumes.
/TEE - Displays to the output window and/or log.
/COPY:DATSOU - For NTFS volumes copies all the attributes, ACLs, auditing, ownership and fun stuff like that.
R:1 - Number of retries on failed copies, the default is 1 MILLION.
W:1 - Wait time in seconds to wait between retries, the default is 30 seconds.
/LOG+:\\server\share\file.log - The log location. I actually use a variable for the log location since it's dynamically named based on date/time.

If you log the copy process when done you can open it in your favorite editor (or notepad) and search for fail and see what files failed. The thing is the copy won't stop on a failure like with the 'normal' copy command so you're not wasting time restarting everytime you find an F'd up file.

Do you ever walk alone like a drifter in the dark?

Here to save you all
Donator V6.0
TheGameguru's picture
Location: Cinemaction!

doh.. I cant get by this error

Invalid Parameter #1 : "G:\The Associated Group MP3\MP3\*.*

Thats my source drive.. copying to a network mapped drive Y:

thats the correct path.. but for some reason Robocopy keeps barfing.

Aint nothing new about the world order..it's been playing since the day they put George Washington on a quarter

85's face the truth you're too dumb.

http://www.myspace.com/armyofthepharaohs

Royale With Cheese
Donator V5.0
Vega's picture
Location: In a mad, mad world

TheGameguru wrote:
Invalid Parameter #1 : "G:The Associated Group MP3MP3*.*

thats the correct path.. but for some reason Robocopy keeps barfing.

Do you have "'s around the entire path?

"G:\The Associated Group MP3\MP3\*.*"

They're usually required when you have folder names with spaces. Or you could just rename the folder without the spaces for the copy, then rename it back when it's done.

Mr T broke the speed of light in the A-Team van because he wanted to prove that quantum physics was a bunch of Jibba Jabba.

Here to save you all
Donator V6.0
TheGameguru's picture
Location: Cinemaction!

welp.. I tried g:\mp3\mp3\*.*

no " this time.. and I still get

Invalid Parameter #1 : "G:\MP3\MP3\*.*"

Aint nothing new about the world order..it's been playing since the day they put George Washington on a quarter

85's face the truth you're too dumb.

http://www.myspace.com/armyofthepharaohs

Royale With Cheese
Donator V5.0
Vega's picture
Location: In a mad, mad world

What's the full command line you're using?

Mr T broke the speed of light in the A-Team van because he wanted to prove that quantum physics was a bunch of Jibba Jabba.

Pimpin' Ain't Eezy
Donator V5.0
Eezy_Bordone's picture
Location: Western Washington

Leave off the *.*

Edit1 - Hang on, error reading my script give a few minutes to review...

Edit2 - Nope, I'm not insane. Leave the \*.* off your source and destination directories, by default it does *.*.

Do you ever walk alone like a drifter in the dark?

Pimpin' Ain't Eezy
Donator V5.0
Eezy_Bordone's picture
Location: Western Washington

Here is my script:

Quote:
:: Stan-BU.bat

@Echo off
@Title Robocopy Script

:: Set your Robocopy (2003) location here.
Set Roboloc=C:\WINDOWS\system32\Robocopy.exe

:: Set your log output folder location here.
Set OutputDir=D:\BU-Files\Logs

:: Set your share list location here.
Set ShareList=D:\BU-Files\Shares.txt

:: Set your source server location here.
Set SourceSrvr=stan\d$

:: Set your destination server location here.
Set DestSrvr=stan\f$\Backup

::*************************************************
:: See Robocopy code below to modify Robocopy options
::*************************************************

For /F "tokens=*" %%i in (%ShareList%) do (Set Share=%%i) & (Call :Checkem)

::Run ends here
@Title Robocopy Run Completed

Goto :EOF

:Checkem

@Title %Share% copy in progress

For /F "tokens=1,2,3 delims=: " %%i in ('Time /T') do Set Timev=%%i-%%jm

:: Configure log location
for /F "tokens=2,3,4 delims=/ " %%i in ("%Date%") do set LogName=%%k-%%i-%%j_%Timev%-%Share%-RoboCopyRun.txt

Set LogLoc=%OutputDir%\%LogName%

:: ******* Modify Robocopy options here **********************

"%ROBOLOC%" "\\%SourceSrvr%\%Share%" "\\%DestSrvr%\%Share%" /E /NP /FFT /TEE /COPY:DATSOU /PURGE /R:1 /W:1 /LOG+:"%LogLoc%"

Goto :EOF

My Shares.txt file just has 1 line per top level folder on the source drive that I want to back up, in my case:
Users
Shares

You can see what the command passed to the CMD Line is by changing the Robocopy line to this:

Quote:
ECHO "%ROBOLOC%" "\\%SourceSrvr%\%Share%" "\\%DestSrvr%\%Share%" /E /NP /FFT /TEE /COPY:DATSOU /PURGE /R:1 /W:1 /LOG+:"%LogLoc%"

And that will spit out something like this:

Quote:
"C:\WINDOWS\system32\Robocopy.exe" "\\stan\d$\Shares" "\\stan\f$\Backup\Shares" /E /NP /FFT /TEE /COPY:DATSOU /PURGE /R:1 /W:1 /LOG+:"D:\BU-Files\Logs\2006-08-07_05-49m-Shares-RoboCopyRun.txt"

Keep in mind these are single lines but have been wrapped by the page.

Do you ever walk alone like a drifter in the dark?

Here to save you all
Donator V6.0
TheGameguru's picture
Location: Cinemaction!

yeah it was the *.*

ok its moving since last night

Aint nothing new about the world order..it's been playing since the day they put George Washington on a quarter

85's face the truth you're too dumb.

http://www.myspace.com/armyofthepharaohs