About

Welcome to Panela, Matt Harrison's take on mostly Open Source, Linux, Python, innovation in those areas, other buzzwords and Dick Proenneke. It comes complete with the illustrations as needed. Note the opinions expressed here are merely my opinions and not the opinions of my employer.

about Matt

Calendar

««Nov 2009»»
SMTWTFS
1234567
8910
11
12
1314
151617
18
192021
22232425262728
2930

Mailing List

My RSS Feeds








resume scp after interrupted downloads (use rsync)

posted 2007.01.24 Wed

If your download got cut off using scp, stop and read this before using scp again!

Telecommuting has it's perks. But one of the downsides can be the network issues. Especially if you have to download large amounts of data often. Having been bit by the interrupted downloads one too many times I found the following solution here.

The problem is that scp doesn't support resume, however rsync does. So create the following alias and you should be good to go:

alias scpresume="rsync --partial --progress --rsh=ssh"

Perks include understanding the same user@host:path syntax as scp as well as being able to resume a broken scp download (Note: I'm not guaranteeing this, but after downloading 90% of 400MB I was able to pick up the rest using this).

tags:        

links: digg this    del.icio.us    reddit




1. Doesnt matter left...
2007.01.31 Wed 3:02 pm

It doesn't seem to work everytime, after downloading 60% of 180MB, I used this command and had to restart the download at 0%. With another download it worked, dunno why...


2. eg left...
2007.06.14 Thu 8:17 am

You are my hero. This works so much better than other solutions I tried, for continuing the download on a 12 GB file.


3. jpatokal left...
2007.09.17 Mon 7:08 pm

Brilliant -- this just made downloading a 1.5 GB Oracle binary through a flaky connection oh so much easier. You, sir, are a genius.


4. nowhereman left...
2007.10.16 Tue 1:48 am

Hello,

This is what I was looking for; unfortunately it doesn't work if there is no rsync installed on the target machine, it seems... I passed OK through the tunnel on my machine but then it threw this error:

$ rsync --partial --progress --port=16703 --rsh=ssh nowhereman@localhost:/tmp/archive.zip /home/nowhereman/tmp/ Password: sh: rsync: not found. rsync: connection unexpectedly closed (0 bytes received so far) rsync error: error in rsync protocol data stream (code 12) at /home/lapo/packaging/tmp/rsync-2.6.9/io.c(453)

And on the target machine I got this:

nowhereman:/tmp# rsync sh: rsync: not found.

Unfortunately I am not authorized to install software.. So I'll have to try again with scp :-(


5. john left...
2008.01.21 Mon 11:26 am

To specify a port when using ssh you must add it to the ssh command. ex: rsync --partial --progress --rsh="ssh -p 16703" user@host:path

This should help nowhereman a little, but if you can't run rsync on the remote system it won't help in the long run.


6. Peter left...
2008.05.10 Sat 8:22 am

If you want something more lightweight than rsync, you can use cURL with resume (-C -) flag: curl -T filename -C - -u yourlogin sftp://example.net/~/filename


7. Ryan left...
2008.06.04 Wed 6:33 pm

Awesome. This is exactly what I needed. My ISP is being a bitch right now and closing my ssh connection every 5 minutes or so.


8. Cozzamara left...
2008.09.10 Wed 6:23 pm

Beautiful... you trick with rsync/scp made my day - I was just able to resume 2-gigabytes scp transfer that got disconnected


9. Jonathan Groß left...
2008.11.11 Tue 3:32 am

Thank you very much. Works great and saved my day.


10. Jean-Luc Cooke left...
2008.12.18 Thu 6:54 am

Sometimes this will not always work. So try these rsync options plus a bit a bash looping

while true; do

  • rsync -P --partial-dir=.partial --exclude=.* SRC DEST;

  • sleep 10;

done

-P is short for --partial and --progresss --partial-dir=.partial will put he partially downloaded file into directory called ".partial" for future downloading. It also tells rsync to look in this directory for a partially downloaded file to resume from. --exclude=.* tells rsync to stop any files for directories that start with a dot (eg .partial)


11. ischnura left...
2008.12.28 Sun 3:53 pm

Thanks for the info.

Still very useful nearly two years after the original post!


12. DarkFader left...
2009.04.23 Thu 1:55 am

Thanks for this post. I use --bwlimit=10 --rsh="ssh -p 443" options at work to get something from home without eating all upload bandwidth. Somehow scp also uses proxy settings from my browser.