integrating rsync with Rev

[Sarah Reichelt-2]

> Once again, you have enlightened the world with your brilliance and left a
> trail of happiness behind you :-)

LOL

> Thanks so much for your brilliant
> script!!! "Expect" was the key!
>
> I am going to try to convert this to a "send tCmd to me in 30 ticks" type
> callback...
>
> And, I don't mind checking for the string "total size", but I am also
> curious if there is an empirical way to find out when a process has
> terminated.

Maybe you could us "ps" to check to see if the process was still running.

Sarah
_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[david bovill]

Sarah - thanks for this work! Just one thing... for general use it would be
really great to use these techniques to interactively generate the ssh key
pairs from within Rev and then ftp or scp them to the server. This
convenience option would make it easy for users to set up secure
connections, and the truely paranoid could always do the ssh key pair
generation themselves on the command line.

I think this would get a lot more acceptance due to security fears - also it
has general application to a whole bunch of shell commands that can use ssh
key pairs for authentication - svn for one.

2008/5/20 Sarah Reichelt <sarah.reichelt@...>:

> > Once again, you have enlightened the world with your brilliance and left
> a
> > trail of happiness behind you :-)
>
> LOL
>
> > Thanks so much for your brilliant
> > script!!! "Expect" was the key!
> >
> > I am going to try to convert this to a "send tCmd to me in 30 ticks" type
> > callback...
> >
> > And, I don't mind checking for the string "total size", but I am also
> > curious if there is an empirical way to find out when a process has
> > terminated.
>
> Maybe you could us "ps" to check to see if the process was still running.
>
> Sarah
> _______________________________________________
> use-revolution mailing list
> use-revolution@...
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution
>

_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[Sarah Reichelt-2]

On Tue, May 20, 2008 at 4:15 PM, David Bovill <david@...> wrote:

> Sarah - thanks for this work! Just one thing... for general use it would be
> really great to use these techniques to interactively generate the ssh key
> pairs from within Rev and then ftp or scp them to the server. This
> convenience option would make it easy for users to set up secure
> connections, and the truely paranoid could always do the ssh key pair
> generation themselves on the command line.
>
> I think this would get a lot more acceptance due to security fears - also it
> has general application to a whole bunch of shell commands that can use ssh
> key pairs for authentication - svn for one.

If you can send me the Terminal commands needed to do this, I would be
happy to incorporate them into a stack and then make rync work that
way.

Cheers,
Sarah
_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[david bovill]

I've done this several times on linux / OSX - I'd have to look it up in the
man pages as once you've done it one you just use it. I've not used "expect"
before... I'll take a look at the way you've used it....

2008/5/20 Sarah Reichelt <sarah.reichelt@...>:

> On Tue, May 20, 2008 at 4:15 PM, David Bovill <david@...>
> wrote:
> > Sarah - thanks for this work! Just one thing... for general use it would
> be
> > really great to use these techniques to interactively generate the ssh
> key
> > pairs from within Rev and then ftp or scp them to the server. This
> > convenience option would make it easy for users to set up secure
> > connections, and the truely paranoid could always do the ssh key pair
> > generation themselves on the command line.
> >
> > I think this would get a lot more acceptance due to security fears - also
> it
> > has general application to a whole bunch of shell commands that can use
> ssh
> > key pairs for authentication - svn for one.
>
>
> If you can send me the Terminal commands needed to do this, I would be
> happy to incorporate them into a stack and then make rync work that
> way.
>
> Cheers,
> Sarah
> _______________________________________________
> use-revolution mailing list
> use-revolution@...
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution
>

_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[Josh Mellicker]

If you're setting up ssh key pairs, you won't need to use "expect"  
because no password is needed.

I started following this article:

http://www.sakana.fr/blog/2008/05/07/securing-automated-rsync-over-ssh/

but got stuck at "We can use the ssh-copy-id utility for this purpose"  
since I have no such utility. I was going to manually add the key pair  
to "~/.ssh/authorized_keys " on the remote host, but I'm not sure what  
"~" signifies on a Linux box. (on OS X it's the user directory)

But Sarah's solution seems to work great.

One caveat, with rsync, I have not figured out how to provide a  
progress slider!


On May 20, 2008, at 1:13 AM, David Bovill wrote:

> I've done this several times on linux / OSX - I'd have to look it up  
> in the
> man pages as once you've done it one you just use it. I've not used  
> "expect"
> before... I'll take a look at the way you've used it....
>
> 2008/5/20 Sarah Reichelt <sarah.reichelt@...>:
>
>> On Tue, May 20, 2008 at 4:15 PM, David Bovill <david@...
>> >
>> wrote:
>>> Sarah - thanks for this work! Just one thing... for general use it  
>>> would
>> be
>>> really great to use these techniques to interactively generate the  
>>> ssh
>> key
>>> pairs from within Rev and then ftp or scp them to the server. This
>>> convenience option would make it easy for users to set up secure
>>> connections, and the truely paranoid could always do the ssh key  
>>> pair
>>> generation themselves on the command line.
>>>
>>> I think this would get a lot more acceptance due to security fears  
>>> - also
>> it
>>> has general application to a whole bunch of shell commands that  
>>> can use
>> ssh
>>> key pairs for authentication - svn for one.
>>
>>
>> If you can send me the Terminal commands needed to do this, I would  
>> be
>> happy to incorporate them into a stack and then make rync work that
>> way.
>>
>> Cheers,
>> Sarah
>> _______________________________________________
>> use-revolution mailing list
>> use-revolution@...
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-revolution
>>
> _______________________________________________
> use-revolution mailing list
> use-revolution@...
> Please visit this url to subscribe, unsubscribe and manage your  
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution

_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[Josh Mellicker]

On May 20, 2008, at 10:40 AM, Josh Mellicker wrote:
>
> One caveat, with rsync, I have not figured out how to provide a  
> progress slider!

The answer:

--progress

_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[david bovill]

I have not come across "ssh-copy-id utility" - it looks like a linux only
utility which simply achieves the purpose of copying the public key to the
server. This is the same thing you would usually do using the secure copy
"scp" utlity which you will find on most *nix based machines including OSX
by default.

You have some interesting comments about limiting the ssh access to commands
etc - but I am not sure if this is regular security advice? My take on this
has been that you MUST have a secure local machine - and the security
critical part is generating the key and transferring it to the server
without someone gaining access at either of these stages. People go as far
as to generate the keys on special hardware or offline machines. Do you have
any references to the text on the page regarding limiting ssh access?

2008/5/20 Josh Mellicker <josh@...>:

> If you're setting up ssh key pairs, you won't need to use "expect" because
> no password is needed.
>
> I started following this article:
>
> http://www.sakana.fr/blog/2008/05/07/securing-automated-rsync-over-ssh/
>
> but got stuck at "We can use the ssh-copy-id utility for this purpose"
> since I have no such utility. I was going to manually add the key pair to
> "~/.ssh/authorized_keys " on the remote host, but I'm not sure what "~"
> signifies on a Linux box. (on OS X it's the user directory)
>
> But Sarah's solution seems to work great.

Yes - I'd still like to apply the same "expect" technique to ssh-keygen as
it has more general application to other command-line utilities and offers
increased security.


> One caveat, with rsync, I have not figured out how to provide a progress
> slider!
>

For that you need I think the 2.9 ability to write and read to the open
process - and then have rsync give some feedback with command line
options??? Not sure.
_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[david bovill]

Josh - I'd like to do some work on the rsync / expect thing this weekend -
so if you have a stack or any code you could post that would be great. I'll
try to extend it to work with svn as well. ````

This is the set-up and issues I've got. I am sure others have had the same
and it would be great to put together a few command line tools that can be
controlled and customized in Rev to help with the job.

   1. Got a new computer and I want to install Rev on it
   2. Got a new NAS that I want to move media files and shared software
   onto.
   3. Got several other computers on the LAN with different operating
   systems that I use to test and develop Rev based software
   4. Got around 30 libraries, assorted plugins and a few hundred archived
   projects and components built in Rev kept in local folders
   5. Got a large media archive on a USB drives (>500MB) that I want to move
   onto the NAS
   6. I have most of the Rev files and scripts in SVN - so I can keep them
   in sync between computers, and I have a Rev interface to SVN.

The problem I have had is that moving the files onto the NAS is slow and
some of the files have problems copying due to file names and other issues.
Failed copies leaves me in a situation where I am not sure all the files
have copied and which ones have not. I'd like to have a flexible set up so
that some of the files are backed up (mirrored locally) while other are just
shifted over now and the local copy deleted.

Solutions:

   - install rsync on the NAS and use rsync to copy over the files and sync
   any folders that want mirroring in the longer term
   - zip the archive into one large 500MB file and copy it over? Will this
   work or is it too large?
   - use some other ftp like software to reliably copy over the entire
   directory structure?
   - hand roll Rev based software to copy over the files and folders
   recovering from any errors
_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[Josh Mellicker]

On May 24, 2008, at 6:57 AM, David Bovill wrote:

> Josh - I'd like to do some work on the rsync / expect thing this  
> weekend -
> so if you have a stack or any code you could post that would be  
> great. I'll
> try to extend it to work with svn as well. ````

Hi David, sorry, I have been out and just saw this.

I have not done much at all after Sarah's "expect" breakthrough...

I will wire up a progress bar and a callback tomorrow and post the  
stack.
_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[Josh Mellicker]

Okay, here's where I'm stuck with using rsync and Rev for remote <->  
local file transfers:

The first time you run an rsync command with a certain server, you get  
this prompt from rsync:

The authenticity of host 'revcoders.org (67.19.54.130)' can't be  
established.
RSA key fingerprint is 9f:8c:ba:a9:5d:3f:b4:ef:f7:4a:2c:20:cd:77:b3:8c.
Are you sure you want to continue connecting (yes/no)? yes

Somehow, we either need to:

1. "expect" this possible response (the first time only) and answer  
"yes"

2. or, would it be a better solution to check for the key fingerprint  
on the local machine and create it if necessary? Or, perhaps just  
create it temporarily and delete it at the end of the session for  
security purposes?

Until I figure out how to properly configure the key pairs on local  
and remote machines I am stuck :-(




On May 26, 2008, at 8:40 PM, Josh Mellicker wrote:

>
> On May 24, 2008, at 6:57 AM, David Bovill wrote:
>
>> Josh - I'd like to do some work on the rsync / expect thing this  
>> weekend -
>> so if you have a stack or any code you could post that would be  
>> great. I'll
>> try to extend it to work with svn as well. ````
>
> Hi David, sorry, I have been out and just saw this.
>
> I have not done much at all after Sarah's "expect" breakthrough...
>
> I will wire up a progress bar and a callback tomorrow and post the  
> stack.
> _______________________________________________
> use-revolution mailing list
> use-revolution@...
> Please visit this url to subscribe, unsubscribe and manage your  
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution

_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[david bovill]

Thanks - Josh. From my limited research its not going to help with syncing
files to the NAS - without installing rsync on the NAS linux server
(unfortunately Buffallo Terastations are one of the few that don't have
this). Still not found a technique to synch 2 folders on different drives?
I'm sure backup software does this - and I'm surprised that there is not a
command line tool for it.

2008/5/27 Josh Mellicker <josh@...>:

>
> On May 24, 2008, at 6:57 AM, David Bovill wrote:
>
>  Josh - I'd like to do some work on the rsync / expect thing this weekend -
>> so if you have a stack or any code you could post that would be great.
>> I'll
>> try to extend it to work with svn as well. ````
>>
>
> Hi David, sorry, I have been out and just saw this.
>
> I have not done much at all after Sarah's "expect" breakthrough...
>
> I will wire up a progress bar and a callback tomorrow and post the stack.
>
> _______________________________________________
> use-revolution mailing list
> use-revolution@...
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution
>

_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[david bovill]

I think you need to expect it and answer "yes"
_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[Sarah Reichelt-2]

On Wed, May 28, 2008 at 2:19 AM, Josh Mellicker <josh@...> wrote:

> Okay, here's where I'm stuck with using rsync and Rev for remote <-> local
> file transfers:
>
> The first time you run an rsync command with a certain server, you get this
> prompt from rsync:
>
> The authenticity of host 'revcoders.org (67.19.54.130)' can't be
> established.
> RSA key fingerprint is 9f:8c:ba:a9:5d:3f:b4:ef:f7:4a:2c:20:cd:77:b3:8c.
> Are you sure you want to continue connecting (yes/no)? yes
>
> Somehow, we either need to:
>
> 1. "expect" this possible response (the first time only) and answer "yes"

This script works for that, whether the fingerprint has already been
stored or not:

#!/usr/bin/expect -f
spawn rsync -avzrt
revcoder@...:/home/revcoder/public_html/revcoder_rsync_test/
/revcoder_rsync_test
expect "connecting" { send "yes\n"}
expect "password:" { send "cookies\n"}
expect "#"

BTW, when testing this on my own site, I found that the "expect"
parameters are case sensitive. My web host sends "Password:" with
upper case "P" and I had to change the script to match.

>
> 2. or, would it be a better solution to check for the key fingerprint on the
> local machine and create it if necessary? Or, perhaps just create it
> temporarily and delete it at the end of the session for security purposes?

I found that the fingerprints are stored in ~/.ssh/known_hosts but it
is encoded. You can detect if the fingerprint already exists and I
guess you could delete it after every transaction.

Cheers,
Sarah
_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[Josh Mellicker]

Just an update... I am stuck on this, because "expect" somehow  
interferes with the downloading of larger files.

Simply running rsync from Terminal works great. (I downloaded 800MB of  
large files at blazing speed with nary a hiccup)

But running a shell script:

#!/usr/bin/expect -f
spawn rsync -avzrt
revcoder@...:/home/revcoder/public_html/
revcoder_rsync_test/ /revcoder_rsync_test
expect "connecting" { send "yes\n"}
expect "password:" { send "cookies\n"}
expect "#"

from Terminal, with:

./rsync.sh

downloads 1 or 2 files, then stalls. This is without Rev involved in  
any way, so this eliminates Rev from the equation, and also makes this  
post [OT]

Also, since I was reminded rsync is not on Windows machines, this  
demotes rsync as a good file transfer solution for Rev, since it's Mac-
only.

Back to libURL!



On May 27, 2008, at 3:49 PM, Sarah Reichelt wrote:

> On Wed, May 28, 2008 at 2:19 AM, Josh Mellicker  
> <josh@...> wrote:
>> Okay, here's where I'm stuck with using rsync and Rev for remote <-
>> > local
>> file transfers:
>>
>> The first time you run an rsync command with a certain server, you  
>> get this
>> prompt from rsync:
>>
>> The authenticity of host 'revcoders.org (67.19.54.130)' can't be
>> established.
>> RSA key fingerprint is 9f:8c:ba:a9:5d:3f:b4:ef:f7:4a:2c:20:cd:
>> 77:b3:8c.
>> Are you sure you want to continue connecting (yes/no)? yes
>>
>> Somehow, we either need to:
>>
>> 1. "expect" this possible response (the first time only) and answer  
>> "yes"
>
> This script works for that, whether the fingerprint has already been
> stored or not:
>
> #!/usr/bin/expect -f
> spawn rsync -avzrt
> revcoder@...:/home/revcoder/public_html/revcoder_rsync_test/
> /revcoder_rsync_test
> expect "connecting" { send "yes\n"}
> expect "password:" { send "cookies\n"}
> expect "#"
>
> BTW, when testing this on my own site, I found that the "expect"
> parameters are case sensitive. My web host sends "Password:" with
> upper case "P" and I had to change the script to match.
>
>>
>> 2. or, would it be a better solution to check for the key  
>> fingerprint on the
>> local machine and create it if necessary? Or, perhaps just create it
>> temporarily and delete it at the end of the session for security  
>> purposes?
>
> I found that the fingerprints are stored in ~/.ssh/known_hosts but it
> is encoded. You can detect if the fingerprint already exists and I
> guess you could delete it after every transaction.
>
> Cheers,
> Sarah
> _______________________________________________
> use-revolution mailing list
> use-revolution@...
> Please visit this url to subscribe, unsubscribe and manage your  
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution

_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[Thierry-24]

Le 30 mai 08 à 01:26, Josh Mellicker a écrit :

> <.....>
>
> Also, since I was reminded rsync is not on Windows machines, this  
> demotes rsync as a good file transfer solution for Rev, since it's  
> Mac-only.



Hi,

Mmmm, would say that rsync is a Linux tool first, then Mac and
with some struggles for the installation, works on Windows too.

Regards,
Thierry

_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[david bovill]

Done a bit more research - rsync is available for Windows - but AFAIK it
requires cygwin, whcih does mean that to distribute with Rev is not as
simple as including a binary.

I've been looking at alternatives - the one I have used before on Linux and
Windows, OSX with Rev is Unison - binaries are available. It's basically
rsync, but lets you do it between two computers with an internet connection:

   - http://alan.petitepomme.net/projets/unison/index.html

I also found this programme which is Java - and can be run from commanline -
so can be used by Rev. It looks like it does not require installation on the
sserver and can do incremental backups to a NAS:

   - http://jfilesync.sourceforge.net/index.shtml


2008/5/30 Josh Mellicker <josh@...>:

> Just an update... I am stuck on this, because "expect" somehow interferes
> with the downloading of larger files.
>
> Simply running rsync from Terminal works great. (I downloaded 800MB of
> large files at blazing speed with nary a hiccup)
>
> But running a shell script:
>
> #!/usr/bin/expect -f
> spawn rsync -avzrt
> revcoder@...:/home/revcoder/public_html/revcoder_rsync_test/
> /revcoder_rsync_test
> expect "connecting" { send "yes\n"}
> expect "password:" { send "cookies\n"}
> expect "#"
>
> from Terminal, with:
>
> ./rsync.sh
>
> downloads 1 or 2 files, then stalls. This is without Rev involved in any
> way, so this eliminates Rev from the equation, and also makes this post [OT]
>
> Also, since I was reminded rsync is not on Windows machines, this demotes
> rsync as a good file transfer solution for Rev, since it's Mac-only.
>
> Back to libURL!
>
>
>
>
> On May 27, 2008, at 3:49 PM, Sarah Reichelt wrote:
>
>  On Wed, May 28, 2008 at 2:19 AM, Josh Mellicker <josh@...>
>> wrote:
>>
>>> Okay, here's where I'm stuck with using rsync and Rev for remote <->
>>> local
>>> file transfers:
>>>
>>> The first time you run an rsync command with a certain server, you get
>>> this
>>> prompt from rsync:
>>>
>>> The authenticity of host 'revcoders.org (67.19.54.130)' can't be
>>> established.
>>> RSA key fingerprint is 9f:8c:ba:a9:5d:3f:b4:ef:f7:4a:2c:20:cd:77:b3:8c.
>>> Are you sure you want to continue connecting (yes/no)? yes
>>>
>>> Somehow, we either need to:
>>>
>>> 1. "expect" this possible response (the first time only) and answer "yes"
>>>
>>
>> This script works for that, whether the fingerprint has already been
>> stored or not:
>>
>> #!/usr/bin/expect -f
>> spawn rsync -avzrt
>> revcoder@...:/home/revcoder/public_html/revcoder_rsync_test/
>> /revcoder_rsync_test
>> expect "connecting" { send "yes\n"}
>> expect "password:" { send "cookies\n"}
>> expect "#"
>>
>> BTW, when testing this on my own site, I found that the "expect"
>> parameters are case sensitive. My web host sends "Password:" with
>> upper case "P" and I had to change the script to match.
>>
>>
>>> 2. or, would it be a better solution to check for the key fingerprint on
>>> the
>>> local machine and create it if necessary? Or, perhaps just create it
>>> temporarily and delete it at the end of the session for security
>>> purposes?
>>>
>>
>> I found that the fingerprints are stored in ~/.ssh/known_hosts but it
>> is encoded. You can detect if the fingerprint already exists and I
>> guess you could delete it after every transaction.
>>
>> Cheers,
>> Sarah
>> _______________________________________________
>> use-revolution mailing list
>> use-revolution@...
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-revolution
>>
>
> _______________________________________________
> use-revolution mailing list
> use-revolution@...
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution
>

_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution