Overall notes on FTP/SFTP component

[Drew McAuliffe]

I've been doing some pretty extensive work with the FTP/SFTP connectors over the past month and wanted to post some of my impressions. I've been working on CAMEL-570 and will be posting some changes to that issue related to auto-delete functionality on the component. But here's a list of issues I've been accumulating over time:

Problems with SFTP/FTP connector
1. No provision for deleting files when read; I added this (see CAMEL-570).
2. Confusing which properties require “consumer.” prefix and which don’t, in documentation. For example, use of “setNames” is incorrect because it requires “consumer.”, though the documentation shows it without.
3. Checking lastPollTime should be turned off when using autoDelete (or similar function). I added this in the patch submitted to CAMEL-570. I ran into problems because my FTP server (CrushFTP) wouldn’t return seconds in last modified time, so file was never checked. Not sure if that’s a problem just with CrushFTP or in how FTPClient gets last modified time.
4. SFTP connector “file” property has to be set using full path or else it doesn’t work. For example, if the user’s home directory is “/home/myuser”, and you’re polling a directory called “pollMe”, the docs lead you to believe that this should work:
sftp://[e-mail]/pollMe
And it does work for ftp, just not for sftp. You actually need to do the following on SFTP:
sftp://[e-mail]//home/myuser/pollMe

Part of the problem with SFTP may be that there don’t appear to be any unit tests with SFTP, only with FTP, which may be a limitation of the test FTP server (apache ftpserver).

[Claus Ibsen]

Hi Drew

Just a quick note to #4:
We currently don't have a SFTP java server that can be used for unit testing. Anyone know of a library or tool etc. that could be used?

Again thank for all the hard work on this patch. We will target it for 1.5.


Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. [phone]
Web: www.silverbullet.dk

-----Original Message-----
From: Drew McAuliffe [mailto:drewmca@...]
Sent: 28. juni 2008 03:26
To: camel-user@...
Subject: Overall notes on FTP/SFTP component


I've been doing some pretty extensive work with the FTP/SFTP connectors over
the past month and wanted to post some of my impressions. I've been working
on CAMEL-570 and will be posting some changes to that issue related to
auto-delete functionality on the component. But here's a list of issues I've
been accumulating over time:

Problems with SFTP/FTP connector
1. No provision for deleting files when read; I added this (see CAMEL-570).
2. Confusing which properties require "consumer." prefix and which don't, in
documentation. For example, use of "setNames" is incorrect because it
requires "consumer.", though the documentation shows it without.
3. Checking lastPollTime should be turned off when using autoDelete (or
similar function). I added this in the patch submitted to CAMEL-570. I ran
into problems because my FTP server (CrushFTP) wouldn't return seconds in
last modified time, so file was never checked. Not sure if that's a problem
just with CrushFTP or in how FTPClient gets last modified time.
4. SFTP connector "file" property has to be set using full path or else it
doesn't work. For example, if the user's home directory is "/home/myuser",
and you're polling a directory called "pollMe", the docs lead you to believe
that this should work:
sftp://myuser@.../pollMe
And it does work for ftp, just not for sftp. You actually need to do the
following on SFTP:
sftp://myuser@...//home/myuser/pollMe

Part of the problem with SFTP may be that there don't appear to be any unit
tests with SFTP, only with FTP, which may be a limitation of the test FTP
server (apache ftpserver).

--
View this message in context: http://www.nabble.com/Overall-notes-on-FTP-SFTP-component-tp18166447s22882p18166447.html
Sent from the Camel - Users mailing list archive at Nabble.com.

[Claus Ibsen]

Hi Drew

Have you based your work on Camel 1.4-SNAPSHOT or Camel 1.3?
We have somehow improved the FTP part in Camel 1.4.


Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. [phone]
Web: www.silverbullet.dk

-----Original Message-----
From: Drew McAuliffe [mailto:drewmca@...]
Sent: 28. juni 2008 03:26
To: camel-user@...
Subject: Overall notes on FTP/SFTP component


I've been doing some pretty extensive work with the FTP/SFTP connectors over
the past month and wanted to post some of my impressions. I've been working
on CAMEL-570 and will be posting some changes to that issue related to
auto-delete functionality on the component. But here's a list of issues I've
been accumulating over time:

Problems with SFTP/FTP connector
1. No provision for deleting files when read; I added this (see CAMEL-570).
2. Confusing which properties require "consumer." prefix and which don't, in
documentation. For example, use of "setNames" is incorrect because it
requires "consumer.", though the documentation shows it without.
3. Checking lastPollTime should be turned off when using autoDelete (or
similar function). I added this in the patch submitted to CAMEL-570. I ran
into problems because my FTP server (CrushFTP) wouldn't return seconds in
last modified time, so file was never checked. Not sure if that's a problem
just with CrushFTP or in how FTPClient gets last modified time.
4. SFTP connector "file" property has to be set using full path or else it
doesn't work. For example, if the user's home directory is "/home/myuser",
and you're polling a directory called "pollMe", the docs lead you to believe
that this should work:
sftp://myuser@.../pollMe
And it does work for ftp, just not for sftp. You actually need to do the
following on SFTP:
sftp://myuser@...//home/myuser/pollMe

Part of the problem with SFTP may be that there don't appear to be any unit
tests with SFTP, only with FTP, which may be a limitation of the test FTP
server (apache ftpserver).

--
View this message in context: http://www.nabble.com/Overall-notes-on-FTP-SFTP-component-tp18166447s22882p18166447.html
Sent from the Camel - Users mailing list archive at Nabble.com.

[borgel]

I tested 1.4.0-SNAPSHOT today and created a route from file to ftp. This worked very well, but i noted two issues:
1. It had problems with folders. Sometimes it worked, sometimes it didn't
2. When putting large files in the watched folder it started working on the file before it was complete. This resultet in error and I didn't find a way to set the delay and initialDelay properties.

Drew McAuliffe wrote:

I've been doing some pretty extensive work with the FTP/SFTP connectors over the past month and wanted to post some of my impressions. I've been working on CAMEL-570 and will be posting some changes to that issue related to auto-delete functionality on the component. But here's a list of issues I've been accumulating over time:

Problems with SFTP/FTP connector
1. No provision for deleting files when read; I added this (see CAMEL-570).
2. Confusing which properties require “consumer.” prefix and which don’t, in documentation. For example, use of “setNames” is incorrect because it requires “consumer.”, though the documentation shows it without.
3. Checking lastPollTime should be turned off when using autoDelete (or similar function). I added this in the patch submitted to CAMEL-570. I ran into problems because my FTP server (CrushFTP) wouldn’t return seconds in last modified time, so file was never checked. Not sure if that’s a problem just with CrushFTP or in how FTPClient gets last modified time.
4. SFTP connector “file” property has to be set using full path or else it doesn’t work. For example, if the user’s home directory is “/home/myuser”, and you’re polling a directory called “pollMe”, the docs lead you to believe that this should work:
sftp://[e-mail]/pollMe
And it does work for ftp, just not for sftp. You actually need to do the following on SFTP:
sftp://[e-mail]//home/myuser/pollMe

Part of the problem with SFTP may be that there don’t appear to be any unit tests with SFTP, only with FTP, which may be a limitation of the test FTP server (apache ftpserver).

[Claus Ibsen]

Hi

Ad 1)
If you find the cause of the problem or how to reproduce it please let us know

Ad 2)
The FTP and File components use the "consumer." Prefix for delay and initialDelay.
http://activemq.apache.org/camel/file.html

It is actually not documented very well which properties needs the consumer. prefix. Beware we might change this in the future to not require the "consumer." prefix as its not much used, and maybe a bit confusing.

There is a JIRA ticket for better documentation.

There are these options:
consumer.initialDelay
consumer.delay
consumer.useFixedDelay


And the times are in millis.


Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. [phone]
Web: www.silverbullet.dk

-----Original Message-----
From: borgel [mailto:borge.lotre@...]
Sent: 30. juni 2008 17:40
To: camel-user@...
Subject: Re: Overall notes on FTP/SFTP component


I tested 1.4.0-SNAPSHOT today and created a route from file to ftp. This
worked very well, but i noted two issues:
1. It had problems with folders. Sometimes it worked, sometimes it didn't
2. When putting large files in the watched folder it started working on the
file before it was complete. This resultet in error and I didn't find a way
to set the delay and initialDelay properties.



Drew McAuliffe wrote:

>
> I've been doing some pretty extensive work with the FTP/SFTP connectors
> over the past month and wanted to post some of my impressions. I've been
> working on CAMEL-570 and will be posting some changes to that issue
> related to auto-delete functionality on the component. But here's a list
> of issues I've been accumulating over time:
>
> Problems with SFTP/FTP connector
> 1. No provision for deleting files when read; I added this (see
> CAMEL-570).
> 2. Confusing which properties require "consumer." prefix and which don't,
> in documentation. For example, use of "setNames" is incorrect because it
> requires "consumer.", though the documentation shows it without.
> 3. Checking lastPollTime should be turned off when using autoDelete (or
> similar function). I added this in the patch submitted to CAMEL-570. I ran
> into problems because my FTP server (CrushFTP) wouldn't return seconds in
> last modified time, so file was never checked. Not sure if that's a
> problem just with CrushFTP or in how FTPClient gets last modified time.
> 4. SFTP connector "file" property has to be set using full path or else it
> doesn't work. For example, if the user's home directory is "/home/myuser",
> and you're polling a directory called "pollMe", the docs lead you to
> believe that this should work:
> sftp://myuser@.../pollMe
> And it does work for ftp, just not for sftp. You actually need to do the
> following on SFTP:
> sftp://myuser@...//home/myuser/pollMe
>
> Part of the problem with SFTP may be that there don't appear to be any
> unit tests with SFTP, only with FTP, which may be a limitation of the test
> FTP server (apache ftpserver).
>
>

--
View this message in context: http://www.nabble.com/Overall-notes-on-FTP-SFTP-component-tp18166447s22882p18198352.html
Sent from the Camel - Users mailing list archive at Nabble.com.

[Claus Ibsen]

Hi

Do you mind creating a JIRA ticket for the FTP issue with:
- should not poll the file before the file has been fully transferred.

Camel should support different strategies for this
- done file (eg. the producer create a 2nd done file when the transfer is complete)
- lock file (eg. the producer creates a .lock file so we know we shouldn't poll the file before the lock file is deleted by the producer)
- timestamp/size not changing (checking the timestamp of the file and that it hasn't changed for the last xxx minutes etc.)

Anyone knows of other strategies that are well used?


Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. [phone]
Web: www.silverbullet.dk

-----Original Message-----
From: borgel [mailto:borge.lotre@...]
Sent: 30. juni 2008 17:40
To: camel-user@...
Subject: Re: Overall notes on FTP/SFTP component


I tested 1.4.0-SNAPSHOT today and created a route from file to ftp. This
worked very well, but i noted two issues:
1. It had problems with folders. Sometimes it worked, sometimes it didn't
2. When putting large files in the watched folder it started working on the
file before it was complete. This resultet in error and I didn't find a way
to set the delay and initialDelay properties.



Drew McAuliffe wrote:

>
> I've been doing some pretty extensive work with the FTP/SFTP connectors
> over the past month and wanted to post some of my impressions. I've been
> working on CAMEL-570 and will be posting some changes to that issue
> related to auto-delete functionality on the component. But here's a list
> of issues I've been accumulating over time:
>
> Problems with SFTP/FTP connector
> 1. No provision for deleting files when read; I added this (see
> CAMEL-570).
> 2. Confusing which properties require "consumer." prefix and which don't,
> in documentation. For example, use of "setNames" is incorrect because it
> requires "consumer.", though the documentation shows it without.
> 3. Checking lastPollTime should be turned off when using autoDelete (or
> similar function). I added this in the patch submitted to CAMEL-570. I ran
> into problems because my FTP server (CrushFTP) wouldn't return seconds in
> last modified time, so file was never checked. Not sure if that's a
> problem just with CrushFTP or in how FTPClient gets last modified time.
> 4. SFTP connector "file" property has to be set using full path or else it
> doesn't work. For example, if the user's home directory is "/home/myuser",
> and you're polling a directory called "pollMe", the docs lead you to
> believe that this should work:
> sftp://myuser@.../pollMe
> And it does work for ftp, just not for sftp. You actually need to do the
> following on SFTP:
> sftp://myuser@...//home/myuser/pollMe
>
> Part of the problem with SFTP may be that there don't appear to be any
> unit tests with SFTP, only with FTP, which may be a limitation of the test
> FTP server (apache ftpserver).
>
>

--
View this message in context: http://www.nabble.com/Overall-notes-on-FTP-SFTP-component-tp18166447s22882p18198352.html
Sent from the Camel - Users mailing list archive at Nabble.com.

[borgel]

https://issues.apache.org/activemq/browse/CAMEL-654

Claus Ibsen wrote:

Hi

Do you mind creating a JIRA ticket for the FTP issue with:
- should not poll the file before the file has been fully transferred.

Camel should support different strategies for this
- done file (eg. the producer create a 2nd done file when the transfer is complete)
- lock file (eg. the producer creates a .lock file so we know we shouldn't poll the file before the lock file is deleted by the producer)
- timestamp/size not changing (checking the timestamp of the file and that it hasn't changed for the last xxx minutes etc.)

Anyone knows of other strategies that are well used?


Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. [phone]
Web: www.silverbullet.dk

-----Original Message-----
From: borgel [mailto:[e-mail]]
Sent: 30. juni 2008 17:40
To: [e-mail]
Subject: Re: Overall notes on FTP/SFTP component


I tested 1.4.0-SNAPSHOT today and created a route from file to ftp. This
worked very well, but i noted two issues:
1. It had problems with folders. Sometimes it worked, sometimes it didn't
2. When putting large files in the watched folder it started working on the
file before it was complete. This resultet in error and I didn't find a way
to set the delay and initialDelay properties.



Drew McAuliffe wrote:
>
> I've been doing some pretty extensive work with the FTP/SFTP connectors
> over the past month and wanted to post some of my impressions. I've been
> working on CAMEL-570 and will be posting some changes to that issue
> related to auto-delete functionality on the component. But here's a list
> of issues I've been accumulating over time:
>
> Problems with SFTP/FTP connector
> 1. No provision for deleting files when read; I added this (see
> CAMEL-570).
> 2. Confusing which properties require "consumer." prefix and which don't,
> in documentation. For example, use of "setNames" is incorrect because it
> requires "consumer.", though the documentation shows it without.
> 3. Checking lastPollTime should be turned off when using autoDelete (or
> similar function). I added this in the patch submitted to CAMEL-570. I ran
> into problems because my FTP server (CrushFTP) wouldn't return seconds in
> last modified time, so file was never checked. Not sure if that's a
> problem just with CrushFTP or in how FTPClient gets last modified time.
> 4. SFTP connector "file" property has to be set using full path or else it
> doesn't work. For example, if the user's home directory is "/home/myuser",
> and you're polling a directory called "pollMe", the docs lead you to
> believe that this should work:
> sftp://[e-mail]/pollMe
> And it does work for ftp, just not for sftp. You actually need to do the
> following on SFTP:
> sftp://[e-mail]//home/myuser/pollMe
>
> Part of the problem with SFTP may be that there don't appear to be any
> unit tests with SFTP, only with FTP, which may be a limitation of the test
> FTP server (apache ftpserver).
>
>


--
View this message in context: http://www.nabble.com/Overall-notes-on-FTP-SFTP-component-tp18166447s22882p18198352.html
Sent from the Camel - Users mailing list archive at Nabble.com.

[Drew McAuliffe]

I think it's based on 1.3, though some of the 1.4 changes to the file connector are incorporated. Specifically, there was a problem in 1.3 with the file connector where an error wouldn't remove the lock file. I based the async code block on the 1.4 file connector and how it handles deleting files. I think. It's been a while since I looked at it.

Do you know specifically what changes were made to the ftp connector in the latest 1.4?

Claus Ibsen wrote:

Hi Drew

Have you based your work on Camel 1.4-SNAPSHOT or Camel 1.3?
We have somehow improved the FTP part in Camel 1.4.


Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. [phone]
Web: www.silverbullet.dk

-----Original Message-----
From: Drew McAuliffe [mailto:[e-mail]]
Sent: 28. juni 2008 03:26
To: [e-mail]
Subject: Overall notes on FTP/SFTP component


I've been doing some pretty extensive work with the FTP/SFTP connectors over
the past month and wanted to post some of my impressions. I've been working
on CAMEL-570 and will be posting some changes to that issue related to
auto-delete functionality on the component. But here's a list of issues I've
been accumulating over time:

Problems with SFTP/FTP connector
1. No provision for deleting files when read; I added this (see CAMEL-570).
2. Confusing which properties require "consumer." prefix and which don't, in
documentation. For example, use of "setNames" is incorrect because it
requires "consumer.", though the documentation shows it without.
3. Checking lastPollTime should be turned off when using autoDelete (or
similar function). I added this in the patch submitted to CAMEL-570. I ran
into problems because my FTP server (CrushFTP) wouldn't return seconds in
last modified time, so file was never checked. Not sure if that's a problem
just with CrushFTP or in how FTPClient gets last modified time.
4. SFTP connector "file" property has to be set using full path or else it
doesn't work. For example, if the user's home directory is "/home/myuser",
and you're polling a directory called "pollMe", the docs lead you to believe
that this should work:
sftp://[e-mail]/pollMe
And it does work for ftp, just not for sftp. You actually need to do the
following on SFTP:
sftp://[e-mail]//home/myuser/pollMe

Part of the problem with SFTP may be that there don't appear to be any unit
tests with SFTP, only with FTP, which may be a limitation of the test FTP
server (apache ftpserver).

--
View this message in context: http://www.nabble.com/Overall-notes-on-FTP-SFTP-component-tp18166447s22882p18166447.html
Sent from the Camel - Users mailing list archive at Nabble.com.

[Claus Ibsen]

Hi Drew

> Do you know specifically what changes were made to the ftp connector in the latest 1.4?
I do think the problem with SFTP requires the full path, so FTP and SFTP have the same code.

But they are tracked here:
http://issues.apache.org/activemq/browse/CAMEL-420
http://issues.apache.org/activemq/browse/CAMEL-424

And the changes can be tracked from the commit logs on CAMEL-420 and its subtasks.


Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. [phone]
Web: www.silverbullet.dk

-----Original Message-----
From: Drew McAuliffe [mailto:drewmca@...]
Sent: 2. juli 2008 03:09
To: camel-user@...
Subject: RE: Overall notes on FTP/SFTP component


I think it's based on 1.3, though some of the 1.4 changes to the file
connector are incorporated. Specifically, there was a problem in 1.3 with
the file connector where an error wouldn't remove the lock file. I based the
async code block on the 1.4 file connector and how it handles deleting
files. I think. It's been a while since I looked at it.

Do you know specifically what changes were made to the ftp connector in the
latest 1.4?

Claus Ibsen wrote:

>
> Hi Drew
>
> Have you based your work on Camel 1.4-SNAPSHOT or Camel 1.3?
> We have somehow improved the FTP part in Camel 1.4.
>
>
> Med venlig hilsen
>  
> Claus Ibsen
> ......................................
> Silverbullet
> Skovsgårdsvænget 21
> 8362 Hørning
> Tlf. [phone]
> Web: www.silverbullet.dk
>
> -----Original Message-----
> From: Drew McAuliffe [mailto:drewmca@...]
> Sent: 28. juni 2008 03:26
> To: camel-user@...
> Subject: Overall notes on FTP/SFTP component
>
>
> I've been doing some pretty extensive work with the FTP/SFTP connectors
> over
> the past month and wanted to post some of my impressions. I've been
> working
> on CAMEL-570 and will be posting some changes to that issue related to
> auto-delete functionality on the component. But here's a list of issues
> I've
> been accumulating over time:
>
> Problems with SFTP/FTP connector
> 1. No provision for deleting files when read; I added this (see
> CAMEL-570).
> 2. Confusing which properties require "consumer." prefix and which don't,
> in
> documentation. For example, use of "setNames" is incorrect because it
> requires "consumer.", though the documentation shows it without.
> 3. Checking lastPollTime should be turned off when using autoDelete (or
> similar function). I added this in the patch submitted to CAMEL-570. I ran
> into problems because my FTP server (CrushFTP) wouldn't return seconds in
> last modified time, so file was never checked. Not sure if that's a
> problem
> just with CrushFTP or in how FTPClient gets last modified time.
> 4. SFTP connector "file" property has to be set using full path or else it
> doesn't work. For example, if the user's home directory is "/home/myuser",
> and you're polling a directory called "pollMe", the docs lead you to
> believe
> that this should work:
> sftp://myuser@.../pollMe
> And it does work for ftp, just not for sftp. You actually need to do the
> following on SFTP:
> sftp://myuser@...//home/myuser/pollMe
>
> Part of the problem with SFTP may be that there don't appear to be any
> unit
> tests with SFTP, only with FTP, which may be a limitation of the test FTP
> server (apache ftpserver).
>
> --
> View this message in context:
> http://www.nabble.com/Overall-notes-on-FTP-SFTP-component-tp18166447s22882p18166447.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>

--
View this message in context: http://www.nabble.com/Overall-notes-on-FTP-SFTP-component-tp18166447s22882p18228375.html
Sent from the Camel - Users mailing list archive at Nabble.com.

[Drew McAuliffe]

Is there an ETA on the autodelete changes for FTP/SFTP? I made these changes before and would like to upgrade to the official 1.4 but I absolutely need that feature, so I can't unless I make the changes again manually myself. Let me know if you want me to resubmit the patch files.

Claus Ibsen wrote:

Hi Drew

> Do you know specifically what changes were made to the ftp connector in the latest 1.4?
I do think the problem with SFTP requires the full path, so FTP and SFTP have the same code.

But they are tracked here:
http://issues.apache.org/activemq/browse/CAMEL-420
http://issues.apache.org/activemq/browse/CAMEL-424

And the changes can be tracked from the commit logs on CAMEL-420 and its subtasks.


Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. [phone]
Web: www.silverbullet.dk

-----Original Message-----
From: Drew McAuliffe [mailto:[e-mail]]
Sent: 2. juli 2008 03:09
To: [e-mail]
Subject: RE: Overall notes on FTP/SFTP component


I think it's based on 1.3, though some of the 1.4 changes to the file
connector are incorporated. Specifically, there was a problem in 1.3 with
the file connector where an error wouldn't remove the lock file. I based the
async code block on the 1.4 file connector and how it handles deleting
files. I think. It's been a while since I looked at it.

Do you know specifically what changes were made to the ftp connector in the
latest 1.4?

Claus Ibsen wrote:
>
> Hi Drew
>
> Have you based your work on Camel 1.4-SNAPSHOT or Camel 1.3?
> We have somehow improved the FTP part in Camel 1.4.
>
>
> Med venlig hilsen
>  
> Claus Ibsen
> ......................................
> Silverbullet
> Skovsgårdsvænget 21
> 8362 Hørning
> Tlf. [phone]
> Web: www.silverbullet.dk
>
> -----Original Message-----
> From: Drew McAuliffe [mailto:[e-mail]]
> Sent: 28. juni 2008 03:26
> To: [e-mail]
> Subject: Overall notes on FTP/SFTP component
>
>
> I've been doing some pretty extensive work with the FTP/SFTP connectors
> over
> the past month and wanted to post some of my impressions. I've been
> working
> on CAMEL-570 and will be posting some changes to that issue related to
> auto-delete functionality on the component. But here's a list of issues
> I've
> been accumulating over time:
>
> Problems with SFTP/FTP connector
> 1. No provision for deleting files when read; I added this (see
> CAMEL-570).
> 2. Confusing which properties require "consumer." prefix and which don't,
> in
> documentation. For example, use of "setNames" is incorrect because it
> requires "consumer.", though the documentation shows it without.
> 3. Checking lastPollTime should be turned off when using autoDelete (or
> similar function). I added this in the patch submitted to CAMEL-570. I ran
> into problems because my FTP server (CrushFTP) wouldn't return seconds in
> last modified time, so file was never checked. Not sure if that's a
> problem
> just with CrushFTP or in how FTPClient gets last modified time.
> 4. SFTP connector "file" property has to be set using full path or else it
> doesn't work. For example, if the user's home directory is "/home/myuser",
> and you're polling a directory called "pollMe", the docs lead you to
> believe
> that this should work:
> sftp://[e-mail]/pollMe
> And it does work for ftp, just not for sftp. You actually need to do the
> following on SFTP:
> sftp://[e-mail]//home/myuser/pollMe
>
> Part of the problem with SFTP may be that there don't appear to be any
> unit
> tests with SFTP, only with FTP, which may be a limitation of the test FTP
> server (apache ftpserver).
>
> --
> View this message in context:
> http://www.nabble.com/Overall-notes-on-FTP-SFTP-component-tp18166447s22882p18166447.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>

--
View this message in context: http://www.nabble.com/Overall-notes-on-FTP-SFTP-component-tp18166447s22882p18228375.html
Sent from the Camel - Users mailing list archive at Nabble.com.

[Claus Ibsen]

Hi Drew

I am afraid its *not implemented in Camel 1.4.0*. Camel 1.4.0 has been released.

But it's implemented in Camel 1.5 that is currently in development.

If you need it asap. You could probably take a camel-ftp-1.5-SNAPSHOT.jar and use as a camel-ftp-1.4.1.jar and install it in your own local m2 repo and use this .jar instead of the camel-ftp-1.4.0.jar.

I don't have a ETA when 1.5 will be released. It will take 2+ months at least I guess. But anyone is free to suggest a release.

Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. [phone]
Web: www.silverbullet.dk
-----Original Message-----
From: Drew McAuliffe [mailto:drewmca@...]
Sent: 13. august 2008 23:59
To: camel-user@...
Subject: RE: Overall notes on FTP/SFTP component


Is there an ETA on the autodelete changes for FTP/SFTP? I made these changes
before and would like to upgrade to the official 1.4 but I absolutely need
that feature, so I can't unless I make the changes again manually myself.
Let me know if you want me to resubmit the patch files.


Claus Ibsen wrote:

>
> Hi Drew
>
>> Do you know specifically what changes were made to the ftp connector in
>> the latest 1.4?
> I do think the problem with SFTP requires the full path, so FTP and SFTP
> have the same code.
>
> But they are tracked here:
> http://issues.apache.org/activemq/browse/CAMEL-420
> http://issues.apache.org/activemq/browse/CAMEL-424
>
> And the changes can be tracked from the commit logs on CAMEL-420 and its
> subtasks.
>
>
> Med venlig hilsen
>  
> Claus Ibsen
> ......................................
> Silverbullet
> Skovsgårdsvænget 21
> 8362 Hørning
> Tlf. [phone]
> Web: www.silverbullet.dk
>
> -----Original Message-----
> From: Drew McAuliffe [mailto:drewmca@...]
> Sent: 2. juli 2008 03:09
> To: camel-user@...
> Subject: RE: Overall notes on FTP/SFTP component
>
>
> I think it's based on 1.3, though some of the 1.4 changes to the file
> connector are incorporated. Specifically, there was a problem in 1.3 with
> the file connector where an error wouldn't remove the lock file. I based
> the
> async code block on the 1.4 file connector and how it handles deleting
> files. I think. It's been a while since I looked at it.
>
> Do you know specifically what changes were made to the ftp connector in
> the
> latest 1.4?
>
> Claus Ibsen wrote:
>>
>> Hi Drew
>>
>> Have you based your work on Camel 1.4-SNAPSHOT or Camel 1.3?
>> We have somehow improved the FTP part in Camel 1.4.
>>
>>
>> Med venlig hilsen
>>  
>> Claus Ibsen
>> ......................................
>> Silverbullet
>> Skovsgårdsvænget 21
>> 8362 Hørning
>> Tlf. [phone]
>> Web: www.silverbullet.dk
>>
>> -----Original Message-----
>> From: Drew McAuliffe [mailto:drewmca@...]
>> Sent: 28. juni 2008 03:26
>> To: camel-user@...
>> Subject: Overall notes on FTP/SFTP component
>>
>>
>> I've been doing some pretty extensive work with the FTP/SFTP connectors
>> over
>> the past month and wanted to post some of my impressions. I've been
>> working
>> on CAMEL-570 and will be posting some changes to that issue related to
>> auto-delete functionality on the component. But here's a list of issues
>> I've
>> been accumulating over time:
>>
>> Problems with SFTP/FTP connector
>> 1. No provision for deleting files when read; I added this (see
>> CAMEL-570).
>> 2. Confusing which properties require "consumer." prefix and which don't,
>> in
>> documentation. For example, use of "setNames" is incorrect because it
>> requires "consumer.", though the documentation shows it without.
>> 3. Checking lastPollTime should be turned off when using autoDelete (or
>> similar function). I added this in the patch submitted to CAMEL-570. I
>> ran
>> into problems because my FTP server (CrushFTP) wouldn't return seconds in
>> last modified time, so file was never checked. Not sure if that's a
>> problem
>> just with CrushFTP or in how FTPClient gets last modified time.
>> 4. SFTP connector "file" property has to be set using full path or else
>> it
>> doesn't work. For example, if the user's home directory is
>> "/home/myuser",
>> and you're polling a directory called "pollMe", the docs lead you to
>> believe
>> that this should work:
>> sftp://myuser@.../pollMe
>> And it does work for ftp, just not for sftp. You actually need to do the
>> following on SFTP:
>> sftp://myuser@...//home/myuser/pollMe
>>
>> Part of the problem with SFTP may be that there don't appear to be any
>> unit
>> tests with SFTP, only with FTP, which may be a limitation of the test FTP
>> server (apache ftpserver).
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Overall-notes-on-FTP-SFTP-component-tp18166447s22882p18166447.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Overall-notes-on-FTP-SFTP-component-tp18166447s22882p18228375.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>

--
View this message in context: http://www.nabble.com/Overall-notes-on-FTP-SFTP-component-tp18166447s22882p18971977.html
Sent from the Camel - Users mailing list archive at Nabble.com.