You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@servicemix.apache.org by abrighton <Al...@t-online.de> on 2006/10/30 17:37:41 UTC
servicemix-ftp: FtpPollingEndpoint?
I tried out the new servicemix-ftp component in the 3.1 snapshot. The current
version supports sending files to
an ftp server, but not yet polling an ftp server for files to download. I
started working on a FtpPollingEndpoint
class, similar to the FilePollingEndpoint class in servicemix-file.
Is someone else also working on this now? Should I create a JIRA issue for
it?
--
Allan
--
View this message in context: http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7077986
Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
Re: servicemix-ftp: FtpPollingEndpoint?
Posted by abrighton <Al...@t-online.de>.
Thanks for checking in the changes.
I tried out the ftp component and noticed a few minor issues so far:
* The oro jar is missing (the version isn't specified in the top level
servicemix pom.xml and it is not
found at runtime. (What is the best way to deal with this? I just copied
it to servicemix/lib for now.)
* I get an endless loop on "." (using pureftp server on Linux):
DEBUG - FtpComponent - Polling directory /tmp/ftpPull
DEBUG - FileComponent - Service unit started
DEBUG - FtpComponent - Polling directory /tmp/ftpPull/.
DEBUG - FtpComponent - Polling directory /tmp/ftpPull/./.
DEBUG - FtpComponent - Polling directory
/tmp/ftpPull/././.
DEBUG - FtpComponent - Polling directory
/tmp/ftpPull/./././.
DEBUG - FtpComponent - Polling directory
/tmp/ftpPull/././././.
...
So we need to ignore "." and "..".
Can we also add servicemix-ftp to the list of components, so that it is
installed in the components dir by default?
Thanks,
Allan
--
View this message in context: http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7082441
Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
Re: servicemix-ftp: FtpPollingEndpoint?
Posted by Guillaume Nodet <gn...@gmail.com>.
On 10/30/06, abrighton <Al...@t-online.de> wrote:
>
> Thanks for the tip about the binaryFileMarshaler. I didn't realize you could
> do it that way.
The marshaler is the easiest way to customize the behavior of endpoints.
>
> As for the memory usage: I was assuming that at some point the entire file
> would be read into memory.
> If this is not the case, it should not be a problem.
It can be, usually if the target component need that to process
the incoming exchange, but the BC themselves won't.
>
> --
> Allan
>
>
> gnodet wrote:
> >
> > Currently, the same pattern as for lightweight components can be
> > used. You can define a BinaryFileMarshaler implementation
> > on the endpoint:
> > <ftp:pollingEndpoint ...>
> > <ftp:marshaler>
> > <sm:binaryFileMarshaler />
> > </ftp:marshaler>
> > </ftp:pollingEndpoint>
> >
> > The DefaultFileMarshaler and BinaryFileMarshaler only send streams
> > (on top of the file or underlying ftp connection), so there should be
> > no memory problems. What are you referring to ?
> >
> > One enhancement would be to use an asynchronous send instead of
> > a sendSync to avoid blocking threads, but it's another problem.
> >
> > On 10/30/06, abrighton <Al...@t-online.de> wrote:
> >>
> >> What is the best way to deal with files that are not XML in the ftp and
> >> file
> >> polling components?
> >> Send them as attachments? Are you working on anything in that direction?
> >> I
> >> suppose it could
> >> be a property of the endpoint class, configured in xbean.xml:
> >> wrapFiles="true"...
> >>
> >> The other issue is how to deal with large files that you don't want to
> >> have
> >> multiple times in memory,
> >> but that is another issue. Maybe that could be another property setting,
> >> indicating to use some kind
> >> of reference to the file instead of the file itself?
> >>
> >> --
> >> Allan
> >>
> >> --
> >> View this message in context:
> >> http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7083459
> >> Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > Cheers,
> > Guillaume Nodet
> >
> >
>
> --
> View this message in context: http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7084079
> Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet
Re: servicemix-ftp: FtpPollingEndpoint?
Posted by abrighton <Al...@t-online.de>.
Thanks for the tip about the binaryFileMarshaler. I didn't realize you could
do it that way.
As for the memory usage: I was assuming that at some point the entire file
would be read into memory.
If this is not the case, it should not be a problem.
--
Allan
gnodet wrote:
>
> Currently, the same pattern as for lightweight components can be
> used. You can define a BinaryFileMarshaler implementation
> on the endpoint:
> <ftp:pollingEndpoint ...>
> <ftp:marshaler>
> <sm:binaryFileMarshaler />
> </ftp:marshaler>
> </ftp:pollingEndpoint>
>
> The DefaultFileMarshaler and BinaryFileMarshaler only send streams
> (on top of the file or underlying ftp connection), so there should be
> no memory problems. What are you referring to ?
>
> One enhancement would be to use an asynchronous send instead of
> a sendSync to avoid blocking threads, but it's another problem.
>
> On 10/30/06, abrighton <Al...@t-online.de> wrote:
>>
>> What is the best way to deal with files that are not XML in the ftp and
>> file
>> polling components?
>> Send them as attachments? Are you working on anything in that direction?
>> I
>> suppose it could
>> be a property of the endpoint class, configured in xbean.xml:
>> wrapFiles="true"...
>>
>> The other issue is how to deal with large files that you don't want to
>> have
>> multiple times in memory,
>> but that is another issue. Maybe that could be another property setting,
>> indicating to use some kind
>> of reference to the file instead of the file itself?
>>
>> --
>> Allan
>>
>> --
>> View this message in context:
>> http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7083459
>> Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Cheers,
> Guillaume Nodet
>
>
--
View this message in context: http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7084079
Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
Re: servicemix-ftp: FtpPollingEndpoint?
Posted by Guillaume Nodet <gn...@gmail.com>.
Currently, the same pattern as for lightweight components can be
used. You can define a BinaryFileMarshaler implementation
on the endpoint:
<ftp:pollingEndpoint ...>
<ftp:marshaler>
<sm:binaryFileMarshaler />
</ftp:marshaler>
</ftp:pollingEndpoint>
The DefaultFileMarshaler and BinaryFileMarshaler only send streams
(on top of the file or underlying ftp connection), so there should be
no memory problems. What are you referring to ?
One enhancement would be to use an asynchronous send instead of
a sendSync to avoid blocking threads, but it's another problem.
On 10/30/06, abrighton <Al...@t-online.de> wrote:
>
> What is the best way to deal with files that are not XML in the ftp and file
> polling components?
> Send them as attachments? Are you working on anything in that direction? I
> suppose it could
> be a property of the endpoint class, configured in xbean.xml:
> wrapFiles="true"...
>
> The other issue is how to deal with large files that you don't want to have
> multiple times in memory,
> but that is another issue. Maybe that could be another property setting,
> indicating to use some kind
> of reference to the file instead of the file itself?
>
> --
> Allan
>
> --
> View this message in context: http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7083459
> Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet
Re: servicemix-ftp: FtpPollingEndpoint?
Posted by abrighton <Al...@t-online.de>.
What is the best way to deal with files that are not XML in the ftp and file
polling components?
Send them as attachments? Are you working on anything in that direction? I
suppose it could
be a property of the endpoint class, configured in xbean.xml:
wrapFiles="true"...
The other issue is how to deal with large files that you don't want to have
multiple times in memory,
but that is another issue. Maybe that could be another property setting,
indicating to use some kind
of reference to the file instead of the file itself?
--
Allan
--
View this message in context: http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7083459
Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
Re: servicemix-ftp: FtpPollingEndpoint?
Posted by Guillaume Nodet <gn...@gmail.com>.
I have just committed why work, sorry about that.
Feel free to take a look at it and add any features you need.
One remaining problem is that ftp / file operations are
not atomic. It means that if you use a FTP writer and a
FTP poller on the same dir, the poller may want to process
the file before the writer has completed the writing.
In this case, the file may be lost. So we may need something
to prevent the poller from reading and deleting the file before
it has been fully written (maybe use a temp file starting with
a known pattern, and use a filter on the poller). Or use a lock file
to prevent a ftp poller / writer to process a file (it would also work
for SM-725 about clustering polling endpoints).
On 10/30/06, Guillaume Nodet <gn...@gmail.com> wrote:
> Well, I've nearly finished it ...
> I will commit soon, and you should be able to give feedback.
>
> On 10/30/06, abrighton <Al...@t-online.de> wrote:
> >
> > I tried out the new servicemix-ftp component in the 3.1 snapshot. The current
> > version supports sending files to
> > an ftp server, but not yet polling an ftp server for files to download. I
> > started working on a FtpPollingEndpoint
> > class, similar to the FilePollingEndpoint class in servicemix-file.
> >
> > Is someone else also working on this now? Should I create a JIRA issue for
> > it?
> >
> > --
> > Allan
> >
> > --
> > View this message in context: http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7077986
> > Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
> >
> >
>
>
> --
> Cheers,
> Guillaume Nodet
>
--
Cheers,
Guillaume Nodet
Re: servicemix-ftp: FtpPollingEndpoint?
Posted by Guillaume Nodet <gn...@gmail.com>.
Well, I've nearly finished it ...
I will commit soon, and you should be able to give feedback.
On 10/30/06, abrighton <Al...@t-online.de> wrote:
>
> I tried out the new servicemix-ftp component in the 3.1 snapshot. The current
> version supports sending files to
> an ftp server, but not yet polling an ftp server for files to download. I
> started working on a FtpPollingEndpoint
> class, similar to the FilePollingEndpoint class in servicemix-file.
>
> Is someone else also working on this now? Should I create a JIRA issue for
> it?
>
> --
> Allan
>
> --
> View this message in context: http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7077986
> Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet
Re: servicemix-ftp: FtpPollingEndpoint?
Posted by Guillaume Nodet <gn...@gmail.com>.
There are different ways:
* add some properties to control the window where we want polling to occur
* the polling endpoint could be a provider which would expose an interface
(on the jbi bus) to suspend / resume polling. This could also be
done on the component
with a kind of "management interface" (a JBI endpoint) where you could
suspend / resume a specific endpoint.
* create another type of endpoint, a new poller, but which would be a
provider rather than a consumer. When receiving an InOut exchange,
it would poll for a single file. This could be used with another type
of polling endpoint, which would only detect changes and send a
notification
event instead of sending the actual content. Philip Dodds was
talking about
that.
I like the second idea which should not be too hard to implement on
the component.
This would also allow to create additional management operations to dynamically
create a consumer endpoint (provider endpoints can be dynamically created
using endpoint resolution). This management interface could be somehow
common to all servicemix BCs. We need to define a nice WSDL for it in
servicemix-common and make all components activate a JBI endpoint.
Given this, we could easily use URIs to automatically create BC consumer
endpoints at deployment time if a service engine has some needed metadata.
This endpoint could be then called by a quartz component, so that you
could defined time windows where you want your endpoint to be active.
However, the first solution would be easier to implement and there would
be no problems to determine the state of the endpoint when the SU
is started: if we use a quartz component, it needs a way to know that
the JBI endpoint has been activate to send a message to suspend / resume
it depending on the current time wrt to the defined windows. But I
suspect that a single property would not be sufficient, and that you would
need to embed quartz to handle cron expressions...
So the third solution (at least the provider poller endpoint) could also
be a good fit. You would need another component to be able to
send requests to the poller and redirect them, and you could configure
the time windows definitions on it.
Thoughts ?
On 10/31/06, abrighton <Al...@t-online.de> wrote:
>
> Thanks, it works now.
>
> Another question: Suppose I want to poll the ftp server only at certain
> times: For example only
> once a week on a Friday at 12:00? Maybe we could add a property for this to
> the endpoint class
> that could be used in place of "delay", if set. Would it make sense to use
> Quartz for this?
> Are there any JBI components that do something similar?
>
> Thanks,
> Allan
>
>
> gnodet wrote:
> >
> > I've committed the fix, thx.
> > Btw, i moved the test up in the loop, so that
> > "." and ".." are checked first, before checking
> > the file type.
> >
>
> --
> View this message in context: http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7092077
> Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet
Re: servicemix-ftp: FtpPollingEndpoint?
Posted by abrighton <Al...@t-online.de>.
Thanks, it works now.
Another question: Suppose I want to poll the ftp server only at certain
times: For example only
once a week on a Friday at 12:00? Maybe we could add a property for this to
the endpoint class
that could be used in place of "delay", if set. Would it make sense to use
Quartz for this?
Are there any JBI components that do something similar?
Thanks,
Allan
gnodet wrote:
>
> I've committed the fix, thx.
> Btw, i moved the test up in the loop, so that
> "." and ".." are checked first, before checking
> the file type.
>
--
View this message in context: http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7092077
Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
Re: servicemix-ftp: FtpPollingEndpoint?
Posted by Guillaume Nodet <gn...@gmail.com>.
I've committed the fix, thx.
Btw, i moved the test up in the loop, so that
"." and ".." are checked first, before checking
the file type.
On 10/31/06, abrighton <Al...@t-online.de> wrote:
>
> I corrected the patch for the ftp polling component
> (The check for "." and ".." was in the wrong place - sorry).
>
> --
> Allan
>
>
> abrighton wrote:
> >
> >
> > Here is a (corrected) patch for the recursion problem:
> > http://www.nabble.com/file/3917/FtpPollingEndpoint.patch
> > FtpPollingEndpoint.patch
> >
> > Here is the changed method:
> >
> > protected void pollFileOrDirectory(FTPClient ftp, String
> > fileOrDirectory, boolean processDir) throws Exception {
> > FTPFile[] files = ftp.listFiles(fileOrDirectory);
> > for (int i = 0; i < files.length; i++) {
> > String file = fileOrDirectory + "/" + files[i].getName();
> > File f = new File(file);
> > if (files[i].isFile()) {
> > if (getFilter() == null || getFilter().accept(f)) {
> > pollFile(file); // process the file
> > }
> > } else {
> > if (processDir && files[i].isDirectory()) {
> > String name = f.getName();
> > if (name.equals(".") || name.equals("..")) {
> > continue; // ignore "." and ".."
> > }
> > logger.debug("Polling directory " + file);
> > pollFileOrDirectory(ftp, file, isRecursive());
> > } else {
> > logger.debug("Skipping directory " + file);
> > }
> > }
> > }
> > }
> >
> >
>
> --
> View this message in context: http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7089699
> Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet
Re: servicemix-ftp: FtpPollingEndpoint?
Posted by abrighton <Al...@t-online.de>.
I corrected the patch for the ftp polling component
(The check for "." and ".." was in the wrong place - sorry).
--
Allan
abrighton wrote:
>
>
> Here is a (corrected) patch for the recursion problem:
> http://www.nabble.com/file/3917/FtpPollingEndpoint.patch
> FtpPollingEndpoint.patch
>
> Here is the changed method:
>
> protected void pollFileOrDirectory(FTPClient ftp, String
> fileOrDirectory, boolean processDir) throws Exception {
> FTPFile[] files = ftp.listFiles(fileOrDirectory);
> for (int i = 0; i < files.length; i++) {
> String file = fileOrDirectory + "/" + files[i].getName();
> File f = new File(file);
> if (files[i].isFile()) {
> if (getFilter() == null || getFilter().accept(f)) {
> pollFile(file); // process the file
> }
> } else {
> if (processDir && files[i].isDirectory()) {
> String name = f.getName();
> if (name.equals(".") || name.equals("..")) {
> continue; // ignore "." and ".."
> }
> logger.debug("Polling directory " + file);
> pollFileOrDirectory(ftp, file, isRecursive());
> } else {
> logger.debug("Skipping directory " + file);
> }
> }
> }
> }
>
>
--
View this message in context: http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7089699
Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
Re: servicemix-ftp: FtpPollingEndpoint?
Posted by abrighton <Al...@t-online.de>.
Here is a patch for the recursion problem:
http://www.nabble.com/file/3916/FtpPollingEndpoint.patch
FtpPollingEndpoint.patch
Here is the changed method:
protected void pollFileOrDirectory(FTPClient ftp, String
fileOrDirectory, boolean processDir) throws Exception {
FTPFile[] files = ftp.listFiles(fileOrDirectory);
for (int i = 0; i < files.length; i++) {
String file = fileOrDirectory + "/" + files[i].getName();
if (!files[i].isDirectory()) {
File f = new File(file);
String name = f.getName();
if (name.equals(".") || name.equals("..")) {
continue; // ignore "." and ".."
}
if (getFilter() == null || getFilter().accept(f)) {
pollFile(file); // process the file
}
} else if (processDir) {
logger.debug("Polling directory " + file);
pollFileOrDirectory(ftp, file, isRecursive());
} else {
logger.debug("Skipping directory " + file);
}
}
}
--
View this message in context: http://www.nabble.com/servicemix-ftp%3A-FtpPollingEndpoint--tf2540539.html#a7082814
Sent from the ServiceMix - Dev mailing list archive at Nabble.com.