You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Sorin Silaghi <so...@gmail.com> on 2011/08/30 15:25:31 UTC
random sort in FTP pooling
Hi all,
We wrote a read lock strategy that allows us to use multiple
FTP consumers for the same directory. The problem we are facing now is that
when using multiple consumers they all go for the same file in the list. To
solve this we would need FTP consumers to go after a random file from the
directory.
I wrote a Comparator that returns a random value and I thought
we could use it to sort the file list. I know it's not the most elegant
solution but it works fine with Collections.sort(). The problem is Camel
doesn't seem to use it. It doesn't even pick it up actually.
My question is what would be the best way to do this (any
other suggestions besides the random comparator) and anybody has any idea
why this wouldn't work?
Sorin.
Re: random sort in FTP pooling
Posted by Sorin Silaghi <so...@gmail.com>.
Hi,
I did some debugging and I think I figured it out: we also
have maxMessagesPerPoll=1 set on that endpoint. This means that the consumer
generates a list of 1 elements that it then tries to sort.
We are using maxMessagesPerPoll=1 because we have multiple routes
connected through NMR (it's all deployed in ServiceMix) and we had problems
with the FTP connector. I remember there was also a different (better)
solution but at the time we thought that this works well enough for now. I
guess it's time to revisit the old solution.
thanks,
Sorin.
ps: should we submit a patch with this read lock strategy? We would love it
if this became part of Camel.
On Tue, Aug 30, 2011 at 4:51 PM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> Have you seen this unit test which uses a custom comparator
>
> https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFtpRemoteFileSorterTest.java
>
> On Tue, Aug 30, 2011 at 3:25 PM, Sorin Silaghi <so...@gmail.com>
> wrote:
> > Hi all,
> >
> >
> > We wrote a read lock strategy that allows us to use multiple
> > FTP consumers for the same directory. The problem we are facing now is
> that
> > when using multiple consumers they all go for the same file in the list.
> To
> > solve this we would need FTP consumers to go after a random file from the
> > directory.
> >
> > I wrote a Comparator that returns a random value and I
> thought
> > we could use it to sort the file list. I know it's not the most elegant
> > solution but it works fine with Collections.sort(). The problem is Camel
> > doesn't seem to use it. It doesn't even pick it up actually.
> >
> > My question is what would be the best way to do this (any
> > other suggestions besides the random comparator) and anybody has any idea
> > why this wouldn't work?
> >
> >
> > Sorin.
> >
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>
Re: random sort in FTP pooling
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Have you seen this unit test which uses a custom comparator
https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFtpRemoteFileSorterTest.java
On Tue, Aug 30, 2011 at 3:25 PM, Sorin Silaghi <so...@gmail.com> wrote:
> Hi all,
>
>
> We wrote a read lock strategy that allows us to use multiple
> FTP consumers for the same directory. The problem we are facing now is that
> when using multiple consumers they all go for the same file in the list. To
> solve this we would need FTP consumers to go after a random file from the
> directory.
>
> I wrote a Comparator that returns a random value and I thought
> we could use it to sort the file list. I know it's not the most elegant
> solution but it works fine with Collections.sort(). The problem is Camel
> doesn't seem to use it. It doesn't even pick it up actually.
>
> My question is what would be the best way to do this (any
> other suggestions besides the random comparator) and anybody has any idea
> why this wouldn't work?
>
>
> Sorin.
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/