You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by Char <ch...@gmx.net> on 2008/12/05 10:29:02 UTC
Filepoller sometimes does not work
Hello,
we are using the jbi:file component to poll a huge amount of files. In a
test szenario, i am deploying 2000 small files, which are polled correctly.
When i deploy another 2000 files, a lot of them (300-500) seem to be ignored
by the filepoller.
Behaviour: You can still delete them, so there is no lock on them. If you
deploy more files, the newer ones are being polled, whereas overwriting the
sticky ones does not help.
Test-Setup: 2000 files with different file names. 2nd deploy with same
filenames.
Configuration:
<file:poller service="FilePoller"
endpoint="FileIn"
targetService="InboundRoutingService"
targetEndpoint="FileInEndpoint"
file="../Data/Inbound/FileIn"
autoCreateDirectory="false"
period="5000"
archive="../Data/Inbound/Archive">
<!-- Custom Marshaler, if you are not going to poll xml files.
-->
<property name="marshaler">
<bean
class="org.apache.servicemix.components.util.BinaryFileMarshaler" />
</property>
</file:poller>
Any ideas how i can analyse this behaviour?
thx in advance
--
View this message in context: http://www.nabble.com/Filepoller-sometimes-does-not-work-tp20850432p20850432.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: Filepoller sometimes does not work
Posted by Lars Heinemann <lh...@apache.org>.
Yesterday I fixed an issue regarding file poller and use of an archive.
See http://issues.apache.org/activemq/browse/SM-1715
Maybe this was the cause of your problems. You should try with the current
trunk version of the
servicemix-file component.
Anyway you should enable DEBUG log level and see what happens in there.
Regards
Lars
2008/12/5 Char <ch...@gmx.net>
>
> Hello,
>
> we are using the jbi:file component to poll a huge amount of files. In a
> test szenario, i am deploying 2000 small files, which are polled correctly.
> When i deploy another 2000 files, a lot of them (300-500) seem to be
> ignored
> by the filepoller.
>
> Behaviour: You can still delete them, so there is no lock on them. If you
> deploy more files, the newer ones are being polled, whereas overwriting the
> sticky ones does not help.
>
> Test-Setup: 2000 files with different file names. 2nd deploy with same
> filenames.
>
> Configuration:
> <file:poller service="FilePoller"
> endpoint="FileIn"
> targetService="InboundRoutingService"
> targetEndpoint="FileInEndpoint"
> file="../Data/Inbound/FileIn"
> autoCreateDirectory="false"
> period="5000"
> archive="../Data/Inbound/Archive">
>
> <!-- Custom Marshaler, if you are not going to poll xml files.
> -->
> <property name="marshaler">
> <bean
> class="org.apache.servicemix.components.util.BinaryFileMarshaler" />
> </property>
> </file:poller>
>
> Any ideas how i can analyse this behaviour?
>
> thx in advance
> --
> View this message in context:
> http://www.nabble.com/Filepoller-sometimes-does-not-work-tp20850432p20850432.html
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
>
--
http://lhein.blogspot.com
Re: Filepoller sometimes does not work
Posted by Lars Heinemann <lh...@apache.org>.
Char,
I applied a patch yesterday which takes care for another problem which may
be the cause in
your case. Sometimes (I monitored this behaviour under Windows OS) the files
being polled are
not yet fully copied into the polling folder. The result is a poller which
tries to lock the file every polling
cycle without any success...seems to be like your problem description.
Please try with servicemix-utils-1.1.0-SNAPSHOT and the current
servicemix-file 2008-02-SNAPSHOT from trunk.
Regards
Lars
2008/12/9 Char <ch...@gmx.net>
>
> Hello Gert,
>
> we are using ServiceMix 3.3, and unfortunately our problem doesn't refer to
> the archive-function. Nevertheless, patching the archive function solves
> other problems...
>
> So what we observe is that the filepoller has problems to handle more than
> 4000 files. There has to be a problem with the locking mechanism, because
> the filepoller tries to get the the locks for the remaining files over and
> over again, but it just won't get them. But if you put another new files
> in,
> they will be handled correctly.
>
> We are currently working on this and try to patch the file-component. I
> will
> let you know when we find a solution that works for us.
>
> regards
> Char
>
>
>
> Gert Vanthienen wrote:
> >
> > L.S.,
> >
> >
> > What version of ServiceMix are you using? Prior to ServiceMix 3.3, the
> > file poller was using synchronous message exchanges which can more
> > easily lead to polling threads being blocked waiting for something else
> > to happen. You can figure this out by connecting to your ServiceMix
> > instance with a JMX console and looking at the stacks for all the
> > threads. If most of your file components threads are WAITING, just post
> > us the stacks so we can help you figure out what needs to change.
> >
> > Regards,
> >
> > Gert
> >
>
> --
> View this message in context:
> http://www.nabble.com/Filepoller-sometimes-does-not-work-tp20850432p20913626.html
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
>
--
http://lhein.blogspot.com
Re: Filepoller sometimes does not work
Posted by Char <ch...@gmx.net>.
Hello Gert,
we are using ServiceMix 3.3, and unfortunately our problem doesn't refer to
the archive-function. Nevertheless, patching the archive function solves
other problems...
So what we observe is that the filepoller has problems to handle more than
4000 files. There has to be a problem with the locking mechanism, because
the filepoller tries to get the the locks for the remaining files over and
over again, but it just won't get them. But if you put another new files in,
they will be handled correctly.
We are currently working on this and try to patch the file-component. I will
let you know when we find a solution that works for us.
regards
Char
Gert Vanthienen wrote:
>
> L.S.,
>
>
> What version of ServiceMix are you using? Prior to ServiceMix 3.3, the
> file poller was using synchronous message exchanges which can more
> easily lead to polling threads being blocked waiting for something else
> to happen. You can figure this out by connecting to your ServiceMix
> instance with a JMX console and looking at the stacks for all the
> threads. If most of your file components threads are WAITING, just post
> us the stacks so we can help you figure out what needs to change.
>
> Regards,
>
> Gert
>
--
View this message in context: http://www.nabble.com/Filepoller-sometimes-does-not-work-tp20850432p20913626.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: Filepoller sometimes does not work
Posted by Gert Vanthienen <ge...@skynet.be>.
L.S.,
What version of ServiceMix are you using? Prior to ServiceMix 3.3, the
file poller was using synchronous message exchanges which can more
easily lead to polling threads being blocked waiting for something else
to happen. You can figure this out by connecting to your ServiceMix
instance with a JMX console and looking at the stacks for all the
threads. If most of your file components threads are WAITING, just post
us the stacks so we can help you figure out what needs to change.
Regards,
Gert
Char wrote:
> Hello,
>
> we are using the jbi:file component to poll a huge amount of files. In a
> test szenario, i am deploying 2000 small files, which are polled correctly.
> When i deploy another 2000 files, a lot of them (300-500) seem to be ignored
> by the filepoller.
>
> Behaviour: You can still delete them, so there is no lock on them. If you
> deploy more files, the newer ones are being polled, whereas overwriting the
> sticky ones does not help.
>
> Test-Setup: 2000 files with different file names. 2nd deploy with same
> filenames.
>
> Configuration:
> <file:poller service="FilePoller"
> endpoint="FileIn"
> targetService="InboundRoutingService"
> targetEndpoint="FileInEndpoint"
> file="../Data/Inbound/FileIn"
> autoCreateDirectory="false"
> period="5000"
> archive="../Data/Inbound/Archive">
>
> <!-- Custom Marshaler, if you are not going to poll xml files.
> -->
> <property name="marshaler">
> <bean
> class="org.apache.servicemix.components.util.BinaryFileMarshaler" />
> </property>
> </file:poller>
>
> Any ideas how i can analyse this behaviour?
>
> thx in advance
>