You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by Bill Simms <si...@gmail.com> on 2009/07/22 22:06:13 UTC
Synchronous exchanges with servicemix-vfs
Hello,
I have a project requirement that needs files to be polled in an ordered
fashion and synchronously from a remote directory. I've been looking at
using servicemix-vfs (for Servicemix 4.0) because it offers so many possible
file systems. However, the one problem I've had so far is that vfs will
always send message exchanges using the asynchronous call. The fact that
multiple threads are used at the same time has lead to vfs delivering files
so quickly that it overloads receiving services and eventually consumes all
of the JVM's heap memory. So my question is this: is there a way
(presently or in planning) to force vfs to wait until a message exchange is
done before it trys to send the next exchange? I've reviewed the
VFSPollingEndpoint and related classes and it seems almost trivial to set
the endpoint to use the sendSync() method in combination with the
synchronous-bypass options of the ExecutorImpl class. Would this be a good
approach, or is there a better way? Thanks.
--
View this message in context: http://www.nabble.com/Synchronous-exchanges-with-servicemix-vfs-tp24613615p24613615.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: Synchronous exchanges with servicemix-vfs
Posted by Lars Heinemann <lh...@apache.org>.
Bill,
the VFSPollingEndpoint extends from the PollingEndpoint class. This class has
a flag for disabling parallel polling so that only one poll cycle runs
at a time. This is
the case by default. So it won't happen, that multiple threads poll files.
Anyway the most elegant and clean way to solve your problem would be
synchronous sending. Please
create a JIRA issue that we introduce a switching possibility between
synchronous and asynchronous
sending. I will to solve this by time.
Regards
Lars
2009/7/23 Bill Simms <si...@gmail.com>:
>
> Thanks for the quick response Lars. Those are great suggestions.
> Unfortunately I don't think adding an additional queue will work to solve
> all of my problem. One of the key requirements for me is that files are
> polled from their directory in lexicographical order by file name. Doing
> this can be accomplished using a file selector, which I've already done.
> However, when there are multiple threads creating exchanges, it may be
> possible that the exchanges are delivered out of order. This may even be
> possible when using ordered queues. If at some point an exchange is sent
> from the VFSFilePollingEndpoint to the queue out of order, and the queue is
> polled before the actual first-in-order exchange is sent to the queue, it
> will appear as if the exchange in the queue is next in line when it is not.
> So adding a JMS queue may not help my case, when the file system is already
> being used as a queue. I realize that few people have the kind of
> requirements that I do, so I'll continue to investigate. Thanks again.
>
>
>
> lhein wrote:
>>
>> Hi Bill,
>>
>> currently we do only send async in the VFS poll endpoint. It would be
>> easy to add a flag which controls
>> the sending mode but here comes another approach:
>>
>> What if you queue the messages inside a JMS queue. If your files
>> aren't that big, that should work fine with only little performance
>> loss. Afterwards you could connect a number of consumers (or only 1)
>> to that queue which polls the messages synchronously from that queue.
>> Another solution...
>> Chris recently added a new possibility of throttling the
>> servicemix-file poller endpoint. I am not sure though since which
>> version it is in the servicemix-file component. Maybe you can try this
>> as well.
>>
>> Regards
>> Lars
>>
>>
>>
>> 2009/7/22 Bill Simms <si...@gmail.com>:
>>>
>>> Hello,
>>>
>>> I have a project requirement that needs files to be polled in an ordered
>>> fashion and synchronously from a remote directory. I've been looking at
>>> using servicemix-vfs (for Servicemix 4.0) because it offers so many
>>> possible
>>> file systems. However, the one problem I've had so far is that vfs will
>>> always send message exchanges using the asynchronous call. The fact that
>>> multiple threads are used at the same time has lead to vfs delivering
>>> files
>>> so quickly that it overloads receiving services and eventually consumes
>>> all
>>> of the JVM's heap memory. So my question is this: is there a way
>>> (presently or in planning) to force vfs to wait until a message exchange
>>> is
>>> done before it trys to send the next exchange? I've reviewed the
>>> VFSPollingEndpoint and related classes and it seems almost trivial to set
>>> the endpoint to use the sendSync() method in combination with the
>>> synchronous-bypass options of the ExecutorImpl class. Would this be a
>>> good
>>> approach, or is there a better way? Thanks.
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Synchronous-exchanges-with-servicemix-vfs-tp24613615p24613615.html
>>> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> http://lhein.blogspot.com
>>
>>
>> -----
>> Regards
>> Lars
>>
>>
>> http://lhein.blogspot.com
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Synchronous-exchanges-with-servicemix-vfs-tp24613615p24631547.html
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
>
--
http://lhein.blogspot.com
Re: Synchronous exchanges with servicemix-vfs
Posted by Bill Simms <si...@gmail.com>.
Thanks for the quick response Lars. Those are great suggestions.
Unfortunately I don't think adding an additional queue will work to solve
all of my problem. One of the key requirements for me is that files are
polled from their directory in lexicographical order by file name. Doing
this can be accomplished using a file selector, which I've already done.
However, when there are multiple threads creating exchanges, it may be
possible that the exchanges are delivered out of order. This may even be
possible when using ordered queues. If at some point an exchange is sent
from the VFSFilePollingEndpoint to the queue out of order, and the queue is
polled before the actual first-in-order exchange is sent to the queue, it
will appear as if the exchange in the queue is next in line when it is not.
So adding a JMS queue may not help my case, when the file system is already
being used as a queue. I realize that few people have the kind of
requirements that I do, so I'll continue to investigate. Thanks again.
lhein wrote:
>
> Hi Bill,
>
> currently we do only send async in the VFS poll endpoint. It would be
> easy to add a flag which controls
> the sending mode but here comes another approach:
>
> What if you queue the messages inside a JMS queue. If your files
> aren't that big, that should work fine with only little performance
> loss. Afterwards you could connect a number of consumers (or only 1)
> to that queue which polls the messages synchronously from that queue.
> Another solution...
> Chris recently added a new possibility of throttling the
> servicemix-file poller endpoint. I am not sure though since which
> version it is in the servicemix-file component. Maybe you can try this
> as well.
>
> Regards
> Lars
>
>
>
> 2009/7/22 Bill Simms <si...@gmail.com>:
>>
>> Hello,
>>
>> I have a project requirement that needs files to be polled in an ordered
>> fashion and synchronously from a remote directory. I've been looking at
>> using servicemix-vfs (for Servicemix 4.0) because it offers so many
>> possible
>> file systems. However, the one problem I've had so far is that vfs will
>> always send message exchanges using the asynchronous call. The fact that
>> multiple threads are used at the same time has lead to vfs delivering
>> files
>> so quickly that it overloads receiving services and eventually consumes
>> all
>> of the JVM's heap memory. So my question is this: is there a way
>> (presently or in planning) to force vfs to wait until a message exchange
>> is
>> done before it trys to send the next exchange? I've reviewed the
>> VFSPollingEndpoint and related classes and it seems almost trivial to set
>> the endpoint to use the sendSync() method in combination with the
>> synchronous-bypass options of the ExecutorImpl class. Would this be a
>> good
>> approach, or is there a better way? Thanks.
>> --
>> View this message in context:
>> http://www.nabble.com/Synchronous-exchanges-with-servicemix-vfs-tp24613615p24613615.html
>> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> http://lhein.blogspot.com
>
>
> -----
> Regards
> Lars
>
>
> http://lhein.blogspot.com
>
>
--
View this message in context: http://www.nabble.com/Synchronous-exchanges-with-servicemix-vfs-tp24613615p24631547.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: Synchronous exchanges with servicemix-vfs
Posted by Lars Heinemann <lh...@apache.org>.
Hi Bill,
currently we do only send async in the VFS poll endpoint. It would be
easy to add a flag which controls
the sending mode but here comes another approach:
What if you queue the messages inside a JMS queue. If your files
aren't that big, that should work fine with only little performance
loss. Afterwards you could connect a number of consumers (or only 1)
to that queue which polls the messages synchronously from that queue.
Another solution...
Chris recently added a new possibility of throttling the
servicemix-file poller endpoint. I am not sure though since which
version it is in the servicemix-file component. Maybe you can try this
as well.
Regards
Lars
2009/7/22 Bill Simms <si...@gmail.com>:
>
> Hello,
>
> I have a project requirement that needs files to be polled in an ordered
> fashion and synchronously from a remote directory. I've been looking at
> using servicemix-vfs (for Servicemix 4.0) because it offers so many possible
> file systems. However, the one problem I've had so far is that vfs will
> always send message exchanges using the asynchronous call. The fact that
> multiple threads are used at the same time has lead to vfs delivering files
> so quickly that it overloads receiving services and eventually consumes all
> of the JVM's heap memory. So my question is this: is there a way
> (presently or in planning) to force vfs to wait until a message exchange is
> done before it trys to send the next exchange? I've reviewed the
> VFSPollingEndpoint and related classes and it seems almost trivial to set
> the endpoint to use the sendSync() method in combination with the
> synchronous-bypass options of the ExecutorImpl class. Would this be a good
> approach, or is there a better way? Thanks.
> --
> View this message in context: http://www.nabble.com/Synchronous-exchanges-with-servicemix-vfs-tp24613615p24613615.html
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
>
--
http://lhein.blogspot.com