Large file processing using ExecutorService and IBM MQ

[dwarak_cit <dw...@gmail.com>]

Hi,
I have a requirement where I need to read quite a lot of large files. For
each file, store each record by record (separated by new line delimiter)
into the database. Now I have used ExecutorService with a fixed thread pool
of 100, in Camel split component to split the file (say has 20,000 records)
and send 200 messages per thread to IBM MQ Websphere. 

Now I created another processor which reads these messages from the IBM's
Message Queue. Here I have configured as
/*<bean id="websphere" class="org.apache.camel.component.jms.JmsComponent">
<property name="concurrentConsumers" value="200" />
<property name="maxConcurrentConsumers" value="200" />
<property name="taskExecutor" ref="taskExecutor" />
* /

1) Does this mean there are 200 threads to consume 200 message?
2) Need some more information on ExecutorService and Concurrent Consumers.

Also, this configuration would invoke 200 parallel message consumption. But
how much ever i increase my thread size and concurrent consumers, my
processing speed is slower than a stand alone program which processes that
file spawning multiple threads.

Thanks!!



--
View this message in context: http://camel.465427.n5.nabble.com/Large-file-processing-using-ExecutorService-and-IBM-MQ-tp5757222.html
Sent from the Camel Development mailing list archive at Nabble.com.

Re: Large file processing using ExecutorService and IBM MQ

[Claus Ibsen <cl...@gmail.com>]

Hi

Please use the user mailing list / user forum for these kind of questions.
http://camel.apache.org/discussion-forums.html

On Tue, Sep 30, 2014 at 5:10 AM, dwarak_cit <dw...@gmail.com> wrote:
> Hi,
> I have a requirement where I need to read quite a lot of large files. For
> each file, store each record by record (separated by new line delimiter)
> into the database. Now I have used ExecutorService with a fixed thread pool
> of 100, in Camel split component to split the file (say has 20,000 records)
> and send 200 messages per thread to IBM MQ Websphere.
>
> Now I created another processor which reads these messages from the IBM's
> Message Queue. Here I have configured as
> /*<bean id="websphere" class="org.apache.camel.component.jms.JmsComponent">
> <property name="concurrentConsumers" value="200" />
> <property name="maxConcurrentConsumers" value="200" />
> <property name="taskExecutor" ref="taskExecutor" />
> * /
>
> 1) Does this mean there are 200 threads to consume 200 message?
> 2) Need some more information on ExecutorService and Concurrent Consumers.
>
> Also, this configuration would invoke 200 parallel message consumption. But
> how much ever i increase my thread size and concurrent consumers, my
> processing speed is slower than a stand alone program which processes that
> file spawning multiple threads.
>
> Thanks!!
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Large-file-processing-using-ExecutorService-and-IBM-MQ-tp5757222.html
> Sent from the Camel Development mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/