You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by ebinsingh <eb...@VerizonWireless.com> on 2011/10/26 21:52:01 UTC

Spliter in Camel

Hi,

I have a file of 300,000 records and i use the split mechanism of Camel to
split them and sends each record to a processer. 
Does Camel store these records on a heap or somewhere before it sends them
to the processer. How does Camel splitter internally work.

I want to make sure that the Splitter does not eat up the memory.

Please provide some insight into this.

Thanks & regards,
Ebe

--
View this message in context: http://camel.465427.n5.nabble.com/Spliter-in-Camel-tp4940967p4940967.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Spliter in Camel

Posted by Christian Schneider <ch...@die-schneider.net>.
Hi Ebe,

the aggregator as you defined it has no notion of completeness of a 
file. It can only aggregate using the correlation expression and the 
aggregationstrategy. It will send out
the aggregated exchange as soon as either the timeout is reached or the 
completionsize is reached. So when you parse a small file the timeout 
will trigger
when you parse a large file the completionsize will trigger. I guess 
what you want is to recombine the messages from the whole file again 
into one exchange.
I just read on the splitter docs that the last exchange is marked with 
the property "CamelSplitComplete" true. So this could help to know when 
the new exchange is complete.
I think you should be able to use the completionPredicate:

.aggregate(header("CamelFileName"), new MyAggregationStrategy()).completionPredicate(property("CamelSplitComplete"))

So the aggregator knows when the file is finished. (Not sure if my snippet is correct .. just typed it in the mail).


Your implementation with the builder will not speed up things as you 
convert to a string in the end. It should rather look like this:

	@Override
	public Exchange aggregate(Exchange exchange1, Exchange exchange2) {
	 if (exchange1 == null) {
		exchange1 = new DefaultExchange();
		exchange1.getIn().setBody(new StringBuilder());
	 }
         builder = exchange1.getIn().getBody(StringBuilder.class);
         builder.append(exchange2.getIn().getBody(String.class));
         return exchange1;
	}

So the idea is that on the first time you create a new exchange with a 
StringBuilder in the body. Then you append each message body to this 
StringBuilder.

Christian



Am 28.10.2011 20:56, schrieb ebinsingh:
> Thanks a lot Christian.
>
> I just noticed that adding an aggregater slows down the process and it's
> huge.
> Without the aggregation, the time taken between parsing each line of data is
> in nanoseconds (very negligible), but as I add the aggregarion to it, the
> time taken by between parsing each line of data goes up to 30 milliseconds.
>
> Also if I do not have the below executer statement, the process terminates
> in the middle of iterating through the file.
> ".executorService(threadPool)"
>
>
> Not sure if I have something totally wrong. Please advise.
>
> 	
> from("file:C:\\camelProject\\data\\inbox?fileName=someFile.txt&delete=true")
> 	        .log("Starting to process big file: ${header.CamelFileName}")
> 	        .split(body().tokenize("\n")).streaming()
> 	        .executorService(threadPool)
> 	        .bean(MyParser.class,"parseString")
> 	        .aggregate(header("CamelFileName"), new
> MyAggregationStrategy()).completionSize(2000)
> 	        .to("jms:queue:test_request_3");
>
> As suggested I am using StringBuilder in my aggregation statergy.
>
> 	StringBuilder builder = new StringBuilder();
> 	@Override
> 	public Exchange aggregate(Exchange exchange1, Exchange exchange2) {
>          if (exchange1 == null) {
>              return exchange2;
>          }
>
> builder.append(exchange1.getIn().getBody(String.class)).append(exchange2.getIn().getBody(String.class));
>          exchange1.getIn().setBody(builder.toString());
>          return exchange1;
> 	}
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spliter-in-Camel-tp4940967p4946971.html
> Sent from the Camel - Users mailing list archive at Nabble.com.


-- 

Christian Schneider
http://www.liquid-reality.de

Open Source Architect
Talend Application Integration Division http://www.talend.com


Re: Spliter in Camel

Posted by ebinsingh <eb...@VerizonWireless.com>.
Thanks a lot Christian.

I just noticed that adding an aggregater slows down the process and it's
huge.
Without the aggregation, the time taken between parsing each line of data is
in nanoseconds (very negligible), but as I add the aggregarion to it, the
time taken by between parsing each line of data goes up to 30 milliseconds.

Also if I do not have the below executer statement, the process terminates
in the middle of iterating through the file.
".executorService(threadPool)"


Not sure if I have something totally wrong. Please advise.

	       
from("file:C:\\camelProject\\data\\inbox?fileName=someFile.txt&delete=true")
	        .log("Starting to process big file: ${header.CamelFileName}")
	        .split(body().tokenize("\n")).streaming()
	        .executorService(threadPool)
	        .bean(MyParser.class,"parseString")
	        .aggregate(header("CamelFileName"), new
MyAggregationStrategy()).completionSize(2000)
	        .to("jms:queue:test_request_3");

As suggested I am using StringBuilder in my aggregation statergy.

	StringBuilder builder = new StringBuilder();
	@Override
	public Exchange aggregate(Exchange exchange1, Exchange exchange2) {
        if (exchange1 == null) {
            return exchange2;
        }
       
builder.append(exchange1.getIn().getBody(String.class)).append(exchange2.getIn().getBody(String.class));
        exchange1.getIn().setBody(builder.toString());
        return exchange1;
	}


--
View this message in context: http://camel.465427.n5.nabble.com/Spliter-in-Camel-tp4940967p4946971.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Spliter in Camel

Posted by Christian Schneider <ch...@die-schneider.net>.
I think you have two issues in your route:
- in the aggregator you concatenate strings. That is really slow. Use a stringbuilder as the body and append to it
- the aggregator waits for either 2000 messages or the default timeout which is 1s if I remember correctly

So for your small example the timeout is the issue. For larger ones the strings will be.

Christian

Am 27.10.2011 um 15:16 schrieb ebinsingh <eb...@VerizonWireless.com>:

> Thanks a lot for your help. 
> Have another memory related question.
> 
> The scenario I have is that, I get a file with n number of records, I split
> them using \n and stream them to a bean. I guess there would be a single
> instance of the bean for each thread (I am using an Executer) in the memory.
> I then pass the output from the bean (Which is also a string) to an
> aggregater to concatinate them into buckets of 2000 using an
> AggregaterStratergy. 
> The performance I see is very slow.
> 
> Please let me know your thoughts on the below code.
> Appreciate your help.
> 
>            from("file:/input?fileName=someFile.txt&delete=true")
>            .log("Starting to process big file: ${header.CamelFileName}")
>           
> .split(body().tokenize("\n")).streaming().executorService(threadPool)
>           
> .bean(MyParser.class,"parseString").aggregate(header("CamelFileName"), new
> MyAggregationStrategy()).completionSize(2000)
>            .executorService(threadPool1).to("jms:queue:test_request_3").end()
>            .log("Sent all messages to queue");
> 
> 
> The AggregationStratery does the below concatination.
> 
> exchange1.getIn().setBody(exchange1.getIn().getBody(String.class)+exchange2.getIn().getBody(String.class));
> 
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spliter-in-Camel-tp4940967p4942950.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Spliter in Camel

Posted by ebinsingh <eb...@VerizonWireless.com>.
I just tried to run this to process a file with 2 records and aggregate them
to a single record and send it to the MQ.
It's taking more than 1 sec to process this data whic is very slow.

--
View this message in context: http://camel.465427.n5.nabble.com/Spliter-in-Camel-tp4940967p4943073.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Spliter in Camel

Posted by ebinsingh <eb...@VerizonWireless.com>.
Thanks a lot for your help. 
Have another memory related question.

The scenario I have is that, I get a file with n number of records, I split
them using \n and stream them to a bean. I guess there would be a single
instance of the bean for each thread (I am using an Executer) in the memory.
I then pass the output from the bean (Which is also a string) to an
aggregater to concatinate them into buckets of 2000 using an
AggregaterStratergy. 
The performance I see is very slow.

Please let me know your thoughts on the below code.
Appreciate your help.

	        from("file:/input?fileName=someFile.txt&delete=true")
	        .log("Starting to process big file: ${header.CamelFileName}")
	       
.split(body().tokenize("\n")).streaming().executorService(threadPool)
	       
.bean(MyParser.class,"parseString").aggregate(header("CamelFileName"), new
MyAggregationStrategy()).completionSize(2000)
        	.executorService(threadPool1).to("jms:queue:test_request_3").end()
            .log("Sent all messages to queue");


The AggregationStratery does the below concatination.

exchange1.getIn().setBody(exchange1.getIn().getBody(String.class)+exchange2.getIn().getBody(String.class));

--
View this message in context: http://camel.465427.n5.nabble.com/Spliter-in-Camel-tp4940967p4942950.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Spliter in Camel

Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Oct 26, 2011 at 9:52 PM, ebinsingh
<eb...@verizonwireless.com> wrote:
> Hi,
>
> I have a file of 300,000 records and i use the split mechanism of Camel to
> split them and sends each record to a processer.
> Does Camel store these records on a heap or somewhere before it sends them
> to the processer. How does Camel splitter internally work.
>
> I want to make sure that the Splitter does not eat up the memory.
>

The splitter supports a .streaming() option that causes it to split it
in chunks.
That requires thought the source message can support this.
http://camel.apache.org/splitter

For example to split a CSV file on a line by line basis in streaming
mode, will not load
the entire file into memory, so you can split very big files this way.

XPath as Christian said could potential cause the message to be loaded
into memory.
Since we recent have optimized XQuery and XSLT, we are looking into
optimize XPath as well if that is possible.


> Please provide some insight into this.
>
> Thanks & regards,
> Ebe
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spliter-in-Camel-tp4940967p4940967.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

Re: Spliter in Camel

Posted by Christian Müller <ch...@gmail.com>.
It depends... ;-)
If you use XPath to split the file, the entire XML document is loaded into
memory (that's the way how XPath works :-( ).
If you use a line tokenizer (e.g. \n) to split the message, you can chose to
stream the file.

[1] http://camel.apache.org/splitter.html

Best,
Christian

On Wed, Oct 26, 2011 at 9:52 PM, ebinsingh <
ebenezer.singh@verizonwireless.com> wrote:

> Hi,
>
> I have a file of 300,000 records and i use the split mechanism of Camel to
> split them and sends each record to a processer.
> Does Camel store these records on a heap or somewhere before it sends them
> to the processer. How does Camel splitter internally work.
>
> I want to make sure that the Splitter does not eat up the memory.
>
> Please provide some insight into this.
>
> Thanks & regards,
> Ebe
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Spliter-in-Camel-tp4940967p4940967.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

Re: Spliter in Camel

Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Nov 1, 2011 at 1:37 PM, ebinsingh
<eb...@verizonwireless.com> wrote:
> Ah ok. I was trying to get some performance statistics and was using a single
> thread.
> Any tips on increasing performance when using Camel (I would use thread
> pool).
>
> My task would be to read a file (csv or xml), spilt them into single
> records, process these records, aggregate 2000 records into 1 and publish it
> to MQ.
>
> I currently see that it takes little less than a second to process a file of
> 2000 records.
>

It only makes sense to parallel process splitted messages if you do
some CPU intensive work on the message, and/or enrich
with data from an external resource (get data from a DB, WebService
call etc.) to get some CPU and IO bound usages.
Then multiple threads can better scale as it can process messages
which waiting for IO to reply etc.

If all you do is to assemble 2000 records, then that would be faster
in a single threaded.





> Thanks & regards,
> Ebe
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spliter-in-Camel-tp4940967p4955063.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

Re: Spliter in Camel

Posted by ebinsingh <eb...@VerizonWireless.com>.
Ah ok. I was trying to get some performance statistics and was using a single
thread.
Any tips on increasing performance when using Camel (I would use thread
pool).

My task would be to read a file (csv or xml), spilt them into single
records, process these records, aggregate 2000 records into 1 and publish it
to MQ.

I currently see that it takes little less than a second to process a file of
2000 records.

Thanks & regards,
Ebe

--
View this message in context: http://camel.465427.n5.nabble.com/Spliter-in-Camel-tp4940967p4955063.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Spliter in Camel

Posted by Claus Ibsen <cl...@gmail.com>.
Are you using a thread pool ?
Its the JDK which controls and executes the tasks and thus its up to
how the JDK handles this.


On Mon, Oct 31, 2011 at 7:09 PM, ebinsingh
<eb...@verizonwireless.com> wrote:
> Thanks a lot all of you.
>
> As you may see the code in above in this thread, I am splitting based on the
> new line token and sending it to a bean.
>
> I noticed som gap in the processing times. I 5 messages beeing processed in
> a single millisecond and there is a gap of 15 milliseconds before the next 5
> message's are sent to the bean  .. this pattern repeats itself.
>
> Any reason for this time delay. I went through the Splitter and streaming
> and do not see any specific property i could set for continous streaming.
>
> 2011-10-31 13:42:43,874 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
> bean invocation result on the IN message: <Some message>
> 2011-10-31 13:42:43,874 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
> bean invocation result on the IN message: <Some message>
> 2011-10-31 13:42:43,*874* DEBUG [pool-1-thread-1] bean.BeanProcessor -
> Setting bean invocation result on the IN message: <Some message>
> 2011-10-31 13:42:43,*890* DEBUG [pool-1-thread-1] bean.BeanProcessor -
> Setting bean invocation result on the IN message: <Some message>
> 2011-10-31 13:42:43,890 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
> bean invocation result on the IN message: <Some message>
> 2011-10-31 13:42:43,890 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
> bean invocation result on the IN message: <Some message>
> 2011-10-31 13:42:43,890 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
> bean invocation result on the IN message: <Some message>
> 2011-10-31 13:42:43,890 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
> bean invocation result on the IN message: <Some message>
> 2011-10-31 13:42:43,890 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
> bean invocation result on the IN message: <Some message>
> 2011-10-31 13:42:43,890 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
> bean invocation result on the IN message: <Some message>
> 2011-10-31 13:42:43,890 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
> bean invocation result on the IN message: <Some message>
> 2011-10-31 13:42:43,*890* DEBUG [pool-1-thread-1] bean.BeanProcessor -
> Setting bean invocation result on the IN message: <Some message>
> 2011-10-31 13:42:43,*905* DEBUG [pool-1-thread-1] bean.BeanProcessor -
> Setting bean invocation result on the IN message: <Some message>
> 2011-10-31 13:42:43,905 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
> bean invocation result on the IN message: <Some message>
> 2011-10-31 13:42:43,905 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
> bean invocation result on the IN message: <Some message>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spliter-in-Camel-tp4940967p4953111.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

Re: Spliter in Camel

Posted by ebinsingh <eb...@VerizonWireless.com>.
Thanks a lot all of you.

As you may see the code in above in this thread, I am splitting based on the
new line token and sending it to a bean.

I noticed som gap in the processing times. I 5 messages beeing processed in
a single millisecond and there is a gap of 15 milliseconds before the next 5
message's are sent to the bean  .. this pattern repeats itself.

Any reason for this time delay. I went through the Splitter and streaming
and do not see any specific property i could set for continous streaming.

2011-10-31 13:42:43,874 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
bean invocation result on the IN message: <Some message>
2011-10-31 13:42:43,874 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
bean invocation result on the IN message: <Some message>
2011-10-31 13:42:43,*874* DEBUG [pool-1-thread-1] bean.BeanProcessor -
Setting bean invocation result on the IN message: <Some message>
2011-10-31 13:42:43,*890* DEBUG [pool-1-thread-1] bean.BeanProcessor -
Setting bean invocation result on the IN message: <Some message>
2011-10-31 13:42:43,890 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
bean invocation result on the IN message: <Some message>
2011-10-31 13:42:43,890 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
bean invocation result on the IN message: <Some message>
2011-10-31 13:42:43,890 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
bean invocation result on the IN message: <Some message>
2011-10-31 13:42:43,890 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
bean invocation result on the IN message: <Some message>
2011-10-31 13:42:43,890 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
bean invocation result on the IN message: <Some message>
2011-10-31 13:42:43,890 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
bean invocation result on the IN message: <Some message>
2011-10-31 13:42:43,890 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
bean invocation result on the IN message: <Some message>
2011-10-31 13:42:43,*890* DEBUG [pool-1-thread-1] bean.BeanProcessor -
Setting bean invocation result on the IN message: <Some message>
2011-10-31 13:42:43,*905* DEBUG [pool-1-thread-1] bean.BeanProcessor -
Setting bean invocation result on the IN message: <Some message>
2011-10-31 13:42:43,905 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
bean invocation result on the IN message: <Some message>
2011-10-31 13:42:43,905 DEBUG [pool-1-thread-1] bean.BeanProcessor - Setting
bean invocation result on the IN message: <Some message>

--
View this message in context: http://camel.465427.n5.nabble.com/Spliter-in-Camel-tp4940967p4953111.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Spliter in Camel

Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Oct 26, 2011 at 9:52 PM, ebinsingh
<eb...@verizonwireless.com> wrote:
> Hi,
>
> I have a file of 300,000 records and i use the split mechanism of Camel to
> split them and sends each record to a processer.
> Does Camel store these records on a heap or somewhere before it sends them
> to the processer. How does Camel splitter internally work.
>
> I want to make sure that the Splitter does not eat up the memory.
>

The splitter by default will build up a list, but you can enable
streaming mode, so it uses a iterator.
http://camel.apache.org/splitter.html

Also the splitte can aggregate the messages back again, so you do not
need to use a separate aggregator.
Instead just supply a aggregation strategy to the splitter.
( like a composed message processor EIP -
http://camel.apache.org/composed-message-processor.html, however I can
see the wiki page hasn't been updated to shown this )

Camel in Action book chapter 8 and 10 covers this in much more details.



> Please provide some insight into this.
>
> Thanks & regards,
> Ebe
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spliter-in-Camel-tp4940967p4940967.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/