You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@camel.apache.org by Jean Francois LE BESCONT <jf...@gmail.com> on 2013/03/12 09:22:58 UTC

doTry docatch slow down the execution

Hey !

I have found a strange thing with the doTry / doCatch / end.

I have a seda queue in from with binding and processing files of 450 000
lines. I am procecessing it like that :

from("seda:AEDConcurrent?concurrentConsumers=3")
.threads(3)
.enrich("direct:referentielZone" , aggregationStrategyAEDReferentielZone)
.split()
.tokenize("\n")
.streaming()
.doTry()
.unmarshal()
.bindy(BindyType.Csv, AED.class)
.to("mock:mock")
.doCatch(Exception.class)
.to("direct:AEDFormatException")
.throwException(new AEDException.AEDFormatException( ))
.end() // end try catch
.end() // split
.end();

With this try catch my time process is 38 seconds per 15000 lines ( and
this time increase with time) whereas if I don't do try catch the same
execution is 3 seconds per 15000 lines.

Is there something that I do wrong ?

Thanks !

JF

Re: doTry docatch slow down the execution

Posted by Christian Müller <ch...@gmail.com>.

Have a look at the following tests:
https://github.com/muellerc/apachecon-na-2013/tree/master/file-component/src/test/java/org/apache/cmueller/camel/apachecon/na2013

The are performance comparison between the different possibilities...

Best,
Christian

On Thu, Mar 14, 2013 at 12:53 AM, Raul Kripalani <ra...@evosent.com> wrote:

> So you append each invalid record to a file (direct:AEDFormatException).
>
> For *each* rejected line, this process occurs: the FileInputStream is
> opened, positioned at the end, the line is written, the
> FileInputStream is closed. That could explain some of the slowness.
>
> Let's gather some performance stats for this endpoint. You have 2 options:
>
> - JMX => within the "processors" subtree you'll find a SendProcessor
> MBean associated with this file endpoint. Check its attributes.
> - Performance stats processors before and after the endpoint: 1)
> before: place a processor that saves System.currentTimeMillis() in an
> Exchange property, and 2) after: print to a log file the difference
> between now and the saved property.
>
> Please share your findings.
>
> Raúl Kripalani
> Apache Camel Committer | Enterprise Architect,
> Program Manager, Open Source Integration specialist
> http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani | http://blog.raulkr.net |
> twitter: @raulvk
>



--

Re: doTry docatch slow down the execution

Posted by Raul Kripalani <ra...@evosent.com>.

So you append each invalid record to a file (direct:AEDFormatException).

For *each* rejected line, this process occurs: the FileInputStream is
opened, positioned at the end, the line is written, the
FileInputStream is closed. That could explain some of the slowness.

Let's gather some performance stats for this endpoint. You have 2 options:

- JMX => within the "processors" subtree you'll find a SendProcessor
MBean associated with this file endpoint. Check its attributes.
- Performance stats processors before and after the endpoint: 1)
before: place a processor that saves System.currentTimeMillis() in an
Exchange property, and 2) after: print to a log file the difference
between now and the saved property.

Please share your findings.

Raúl Kripalani
Apache Camel Committer | Enterprise Architect,
Program Manager, Open Source Integration specialist
http://about.me/raulkripalani |
http://www.linkedin.com/in/raulkripalani | http://blog.raulkr.net |
twitter: @raulvk

Re: doTry docatch slow down the execution

Posted by Jean Francois LE BESCONT <jf...@gmail.com>.

Thanks Raul !

I will have lot of different IllegalArgumentException ( one per file type)
 each has to be processed differently, that is which I trap it by my self.

This is my main route :
        from("direct:AEDRoute")
                .id("AEDRoute")
                .startupOrder(1)

                .setProperty("OutputFileName"  ,
simple(propertiesKeys.AED_OUTPUT_DIR+"${headers.CamelFileNameOnly}"))

                .to("seda:AEDConcurrent")

                .end();

I manage each file in concurrence :
        // Route per file
        from("seda:AEDConcurrent?concurrentConsumers=3")
                .threads(3)
                .id("AEDConcurrent")
                .setProperty("OutputFileName"  ,
simple(propertiesKeys.AED_OUTPUT_DIR+"${headers.CamelFileNameOnly}"))
                // Enrichissement des référentiels
                .enrich("direct:referentielZone" ,
aggregationStrategyAEDReferentielZone)
                .split()
                    .tokenize("\n")
                    .streaming()
                        .doTry()
                .unmarshal()
                .bindy(BindyType.Csv, AED.class)
                .to("mock:mock")
                .doCatch(Exception.class)
                .to("direct:AEDFormatException")
                .throwException(new AEDException.AEDFormatException( ))
                .end() // end try catch

                        .process(eADProcessor)

                        .choice()

.when(header(AEDConstantes.HEADER_REJECT_ZONE_INCONNU).isEqualTo(new
Boolean(true)))
                                .to("direct:REJECT_AED_ZONE_INCONNU")

.when(header(AEDConstantes.HEADER_REJECT_JURIDICTION_INCONNU).isEqualTo(new
Boolean(true)))
                                .to("direct:REJECT_AED_JURIDICTION_INCONNU")
                            .otherwise()
                                .to("direct:AED_CONCURRENT_VALIDATE")
                            .end() // end otherwise
                        .end() // end choice
                    .end() // split

                .end();

Exception are trapped :
onException(AEDException.AEDFormatException.class)
.handled(true)
.end();

My exception personnal management :
        from("direct:AEDFormatException")
.id("erreurAEDInconnu")
.setBody(simple("${body} - ${exception.message}"))
.convertBodyTo(String.class)
.to("file://" + propertiesKeys.AED_OUTPUT_DIR +
"?fileName=${file:name}_format.csv&fileExist=Append")
.log("Erreur validation technique ligne '${body}' - ${exception.message} ")
        .end();


Thanks !


JF

Re: doTry docatch slow down the execution

Posted by Raul Kripalani <ra...@evosent.com>.

Ok.

Obviously there will be more overhead in capturing the exception and
re-throwing under a different exception type, but it should not make the
route 12x slower. Something's going on.

Do you have any additional error handlers? onExceptions, errorHandlers,
etc.?

Could you paste your entire configuration? I'd like to see your Camel
Context configuration.

And please paste exactly the version of the route that leads to these
results. Seeing a direct:AEDFormatException call when it didn't exist in
the benchmark misleads us ;)

Thanks,

*Raúl Kripalani*
Apache Camel Committer
Enterprise Architect, Program Manager, Open Source Integration specialist
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk <http://twitter.com/raulvk>


On Tue, Mar 12, 2013 at 8:47 AM, Jean Francois LE BESCONT <
jflebescont@gmail.com> wrote:

> Yes Raul , I opted for your suggestion,and it works fine ! Yes I put it in
> a property and acces it later ( has you suggest :)
>
> OH I ommited to say that my file contains only valid line, the
> direct:AEDFormatException
> which is a reject route is never call for the moment.
>
>
>
>
>
> 2013/3/12 Raul Kripalani <ra...@evosent.com>
>
> > Btw - It looks like you decided to go with the approach I suggested.
> First
> > enrich, then split.
> >
> > Is the aggregation strategy storing the SQL results in a property, which
> > you access later?
> > On 12 Mar 2013 08:23, "Jean Francois LE BESCONT" <jf...@gmail.com>
> > wrote:
> >
> > > Hey !
> > >
> > > I have found a strange thing with the doTry / doCatch / end.
> > >
> > > I have a seda queue in from with binding and processing files of 450
> 000
> > > lines. I am procecessing it like that :
> > >
> > > from("seda:AEDConcurrent?concurrentConsumers=3")
> > > .threads(3)
> > > .enrich("direct:referentielZone" ,
> aggregationStrategyAEDReferentielZone)
> > > .split()
> > > .tokenize("\n")
> > > .streaming()
> > > .doTry()
> > > .unmarshal()
> > > .bindy(BindyType.Csv, AED.class)
> > > .to("mock:mock")
> > > .doCatch(Exception.class)
> > > .to("direct:AEDFormatException")
> > > .throwException(new AEDException.AEDFormatException( ))
> > > .end() // end try catch
> > > .end() // split
> > > .end();
> > >
> > > With this try catch my time process is 38 seconds per 15000 lines ( and
> > > this time increase with time) whereas if I don't do try catch the same
> > > execution is 3 seconds per 15000 lines.
> > >
> > > Is there something that I do wrong ?
> > >
> > > Thanks !
> > >
> > > JF
> > >
> >
>

Re: doTry docatch slow down the execution

Posted by Jean Francois LE BESCONT <jf...@gmail.com>.

Yes Raul , I opted for your suggestion,and it works fine ! Yes I put it in
a property and acces it later ( has you suggest :)

OH I ommited to say that my file contains only valid line, the
direct:AEDFormatException
which is a reject route is never call for the moment.





2013/3/12 Raul Kripalani <ra...@evosent.com>

> Btw - It looks like you decided to go with the approach I suggested. First
> enrich, then split.
>
> Is the aggregation strategy storing the SQL results in a property, which
> you access later?
> On 12 Mar 2013 08:23, "Jean Francois LE BESCONT" <jf...@gmail.com>
> wrote:
>
> > Hey !
> >
> > I have found a strange thing with the doTry / doCatch / end.
> >
> > I have a seda queue in from with binding and processing files of 450 000
> > lines. I am procecessing it like that :
> >
> > from("seda:AEDConcurrent?concurrentConsumers=3")
> > .threads(3)
> > .enrich("direct:referentielZone" , aggregationStrategyAEDReferentielZone)
> > .split()
> > .tokenize("\n")
> > .streaming()
> > .doTry()
> > .unmarshal()
> > .bindy(BindyType.Csv, AED.class)
> > .to("mock:mock")
> > .doCatch(Exception.class)
> > .to("direct:AEDFormatException")
> > .throwException(new AEDException.AEDFormatException( ))
> > .end() // end try catch
> > .end() // split
> > .end();
> >
> > With this try catch my time process is 38 seconds per 15000 lines ( and
> > this time increase with time) whereas if I don't do try catch the same
> > execution is 3 seconds per 15000 lines.
> >
> > Is there something that I do wrong ?
> >
> > Thanks !
> >
> > JF
> >
>

Re: doTry docatch slow down the execution

Posted by Raul Kripalani <ra...@evosent.com>.

Btw - It looks like you decided to go with the approach I suggested. First
enrich, then split.

Is the aggregation strategy storing the SQL results in a property, which
you access later?
On 12 Mar 2013 08:23, "Jean Francois LE BESCONT" <jf...@gmail.com>
wrote:

> Hey !
>
> I have found a strange thing with the doTry / doCatch / end.
>
> I have a seda queue in from with binding and processing files of 450 000
> lines. I am procecessing it like that :
>
> from("seda:AEDConcurrent?concurrentConsumers=3")
> .threads(3)
> .enrich("direct:referentielZone" , aggregationStrategyAEDReferentielZone)
> .split()
> .tokenize("\n")
> .streaming()
> .doTry()
> .unmarshal()
> .bindy(BindyType.Csv, AED.class)
> .to("mock:mock")
> .doCatch(Exception.class)
> .to("direct:AEDFormatException")
> .throwException(new AEDException.AEDFormatException( ))
> .end() // end try catch
> .end() // split
> .end();
>
> With this try catch my time process is 38 seconds per 15000 lines ( and
> this time increase with time) whereas if I don't do try catch the same
> execution is 3 seconds per 15000 lines.
>
> Is there something that I do wrong ?
>
> Thanks !
>
> JF
>

Re: doTry docatch slow down the execution

Posted by Raul Kripalani <ra...@evosent.com>.

In your test, do exceptions happen? How many inside the 15.000-line file?

What does the direct:AEDFormatException route do? Please post its code. It
could be the reason for slowness.

Thanks.
On 12 Mar 2013 08:23, "Jean Francois LE BESCONT" <jf...@gmail.com>
wrote:

> Hey !
>
> I have found a strange thing with the doTry / doCatch / end.
>
> I have a seda queue in from with binding and processing files of 450 000
> lines. I am procecessing it like that :
>
> from("seda:AEDConcurrent?concurrentConsumers=3")
> .threads(3)
> .enrich("direct:referentielZone" , aggregationStrategyAEDReferentielZone)
> .split()
> .tokenize("\n")
> .streaming()
> .doTry()
> .unmarshal()
> .bindy(BindyType.Csv, AED.class)
> .to("mock:mock")
> .doCatch(Exception.class)
> .to("direct:AEDFormatException")
> .throwException(new AEDException.AEDFormatException( ))
> .end() // end try catch
> .end() // split
> .end();
>
> With this try catch my time process is 38 seconds per 15000 lines ( and
> this time increase with time) whereas if I don't do try catch the same
> execution is 3 seconds per 15000 lines.
>
> Is there something that I do wrong ?
>
> Thanks !
>
> JF
>