You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by MatSM <ab...@gmail.com> on 2007/12/14 21:23:30 UTC
Seda Flow and ST flow
Hi,
Iam trying to understand the performance metrics for the above two flow.
Please clarify.
The scenario for the flow is :
1. Different files of with a few recs totalling to xml recs- 500 is dropped
in a Queue.In our case MQ.
2. That is read by Smix through JMS binding - The JBI components in Smix
does - splitting of xml, validation of xml(Node level) and calls a web
service running in Geronimo to insert recs into database.
The metrics
Appromixately 1.5 minutes for 480 recs inserted into db.. (Some exceptions
thrown, those recs not inserted into db) - Flow -SEDA
Appromixately 8.5 minutes for 480 recs inserted into db.. (Some exceptions
thrown, those recs not inserted into db) - Flow -STRAIGHT
Any explanation to the above poor performance for the ST flow.
Cheers
Mat
--
View this message in context: http://www.nabble.com/Seda-Flow-and-ST-flow-tp14339781s12049p14339781.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: Seda Flow and ST flow
Posted by MatSM <ab...@gmail.com>.
Thanks for the explanation and the link to SEDA flow.
So I assume, even the JMS flow with VM or JMS flow+ cluster + VM may not be
too efficient than the SEDA in the Smix ..cluster.
Mat
bsnyder wrote:
>
> On Dec 14, 2007 1:23 PM, MatSM <ab...@gmail.com> wrote:
>>
>> Hi,
>>
>> Iam trying to understand the performance metrics for the above two flow.
>> Please clarify.
>>
>> The scenario for the flow is :
>>
>> 1. Different files of with a few recs totalling to xml recs- 500 is
>> dropped
>> in a Queue.In our case MQ.
>>
>> 2. That is read by Smix through JMS binding - The JBI components in Smix
>> does - splitting of xml, validation of xml(Node level) and calls a web
>> service running in Geronimo to insert recs into database.
>>
>> The metrics
>> Appromixately 1.5 minutes for 480 recs inserted into db.. (Some
>> exceptions
>> thrown, those recs not inserted into db) - Flow -SEDA
>>
>> Appromixately 8.5 minutes for 480 recs inserted into db.. (Some
>> exceptions
>> thrown, those recs not inserted into db) - Flow -STRAIGHT
>>
>>
>> Any explanation to the above poor performance for the ST flow.
>
> This is straightforward difference between the straight through flow
> and the SEDA flow. The ST flow does no buffering whatsoever so it's
> not as efficient and doesn't scale very well. The SEDA flow is based
> on the concepts in the SEDA paper
> (http://www.eecs.harvard.edu/~mdw/proj/seda/) so it is designed to be
> more efficient and scale much further. This is why the SEDA flow is
> the default flow.
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.org/
> Apache Camel - http://activemq.org/camel/
> Apache ServiceMix - http://servicemix.org/
> Apache Geronimo - http://geronimo.apache.org/
>
> Blog: http://bruceblog.org/
>
>
--
View this message in context: http://www.nabble.com/Seda-Flow-and-ST-flow-tp14339781s12049p14420334.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: Seda Flow and ST flow
Posted by Bruce Snyder <br...@gmail.com>.
On Dec 19, 2007 10:57 AM, MatSM <ab...@gmail.com> wrote:
>
> Yes, internally, the JBI components we developed have implementation of
> marshalers for our requirements. These were developed and configured in the
> servicemix.xml was doing the processing of xml - viz validation, total
> validation ,converting to flat file sequentially.
>
> When we introduced threading in our code , the components / marshallers did
> a way better job.
That's why the marshaler API is there; to allow for extensions such as
these. Nice work :-).
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Apache Geronimo - http://geronimo.apache.org/
Blog: http://bruceblog.org/
Re: Seda Flow and ST flow
Posted by MatSM <ab...@gmail.com>.
Yes, internally, the JBI components we developed have implementation of
marshalers for our requirements. These were developed and configured in the
servicemix.xml was doing the processing of xml - viz validation, total
validation ,converting to flat file sequentially.
When we introduced threading in our code , the components / marshallers did
a way better job.
Thanks for the feedback
Mat
bsnyder wrote:
>
> On Dec 19, 2007 9:53 AM, MatSM <ab...@gmail.com> wrote:
>>
>> I did try playing with the thread pool size in the config file for
>> smix..
>> However, st flow doesn't scale better than SEDA.
>
> The ST flow is not going to be affected by adjusting the thread pool
> size. That will only affect the SEDA flow.
>
>> Our next option is to go with clustering of SMix with seda and JMS flow.
>>
>> We had to use threading within our application to increase the rate at
>> which
>> Smix does all the JBI components processing on our input files, which we
>> have developed.
>>
>> The difference, with threading in our application process , image+xml
>> file
>> to extend of 1000 rec /minute
>>
>> With out threading in our application , the process did close to 50 recs
>> /
>> minute.
>
> So how did you add this functionality? Did you create a marshaler for
> one of the ServiceMix components?
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.org/
> Apache Camel - http://activemq.org/camel/
> Apache ServiceMix - http://servicemix.org/
> Apache Geronimo - http://geronimo.apache.org/
>
> Blog: http://bruceblog.org/
>
>
--
View this message in context: http://www.nabble.com/Seda-Flow-and-ST-flow-tp14339781s12049p14421810.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: Seda Flow and ST flow
Posted by Bruce Snyder <br...@gmail.com>.
On Dec 19, 2007 9:53 AM, MatSM <ab...@gmail.com> wrote:
>
> I did try playing with the thread pool size in the config file for smix..
> However, st flow doesn't scale better than SEDA.
The ST flow is not going to be affected by adjusting the thread pool
size. That will only affect the SEDA flow.
> Our next option is to go with clustering of SMix with seda and JMS flow.
>
> We had to use threading within our application to increase the rate at which
> Smix does all the JBI components processing on our input files, which we
> have developed.
>
> The difference, with threading in our application process , image+xml file
> to extend of 1000 rec /minute
>
> With out threading in our application , the process did close to 50 recs /
> minute.
So how did you add this functionality? Did you create a marshaler for
one of the ServiceMix components?
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Apache Geronimo - http://geronimo.apache.org/
Blog: http://bruceblog.org/
Re: Seda Flow and ST flow
Posted by MatSM <ab...@gmail.com>.
I did try playing with the thread pool size in the config file for smix..
However, st flow doesn't scale better than SEDA.
Our next option is to go with clustering of SMix with seda and JMS flow.
We had to use threading within our application to increase the rate at which
Smix does all the JBI components processing on our input files, which we
have developed.
The difference, with threading in our application process , image+xml file
to extend of 1000 rec /minute
With out threading in our application , the process did close to 50 recs /
minute.
Thanks
Mat
gnodet wrote:
>
> In addition, I would also try playing with the thread pool size,
> especially
> if your CPU consumption is not maxed.
>
> On Dec 19, 2007 3:33 PM, Bruce Snyder <br...@gmail.com> wrote:
>
>> On Dec 14, 2007 1:23 PM, MatSM <ab...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > Iam trying to understand the performance metrics for the above two
>> flow.
>> > Please clarify.
>> >
>> > The scenario for the flow is :
>> >
>> > 1. Different files of with a few recs totalling to xml recs- 500 is
>> dropped
>> > in a Queue.In our case MQ.
>> >
>> > 2. That is read by Smix through JMS binding - The JBI components in
>> Smix
>> > does - splitting of xml, validation of xml(Node level) and calls a web
>> > service running in Geronimo to insert recs into database.
>> >
>> > The metrics
>> > Appromixately 1.5 minutes for 480 recs inserted into db.. (Some
>> exceptions
>> > thrown, those recs not inserted into db) - Flow -SEDA
>> >
>> > Appromixately 8.5 minutes for 480 recs inserted into db.. (Some
>> exceptions
>> > thrown, those recs not inserted into db) - Flow -STRAIGHT
>> >
>> >
>> > Any explanation to the above poor performance for the ST flow.
>>
>> This is straightforward difference between the straight through flow
>> and the SEDA flow. The ST flow does no buffering whatsoever so it's
>> not as efficient and doesn't scale very well. The SEDA flow is based
>> on the concepts in the SEDA paper
>> (http://www.eecs.harvard.edu/~mdw/proj/seda/<http://www.eecs.harvard.edu/%7Emdw/proj/seda/>)
>> so it is designed to be
>> more efficient and scale much further. This is why the SEDA flow is
>> the default flow.
>>
>> Bruce
>> --
>> perl -e 'print
>> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
>> );'
>>
>> Apache ActiveMQ - http://activemq.org/
>> Apache Camel - http://activemq.org/camel/
>> Apache ServiceMix - http://servicemix.org/
>> Apache Geronimo - http://geronimo.apache.org/
>>
>> Blog: http://bruceblog.org/
>>
>
>
>
> --
> Cheers,
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
>
>
--
View this message in context: http://www.nabble.com/Seda-Flow-and-ST-flow-tp14339781s12049p14420813.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: Seda Flow and ST flow
Posted by Guillaume Nodet <gn...@gmail.com>.
In addition, I would also try playing with the thread pool size, especially
if your CPU consumption is not maxed.
On Dec 19, 2007 3:33 PM, Bruce Snyder <br...@gmail.com> wrote:
> On Dec 14, 2007 1:23 PM, MatSM <ab...@gmail.com> wrote:
> >
> > Hi,
> >
> > Iam trying to understand the performance metrics for the above two flow.
> > Please clarify.
> >
> > The scenario for the flow is :
> >
> > 1. Different files of with a few recs totalling to xml recs- 500 is
> dropped
> > in a Queue.In our case MQ.
> >
> > 2. That is read by Smix through JMS binding - The JBI components in Smix
> > does - splitting of xml, validation of xml(Node level) and calls a web
> > service running in Geronimo to insert recs into database.
> >
> > The metrics
> > Appromixately 1.5 minutes for 480 recs inserted into db.. (Some
> exceptions
> > thrown, those recs not inserted into db) - Flow -SEDA
> >
> > Appromixately 8.5 minutes for 480 recs inserted into db.. (Some
> exceptions
> > thrown, those recs not inserted into db) - Flow -STRAIGHT
> >
> >
> > Any explanation to the above poor performance for the ST flow.
>
> This is straightforward difference between the straight through flow
> and the SEDA flow. The ST flow does no buffering whatsoever so it's
> not as efficient and doesn't scale very well. The SEDA flow is based
> on the concepts in the SEDA paper
> (http://www.eecs.harvard.edu/~mdw/proj/seda/<http://www.eecs.harvard.edu/%7Emdw/proj/seda/>)
> so it is designed to be
> more efficient and scale much further. This is why the SEDA flow is
> the default flow.
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.org/
> Apache Camel - http://activemq.org/camel/
> Apache ServiceMix - http://servicemix.org/
> Apache Geronimo - http://geronimo.apache.org/
>
> Blog: http://bruceblog.org/
>
--
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
Re: Seda Flow and ST flow
Posted by Bruce Snyder <br...@gmail.com>.
On Dec 14, 2007 1:23 PM, MatSM <ab...@gmail.com> wrote:
>
> Hi,
>
> Iam trying to understand the performance metrics for the above two flow.
> Please clarify.
>
> The scenario for the flow is :
>
> 1. Different files of with a few recs totalling to xml recs- 500 is dropped
> in a Queue.In our case MQ.
>
> 2. That is read by Smix through JMS binding - The JBI components in Smix
> does - splitting of xml, validation of xml(Node level) and calls a web
> service running in Geronimo to insert recs into database.
>
> The metrics
> Appromixately 1.5 minutes for 480 recs inserted into db.. (Some exceptions
> thrown, those recs not inserted into db) - Flow -SEDA
>
> Appromixately 8.5 minutes for 480 recs inserted into db.. (Some exceptions
> thrown, those recs not inserted into db) - Flow -STRAIGHT
>
>
> Any explanation to the above poor performance for the ST flow.
This is straightforward difference between the straight through flow
and the SEDA flow. The ST flow does no buffering whatsoever so it's
not as efficient and doesn't scale very well. The SEDA flow is based
on the concepts in the SEDA paper
(http://www.eecs.harvard.edu/~mdw/proj/seda/) so it is designed to be
more efficient and scale much further. This is why the SEDA flow is
the default flow.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Apache Geronimo - http://geronimo.apache.org/
Blog: http://bruceblog.org/