You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by BlackTie <jt...@hotmail.com> on 2014/10/16 22:41:34 UTC
Not all processors are listed in JMX, preventing detailed route
statistics/profiling
I have a webapp using Camel (v2.13.2) about to go into production, at the
point of trying to tune things for performance. I'm having an issue with
getting detail statistics for some routes, though. I'm using Hawtio 1.2,
and for some routes, Hawtio is displaying the statistics for each processor
in the route, and others it only shows the statistics at the route level. I
checked JMX with jconsole and noticed that, for the routes that don't show
specifics for the processors, the processors in those routes are not listed
under the processors section in JMX. Some routes with this problem are
complex, others are fairly simple.
Any ideas what would cause some to/choice/multicast/etc processors from not
being listed in JMX at all? Without the statistics, it makes it very
difficult to figure out where things are slowing down when the system is
under heavy load.
The jmxAgent is enabled for the camel context, with statisticsLevel set to
All.
Thanks for any help you can provide on this problem!
--
View this message in context: http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Not all processors are listed in JMX, preventing detailed route
statistics/profiling
Posted by BlackTie <jt...@hotmail.com>.
I don't have a small project that reproduces the issue, just the large app
I'm performance-tuning now. I'm willing to try anything you recommend to
figure out exactly what's wrong. I have a feeling that if I created a
smaller project to mock this problem, I wouldn't be able to reproduce it. :)
The interesting thing to note is that all the routes work as expected, the
transactional error handler works to retry errors according to the policies
we have set. The only indication that something isn't right with these
transacted routes is that no statistics get recorded for the processors of
those routes in JMX.
Charles Moulliard-2 wrote
> Do you have a use case that we could use to reproduce the issue ?
--
View this message in context: http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634p5758000.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Not all processors are listed in JMX, preventing detailed route statistics/profiling
Posted by Charles Moulliard <ch...@gmail.com>.
Do you have a use case that we could use to reproduce the issue ?
On Thu, Oct 23, 2014 at 10:08 PM, BlackTie <jt...@hotmail.com> wrote:
> I tried configuring the JMX agent with registerAlways=true and that didn't
> make a difference.
>
> Still thinking about the transacted routes not being registered at the
> processor level in JMX, I removed the transacted method from the route
> builder for one of the routes that didn't have the detailed processor
> statistics and redeployed. Hawtio now shows processor statistics for that
> particular route.
>
> Something regarding or in conjunction with the transactions is stopping the
> processors from being registered by the agent.
>
> Can you think of any reason why transacting the routes would cause this JMX
> issue?
>
> Some information regarding our transaction config:
> * using Spring's JmsTransactionManager
> * using Spring's CachingConnectionFactory with the transaction manager
> * using Camel's SpringTransactionPolicy beans with the transaction manager
> * using ActiveMQ 5.10.0
> * ActiveMQ Camel component's JMS config has transacted=true
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634p5757959.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Charles Moulliard
Apache Committer / Architect @RedHat
Twitter : @cmoulliard | Blog : http://cmoulliard.github.io
Re: Not all processors are listed in JMX, preventing detailed route
statistics/profiling
Posted by BlackTie <jt...@hotmail.com>.
I tried configuring the JMX agent with registerAlways=true and that didn't
make a difference.
Still thinking about the transacted routes not being registered at the
processor level in JMX, I removed the transacted method from the route
builder for one of the routes that didn't have the detailed processor
statistics and redeployed. Hawtio now shows processor statistics for that
particular route.
Something regarding or in conjunction with the transactions is stopping the
processors from being registered by the agent.
Can you think of any reason why transacting the routes would cause this JMX
issue?
Some information regarding our transaction config:
* using Spring's JmsTransactionManager
* using Spring's CachingConnectionFactory with the transaction manager
* using Camel's SpringTransactionPolicy beans with the transaction manager
* using ActiveMQ 5.10.0
* ActiveMQ Camel component's JMS config has transacted=true
--
View this message in context: http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634p5757959.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Not all processors are listed in JMX, preventing detailed route statistics/profiling
Posted by Charles Moulliard <ch...@gmail.com>.
You could also use breakpoint within the method shouldRegister(Object
service, Route route) of DefaultManagementLifecycleStrategy class to check
why we return false
On Tue, Oct 21, 2014 at 8:24 AM, Charles Moulliard <ch...@gmail.com> wrote:
> Hi Blacktie,
>
> Can you enable this camel jmx property (registerAlways=true)to see if all
> the processors are registered and visible within JMX ?
>
> Regards,
>
> On Mon, Oct 20, 2014 at 7:36 PM, BlackTie <jt...@hotmail.com> wrote:
>
>> A mix of processors are missing... to, validate, multicast, log, etc...
>> it
>> seems like all the processors under some routes are missing. I haven't
>> been
>> able to figure out what the cause is.
>>
>> Can route syntax have an issue with statistics being gathered?
>>
>> Can transactions prevent statistics from being gathered? A lot of our
>> routes use JMS transactions so that we don't drop in-flight messages.
>>
>> I just noticed while looking in hawtio at the mbeans for Camel that there
>> are no transacted processors listed.
>>
>>
>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634p5757820.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>
>
>
> --
> Charles Moulliard
> Apache Committer / Architect @RedHat
> Twitter : @cmoulliard | Blog : http://cmoulliard.github.io
>
>
--
Charles Moulliard
Apache Committer / Architect @RedHat
Twitter : @cmoulliard | Blog : http://cmoulliard.github.io
Re: Not all processors are listed in JMX, preventing detailed route statistics/profiling
Posted by Charles Moulliard <ch...@gmail.com>.
Hi Blacktie,
Can you enable this camel jmx property (registerAlways=true)to see if all
the processors are registered and visible within JMX ?
Regards,
On Mon, Oct 20, 2014 at 7:36 PM, BlackTie <jt...@hotmail.com> wrote:
> A mix of processors are missing... to, validate, multicast, log, etc... it
> seems like all the processors under some routes are missing. I haven't
> been
> able to figure out what the cause is.
>
> Can route syntax have an issue with statistics being gathered?
>
> Can transactions prevent statistics from being gathered? A lot of our
> routes use JMS transactions so that we don't drop in-flight messages.
>
> I just noticed while looking in hawtio at the mbeans for Camel that there
> are no transacted processors listed.
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634p5757820.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Charles Moulliard
Apache Committer / Architect @RedHat
Twitter : @cmoulliard | Blog : http://cmoulliard.github.io
Re: Not all processors are listed in JMX, preventing detailed route
statistics/profiling
Posted by BlackTie <jt...@hotmail.com>.
A mix of processors are missing... to, validate, multicast, log, etc... it
seems like all the processors under some routes are missing. I haven't been
able to figure out what the cause is.
Can route syntax have an issue with statistics being gathered?
Can transactions prevent statistics from being gathered? A lot of our
routes use JMS transactions so that we don't drop in-flight messages.
I just noticed while looking in hawtio at the mbeans for Camel that there
are no transacted processors listed.
--
View this message in context: http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634p5757820.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Not all processors are listed in JMX, preventing detailed
route statistics/profiling
Posted by Willem Jiang <wi...@gmail.com>.
What kind of processors are missing in the JMX?
Can you give us an example (Camel route and processors) for it?
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On October 18, 2014 at 12:40:06 AM, BlackTie (jtstor@hotmail.com) wrote:
> I am using hawtio as a standalone WAR, and did some research about accessing
> jolokia through hawtio. Browsing through the info that jolokia retrieves
> from JMX, not all the processors are listed. I added the maxObjects query
> param (with a value of 5000, which is higher than the number of beans we
> have) and still, not all the processors are listed. Coupled with the fact
> that if you set maxObjects lower than the max number of processor objects
> that are available to be retrieved, the json returned contains "[Object
> limit exceeded]" after it hits the number specified in maxObjects.
>
> Not only are some of the mbeans missing, but for some routes that actually
> did display statistics, after a while the statistics for some processors
> stop being shown on the profiling tab in Hawtio. The statistics for the
> processors that have stopped being shown in Hawtio are still in JMX. Weird!
>
> I have used jconsole to look at the list of processors, and it looks the
> same as the list through hawtio. Some of the processors are missing, even
> on a fresh startup of my webapp.
>
> It sounds like I am running into two separate problems:
> * Some processors are never showing up in JMX at all
> * Hawtio stops showing statistics at some point for some processors, but I
> can still look at the statistics through JMX
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634p5757739.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Not all processors are listed in JMX, preventing detailed route statistics/profiling
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I have reproduced the issue and logged a ticket
https://issues.apache.org/jira/browse/CAMEL-7981
On Tue, Oct 28, 2014 at 10:35 PM, BlackTie <jt...@hotmail.com> wrote:
> Charles Moulliard-2 wrote
>> Can you debug the project and add a breakpoint in this method (protected
>> boolean shouldRegister(Object service, Route route) {)) of this class
>> (org.apache.camel.management.DefaultManagementLifecycleStrategy) of
>> camel-core to see if the conditions are matched to register the mbean ?
>
> I added a breakpoint in shouldRegister with the condition of "service
> instanceof SendProcessor", just to narrow down the search. I was looking
> for a specifically-named processor that is in a transacted route.
>
> On the first run with the debugger, the breakpoint was hit several times,
> but for other routes. It never hit the breakpoint for the processor in
> question.
>
> For a second attempt, I removed the transacted() call from the route, and
> ran with the debugger, and it hit the breakpoint with the processor I was
> looking for.
>
> For what it's worth, I also set a breakpoint in registerProcessor where
> getId() contained the name of the processor I was looking for, and
> regardless of the route being transacted, the breakpoint was hit.
>
> To sum up, registerProcessor gets called (and returns true) for all the
> processors, it doesn't matter if they're in transacted routes or not.
> shouldRegister doesn't get called for processors in transacted routes, but
> it does get called for processors in non-transacted routes.
>
> The debug logging from org.apache.camel.management supports this. I can
> comment out the transacted() call from a route and see all the processors
> get registered in the debug logging of DefaultManagementAgent. If I rerun
> with the route actually being transacted, those processors don't show up in
> the logging.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634p5758257.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: Not all processors are listed in JMX, preventing detailed route
statistics/profiling
Posted by BlackTie <jt...@hotmail.com>.
Charles Moulliard-2 wrote
> Can you debug the project and add a breakpoint in this method (protected
> boolean shouldRegister(Object service, Route route) {)) of this class
> (org.apache.camel.management.DefaultManagementLifecycleStrategy) of
> camel-core to see if the conditions are matched to register the mbean ?
I added a breakpoint in shouldRegister with the condition of "service
instanceof SendProcessor", just to narrow down the search. I was looking
for a specifically-named processor that is in a transacted route.
On the first run with the debugger, the breakpoint was hit several times,
but for other routes. It never hit the breakpoint for the processor in
question.
For a second attempt, I removed the transacted() call from the route, and
ran with the debugger, and it hit the breakpoint with the processor I was
looking for.
For what it's worth, I also set a breakpoint in registerProcessor where
getId() contained the name of the processor I was looking for, and
regardless of the route being transacted, the breakpoint was hit.
To sum up, registerProcessor gets called (and returns true) for all the
processors, it doesn't matter if they're in transacted routes or not.
shouldRegister doesn't get called for processors in transacted routes, but
it does get called for processors in non-transacted routes.
The debug logging from org.apache.camel.management supports this. I can
comment out the transacted() call from a route and see all the processors
get registered in the debug logging of DefaultManagementAgent. If I rerun
with the route actually being transacted, those processors don't show up in
the logging.
--
View this message in context: http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634p5758257.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Not all processors are listed in JMX, preventing detailed route statistics/profiling
Posted by Charles Moulliard <ch...@gmail.com>.
Can you debug the project and add a breakpoint in this method (protected
boolean shouldRegister(Object service, Route route) {)) of this class
(org.apache.camel.management.DefaultManagementLifecycleStrategy) of
camel-core to see if the conditions are matched to register the mbean ?
On Mon, Oct 27, 2014 at 11:24 PM, BlackTie <jt...@hotmail.com> wrote:
> Claus Ibsen-2 wrote
> > You can enable DEBUG logging for
> > org.apache.camel.management. And it should log when it enlists a
> > processor as MBean. Maybe you can see if the missing processors at
> > least is DEBUG logged.
>
> Claus,
>
> I enabled DEBUG for org.apache.camel.management as you requested. There is
> no debug logging output for any processors that belong to a route that is
> transacted. I didn't double check every single one, but from earlier
> attempts at figuring out where things were going wrong, it looks like none
> of the processors from transacted routes are registered in JMX, and no
> statistics are collected on them. If I remove the transaction from the
> route, then those processors get registered and I can see the statistics
> through hawtio (or jconsole).
>
> I've looked at the configuration of Camel and JMS and nothing looks out of
> the ordinary to me, but there seems to be something with transacted routes
> that prevent those processors from being registered.
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634p5758143.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Charles Moulliard
Apache Committer / Architect @RedHat
Twitter : @cmoulliard | Blog : http://cmoulliard.github.io
Re: Not all processors are listed in JMX, preventing detailed route
statistics/profiling
Posted by BlackTie <jt...@hotmail.com>.
Claus Ibsen-2 wrote
> You can enable DEBUG logging for
> org.apache.camel.management. And it should log when it enlists a
> processor as MBean. Maybe you can see if the missing processors at
> least is DEBUG logged.
Claus,
I enabled DEBUG for org.apache.camel.management as you requested. There is
no debug logging output for any processors that belong to a route that is
transacted. I didn't double check every single one, but from earlier
attempts at figuring out where things were going wrong, it looks like none
of the processors from transacted routes are registered in JMX, and no
statistics are collected on them. If I remove the transaction from the
route, then those processors get registered and I can see the statistics
through hawtio (or jconsole).
I've looked at the configuration of Camel and JMS and nothing looks out of
the ordinary to me, but there seems to be something with transacted routes
that prevent those processors from being registered.
--
View this message in context: http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634p5758143.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Not all processors are listed in JMX, preventing detailed route statistics/profiling
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Oct 17, 2014 at 6:39 PM, BlackTie <jt...@hotmail.com> wrote:
> I am using hawtio as a standalone WAR, and did some research about accessing
> jolokia through hawtio. Browsing through the info that jolokia retrieves
> from JMX, not all the processors are listed. I added the maxObjects query
> param (with a value of 5000, which is higher than the number of beans we
> have) and still, not all the processors are listed. Coupled with the fact
> that if you set maxObjects lower than the max number of processor objects
> that are available to be retrieved, the json returned contains "[Object
> limit exceeded]" after it hits the number specified in maxObjects.
>
I logged a ticket at hawtio to take a look at if we can optimize this
or have some initial load that has no limitations
https://github.com/hawtio/hawtio/issues/1698
> Not only are some of the mbeans missing, but for some routes that actually
> did display statistics, after a while the statistics for some processors
> stop being shown on the profiling tab in Hawtio. The statistics for the
> processors that have stopped being shown in Hawtio are still in JMX. Weird!
>
> I have used jconsole to look at the list of processors, and it looks the
> same as the list through hawtio. Some of the processors are missing, even
> on a fresh startup of my webapp.
>
That is odd if jconsole does not also list all the processors, then
hawtio can neither find them. You can enable DEBUG logging for
org.apache.camel.management. And it should log when it enlists a
processor as MBean. Maybe you can see if the missing processors at
least is DEBUG logged.
Also mind that if you turn on the option
onlyRegisterProcessorWithCustomId=true then only processors with a
custom id is enlisted. But that is not enabled by default.
> It sounds like I am running into two separate problems:
> * Some processors are never showing up in JMX at all
> * Hawtio stops showing statistics at some point for some processors, but I
> can still look at the statistics through JMX
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634p5757739.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: Not all processors are listed in JMX, preventing detailed route
statistics/profiling
Posted by BlackTie <jt...@hotmail.com>.
I am using hawtio as a standalone WAR, and did some research about accessing
jolokia through hawtio. Browsing through the info that jolokia retrieves
from JMX, not all the processors are listed. I added the maxObjects query
param (with a value of 5000, which is higher than the number of beans we
have) and still, not all the processors are listed. Coupled with the fact
that if you set maxObjects lower than the max number of processor objects
that are available to be retrieved, the json returned contains "[Object
limit exceeded]" after it hits the number specified in maxObjects.
Not only are some of the mbeans missing, but for some routes that actually
did display statistics, after a while the statistics for some processors
stop being shown on the profiling tab in Hawtio. The statistics for the
processors that have stopped being shown in Hawtio are still in JMX. Weird!
I have used jconsole to look at the list of processors, and it looks the
same as the list through hawtio. Some of the processors are missing, even
on a fresh startup of my webapp.
It sounds like I am running into two separate problems:
* Some processors are never showing up in JMX at all
* Hawtio stops showing statistics at some point for some processors, but I
can still look at the statistics through JMX
--
View this message in context: http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634p5757739.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Not all processors are listed in JMX, preventing detailed route statistics/profiling
Posted by Andrew Thorburn <nz...@gmail.com>.
I've seen issues like this in HawtIO, and it's actually a Jolokia issue.
Basically, Jolokia will only fetch the first N elements (unsorted, I
believe) from the server, and if your MBean isn't one of those N, then...
bugger?
The solution is to go into HawtIO / Preferences / Jolokia, and change Max
Collection Size from 500 (the default) up to a larger number. I went to
5000, and all of a sudden all my routes appeared in the Camel console, and
in the JMX tab. This will have an impact on browser performance, but I'm
not sure there's much that can be done, apart from raising a feature
request with Jolokia / HawtIO-Camel to fetch only a subset of available
MBeans, rather than fetching them all and filtering on the client-side.
Now, if you're looking at the JMX info through a different source (not
HawtIO/Jolokia) and still not seeing them all, then this probably won't
help much.
- Andrew
On Fri, Oct 17, 2014 at 7:41 AM, BlackTie <jt...@hotmail.com> wrote:
> I have a webapp using Camel (v2.13.2) about to go into production, at the
> point of trying to tune things for performance. I'm having an issue with
> getting detail statistics for some routes, though. I'm using Hawtio 1.2,
> and for some routes, Hawtio is displaying the statistics for each processor
> in the route, and others it only shows the statistics at the route level.
> I
> checked JMX with jconsole and noticed that, for the routes that don't show
> specifics for the processors, the processors in those routes are not listed
> under the processors section in JMX. Some routes with this problem are
> complex, others are fairly simple.
>
> Any ideas what would cause some to/choice/multicast/etc processors from not
> being listed in JMX at all? Without the statistics, it makes it very
> difficult to figure out where things are slowing down when the system is
> under heavy load.
>
> The jmxAgent is enabled for the camel context, with statisticsLevel set to
> All.
>
> Thanks for any help you can provide on this problem!
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Not-all-processors-are-listed-in-JMX-preventing-detailed-route-statistics-profiling-tp5757634.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>