You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Denis DEBARBIEUX <dd...@norsys.fr> on 2016/03/18 17:07:53 UTC
Problem when consuming storm metrics
Dear all,
I have a question about metrics and how they are consumed:
I launch my topology. During 3 or 4 minutes, I receive my metrics. Then
no more logs.
It's look like the process that is consuming the metrics has a dead lock
or is blocked by a time out.
Here comes some information about the context:
- I am using storm 0.9.4.
- Some metrics are defined by defaut (eg emit-count.default) some
other are customed.
- I am sure that my topology is still running.
- According to Storm UI, the number of messages emiting by my bolts is
increasing. In other words, metrics are still computing and send somewhere.
- I try two instances of LoggingMetricsConsumer: in file
(backtype.storm.metric.LoggingMetricsConsumer) and in graphite
(com.verisign.storm.metrics.GraphiteMetricsConsumer).
- I can reproduce it on 2 different machines.
- No error are logged (worker, metric, supervisor).
- I am using storm since 1 year and I never seen this problem.
Thanks for your help.
Denis
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Re: Problem when consuming storm metrics
Posted by Denis DEBARBIEUX <dd...@norsys.fr>.
Dear all,
I hope that this issue is now fixed.
I rewrote all my code related to metrics and now it is working.
- All my code is now based on ReducedMetric
- I removed all my static code.
Thanks a lot for your patience and your help.
Denis
On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX" <ddebarbieux@norsys.fr
<ma...@norsys.fr>> wrote:
Dear all,
I have a question about metrics and how they are consumed:
I launch my topology. During 3 or 4 minutes, I receive my metrics.
Then no more logs.
It's look like the process that is consuming the metrics has a dead
lock or is blocked by a time out.
Here comes some information about the context:
- I am using storm 0.9.4.
- Some metrics are defined by defaut (eg emit-count.default) some
other are customed.
- I am sure that my topology is still running.
- According to Storm UI, the number of messages emiting by my
bolts is increasing. In other words, metrics are still computing and
send somewhere.
- I try two instances of LoggingMetricsConsumer: in file
(backtype.storm.metric.LoggingMetricsConsumer) and in graphite
(com.verisign.storm.metrics.GraphiteMetricsConsumer).
- I can reproduce it on 2 different machines.
- No error are logged (worker, metric, supervisor).
- I am using storm since 1 year and I never seen this problem.
Thanks for your help.
Denis
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Re: Problem when consuming storm metrics
Posted by Denis DEBARBIEUX <dd...@norsys.fr>.
Dear all,
Please find enclose the full thread dump. I hope that you can find some
promishing information.
I did the following experiments:
- Reproduce the problem with the ExclamationTopology (storm starter).
But I fail.
- Activate acking (there is no acking in my toplogy). But I still have
the problem.
Sorry for the delay between my two posts.
Thanks for our help.
Denis
Le 11/04/2016 13:47, Abhishek Agarwal a écrit :
> I wanted to look at bolt stack specifically. With the names removed,
> it is difficult to figure out. can you tell me what are the
> bolts/spouts busy in? Are they very slow? The timer which publishes
> metric tick is blocked because the receive queue of some executor is
> full.
>
> Some related details are captured here -
> https://issues.apache.org/jira/browse/STORM-972
>
> On Mon, Apr 11, 2016 at 2:28 PM, Denis DEBARBIEUX
> <ddebarbieux@norsys.fr <ma...@norsys.fr>> wrote:
>
> Jungtaek,
>
> Thanks for your help.
>
> Please find enclose the jstack file:
> - I remove by hand all name related to my project (spout, bolt...).
> - The running jdk was openJDK (not an oracle JDK).
>
> I also increase the buffer size
> (topology.executor.receive.buffer.size and
> topology.executor.send.buffer.size) to 32768 and lauch my
> topology on 2 workers. Then the dead lock occurs after 3 days. At
> this moment 24577 points have been send.
>
> Please feel free to contact me if you need further information.
>
> Denis
>
> Le 29/03/2016 11:11, Jungtaek Lim a écrit :
>> Denis,
>>
>> Could you share your jstack output which contains locked information?
>> And I'm not sure but you may hit STORM-1027
>> <https://issues.apache.org/jira/browse/STORM-1027> which is
>> resolved at 0.10.0 and 0.9.6.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>> 2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX
>> <ddebarbieux@norsys.fr <ma...@norsys.fr>> 님이 작성:
>>
>> Hi all,
>>
>> I profile my code and I see that the thread
>> <BoldName>MetricConsumer is locked and never unlocked. It is
>> why my metrics are never published.
>>
>> More over, the problem is related to buffer size:
>> * topology.executor.receive.buffer.size=16 and
>> topology.executor.send.buffer.size=16 implies a blocking
>> after 4 minutes.
>> * with a size=512, I am able to run my test for more than 1 hour.
>>
>>
>> Thanks for your help.
>>
>> Denis
>>
>> Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>>>
>>> Do you get this problem if you use only the
>>> LoggingMetricsConsumer?
>>>
>>> Excuse typos
>>>
>>> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX"
>>> <ddebarbieux@norsys.fr <ma...@norsys.fr>> wrote:
>>>
>>> Dear all,
>>>
>>> I have a question about metrics and how they are consumed:
>>> I launch my topology. During 3 or 4 minutes, I receive
>>> my metrics. Then no more logs.
>>> It's look like the process that is consuming the metrics
>>> has a dead lock or is blocked by a time out.
>>>
>>> Here comes some information about the context:
>>> - I am using storm 0.9.4.
>>> - Some metrics are defined by defaut (eg
>>> emit-count.default) some other are customed.
>>> - I am sure that my topology is still running.
>>> - According to Storm UI, the number of messages emiting
>>> by my bolts is increasing. In other words, metrics are
>>> still computing and send somewhere.
>>> - I try two instances of LoggingMetricsConsumer: in
>>> file (backtype.storm.metric.LoggingMetricsConsumer) and
>>> in graphite
>>> (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>>> - I can reproduce it on 2 different machines.
>>> - No error are logged (worker, metric, supervisor).
>>> - I am using storm since 1 year and I never seen this
>>> problem.
>>>
>>> Thanks for your help.
>>>
>>> Denis
>>>
>>> ---
>>> L'absence de virus dans ce courrier électronique a été
>>> vérifiée par le logiciel antivirus Avast.
>>> https://www.avast.com/antivirus
>>>
>>
>>
>>
>> ------------------------------------------------------------------------
>> Avast logo
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>
>>
>> L'absence de virus dans ce courrier électronique a été
>> vérifiée par le logiciel antivirus Avast.
>>
>> www.avast.com <http://www.avast.com>
>>
>>
>
>
>
> ------------------------------------------------------------------------
> Avast logo
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
> L'absence de virus dans ce courrier électronique a été vérifiée
> par le logiciel antivirus Avast.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
>
>
>
> --
> Regards,
> Abhishek Agarwal
>
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Re: Problem when consuming storm metrics
Posted by Abhishek Agarwal <ab...@gmail.com>.
I wanted to look at bolt stack specifically. With the names removed, it is
difficult to figure out. can you tell me what are the bolts/spouts busy in?
Are they very slow? The timer which publishes metric tick is blocked
because the receive queue of some executor is full.
Some related details are captured here -
https://issues.apache.org/jira/browse/STORM-972
On Mon, Apr 11, 2016 at 2:28 PM, Denis DEBARBIEUX <dd...@norsys.fr>
wrote:
> Jungtaek,
>
> Thanks for your help.
>
> Please find enclose the jstack file:
> - I remove by hand all name related to my project (spout, bolt...).
> - The running jdk was openJDK (not an oracle JDK).
>
> I also increase the buffer size (topology.executor.receive.buffer.size and
> topology.executor.send.buffer.size) to 32768 and lauch my topology on 2
> workers. Then the dead lock occurs after 3 days. At this moment 24577
> points have been send.
>
> Please feel free to contact me if you need further information.
>
> Denis
>
> Le 29/03/2016 11:11, Jungtaek Lim a écrit :
>
> Denis,
>
> Could you share your jstack output which contains locked information?
> And I'm not sure but you may hit STORM-1027
> <https://issues.apache.org/jira/browse/STORM-1027> which is resolved at
> 0.10.0 and 0.9.6.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX < <dd...@norsys.fr>
> ddebarbieux@norsys.fr> 님이 작성:
>
>> Hi all,
>>
>> I profile my code and I see that the thread <BoldName>MetricConsumer is
>> locked and never unlocked. It is why my metrics are never published.
>>
>> More over, the problem is related to buffer size:
>> * topology.executor.receive.buffer.size=16 and
>> topology.executor.send.buffer.size=16 implies a blocking after 4 minutes.
>> * with a size=512, I am able to run my test for more than 1 hour.
>>
>>
>> Thanks for your help.
>>
>> Denis
>>
>> Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>>
>> Do you get this problem if you use only the LoggingMetricsConsumer?
>>
>> Excuse typos
>> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX" <dd...@norsys.fr>
>> wrote:
>>
>>> Dear all,
>>>
>>> I have a question about metrics and how they are consumed:
>>> I launch my topology. During 3 or 4 minutes, I receive my metrics. Then
>>> no more logs.
>>> It's look like the process that is consuming the metrics has a dead lock
>>> or is blocked by a time out.
>>>
>>> Here comes some information about the context:
>>> - I am using storm 0.9.4.
>>> - Some metrics are defined by defaut (eg emit-count.default) some other
>>> are customed.
>>> - I am sure that my topology is still running.
>>> - According to Storm UI, the number of messages emiting by my bolts is
>>> increasing. In other words, metrics are still computing and send somewhere.
>>> - I try two instances of LoggingMetricsConsumer: in file
>>> (backtype.storm.metric.LoggingMetricsConsumer) and in graphite
>>> (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>>> - I can reproduce it on 2 different machines.
>>> - No error are logged (worker, metric, supervisor).
>>> - I am using storm since 1 year and I never seen this problem.
>>>
>>> Thanks for your help.
>>>
>>> Denis
>>>
>>> ---
>>> L'absence de virus dans ce courrier électronique a été vérifiée par le
>>> logiciel antivirus Avast.
>>> https://www.avast.com/antivirus
>>>
>>>
>>
>>
>> ------------------------------
>> [image: Avast logo]
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>
>> L'absence de virus dans ce courrier électronique a été vérifiée par le
>> logiciel antivirus Avast.
>>
>>
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>> www.avast.com
>>
>>
>
>
> ------------------------------
> [image: Avast logo]
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
--
Regards,
Abhishek Agarwal
Re: Problem when consuming storm metrics
Posted by Denis DEBARBIEUX <dd...@norsys.fr>.
Jungtaek,
Thanks for your help.
Please find enclose the jstack file:
- I remove by hand all name related to my project (spout, bolt...).
- The running jdk was openJDK (not an oracle JDK).
I also increase the buffer size (topology.executor.receive.buffer.size
and topology.executor.send.buffer.size) to 32768 and lauch my topology
on 2 workers. Then the dead lock occurs after 3 days. At this moment
24577 points have been send.
Please feel free to contact me if you need further information.
Denis
Le 29/03/2016 11:11, Jungtaek Lim a écrit :
> Denis,
>
> Could you share your jstack output which contains locked information?
> And I'm not sure but you may hit STORM-1027
> <https://issues.apache.org/jira/browse/STORM-1027> which is resolved
> at 0.10.0 and 0.9.6.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX
> <ddebarbieux@norsys.fr <ma...@norsys.fr>> 님이 작성:
>
> Hi all,
>
> I profile my code and I see that the thread
> <BoldName>MetricConsumer is locked and never unlocked. It is why
> my metrics are never published.
>
> More over, the problem is related to buffer size:
> * topology.executor.receive.buffer.size=16 and
> topology.executor.send.buffer.size=16 implies a blocking after 4
> minutes.
> * with a size=512, I am able to run my test for more than 1 hour.
>
>
> Thanks for your help.
>
> Denis
>
> Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>>
>> Do you get this problem if you use only the LoggingMetricsConsumer?
>>
>> Excuse typos
>>
>> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX"
>> <ddebarbieux@norsys.fr <ma...@norsys.fr>> wrote:
>>
>> Dear all,
>>
>> I have a question about metrics and how they are consumed:
>> I launch my topology. During 3 or 4 minutes, I receive my
>> metrics. Then no more logs.
>> It's look like the process that is consuming the metrics has
>> a dead lock or is blocked by a time out.
>>
>> Here comes some information about the context:
>> - I am using storm 0.9.4.
>> - Some metrics are defined by defaut (eg emit-count.default)
>> some other are customed.
>> - I am sure that my topology is still running.
>> - According to Storm UI, the number of messages emiting by
>> my bolts is increasing. In other words, metrics are still
>> computing and send somewhere.
>> - I try two instances of LoggingMetricsConsumer: in file
>> (backtype.storm.metric.LoggingMetricsConsumer) and in
>> graphite (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>> - I can reproduce it on 2 different machines.
>> - No error are logged (worker, metric, supervisor).
>> - I am using storm since 1 year and I never seen this problem.
>>
>> Thanks for your help.
>>
>> Denis
>>
>> ---
>> L'absence de virus dans ce courrier électronique a été
>> vérifiée par le logiciel antivirus Avast.
>> https://www.avast.com/antivirus
>>
>
>
>
> ------------------------------------------------------------------------
> Avast logo
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
> L'absence de virus dans ce courrier électronique a été vérifiée
> par le logiciel antivirus Avast.
>
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Re: Problem when consuming storm metrics
Posted by Denis DEBARBIEUX <dd...@norsys.fr>.
Hi,
Thanks for you help.
I did many tests with version 0.9.6 but I did not find any improvement.
Denis
Le 29/03/2016 11:11, Jungtaek Lim a écrit :
> Denis,
>
> Could you share your jstack output which contains locked information?
> And I'm not sure but you may hit STORM-1027
> <https://issues.apache.org/jira/browse/STORM-1027> which is resolved
> at 0.10.0 and 0.9.6.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX
> <ddebarbieux@norsys.fr <ma...@norsys.fr>> 님이 작성:
>
> Hi all,
>
> I profile my code and I see that the thread
> <BoldName>MetricConsumer is locked and never unlocked. It is why
> my metrics are never published.
>
> More over, the problem is related to buffer size:
> * topology.executor.receive.buffer.size=16 and
> topology.executor.send.buffer.size=16 implies a blocking after 4
> minutes.
> * with a size=512, I am able to run my test for more than 1 hour.
>
>
> Thanks for your help.
>
> Denis
>
> Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>>
>> Do you get this problem if you use only the LoggingMetricsConsumer?
>>
>> Excuse typos
>>
>> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX"
>> <ddebarbieux@norsys.fr <ma...@norsys.fr>> wrote:
>>
>> Dear all,
>>
>> I have a question about metrics and how they are consumed:
>> I launch my topology. During 3 or 4 minutes, I receive my
>> metrics. Then no more logs.
>> It's look like the process that is consuming the metrics has
>> a dead lock or is blocked by a time out.
>>
>> Here comes some information about the context:
>> - I am using storm 0.9.4.
>> - Some metrics are defined by defaut (eg emit-count.default)
>> some other are customed.
>> - I am sure that my topology is still running.
>> - According to Storm UI, the number of messages emiting by
>> my bolts is increasing. In other words, metrics are still
>> computing and send somewhere.
>> - I try two instances of LoggingMetricsConsumer: in file
>> (backtype.storm.metric.LoggingMetricsConsumer) and in
>> graphite (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>> - I can reproduce it on 2 different machines.
>> - No error are logged (worker, metric, supervisor).
>> - I am using storm since 1 year and I never seen this problem.
>>
>> Thanks for your help.
>>
>> Denis
>>
>> ---
>> L'absence de virus dans ce courrier électronique a été
>> vérifiée par le logiciel antivirus Avast.
>> https://www.avast.com/antivirus
>>
>
>
>
> ------------------------------------------------------------------------
> Avast logo
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
> L'absence de virus dans ce courrier électronique a été vérifiée
> par le logiciel antivirus Avast.
>
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Re: Problem when consuming storm metrics
Posted by Jungtaek Lim <ka...@gmail.com>.
Denis,
Could you share your jstack output which contains locked information?
And I'm not sure but you may hit STORM-1027
<https://issues.apache.org/jira/browse/STORM-1027> which is resolved at
0.10.0 and 0.9.6.
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX <dd...@norsys.fr>님이 작성:
> Hi all,
>
> I profile my code and I see that the thread <BoldName>MetricConsumer is
> locked and never unlocked. It is why my metrics are never published.
>
> More over, the problem is related to buffer size:
> * topology.executor.receive.buffer.size=16 and
> topology.executor.send.buffer.size=16 implies a blocking after 4 minutes.
> * with a size=512, I am able to run my test for more than 1 hour.
>
>
> Thanks for your help.
>
> Denis
>
> Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>
> Do you get this problem if you use only the LoggingMetricsConsumer?
>
> Excuse typos
> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX" <dd...@norsys.fr> wrote:
>
>> Dear all,
>>
>> I have a question about metrics and how they are consumed:
>> I launch my topology. During 3 or 4 minutes, I receive my metrics. Then
>> no more logs.
>> It's look like the process that is consuming the metrics has a dead lock
>> or is blocked by a time out.
>>
>> Here comes some information about the context:
>> - I am using storm 0.9.4.
>> - Some metrics are defined by defaut (eg emit-count.default) some other
>> are customed.
>> - I am sure that my topology is still running.
>> - According to Storm UI, the number of messages emiting by my bolts is
>> increasing. In other words, metrics are still computing and send somewhere.
>> - I try two instances of LoggingMetricsConsumer: in file
>> (backtype.storm.metric.LoggingMetricsConsumer) and in graphite
>> (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>> - I can reproduce it on 2 different machines.
>> - No error are logged (worker, metric, supervisor).
>> - I am using storm since 1 year and I never seen this problem.
>>
>> Thanks for your help.
>>
>> Denis
>>
>> ---
>> L'absence de virus dans ce courrier électronique a été vérifiée par le
>> logiciel antivirus Avast.
>> https://www.avast.com/antivirus
>>
>>
>
>
> ------------------------------
> [image: Avast logo]
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
>
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
Re: Problem when consuming storm metrics
Posted by Denis DEBARBIEUX <dd...@norsys.fr>.
Hi all,
I profile my code and I see that the thread <BoldName>MetricConsumer is
locked and never unlocked. It is why my metrics are never published.
More over, the problem is related to buffer size:
* topology.executor.receive.buffer.size=16 and
topology.executor.send.buffer.size=16 implies a blocking after 4 minutes.
* with a size=512, I am able to run my test for more than 1 hour.
Thanks for your help.
Denis
Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>
> Do you get this problem if you use only the LoggingMetricsConsumer?
>
> Excuse typos
>
> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX" <ddebarbieux@norsys.fr
> <ma...@norsys.fr>> wrote:
>
> Dear all,
>
> I have a question about metrics and how they are consumed:
> I launch my topology. During 3 or 4 minutes, I receive my metrics.
> Then no more logs.
> It's look like the process that is consuming the metrics has a
> dead lock or is blocked by a time out.
>
> Here comes some information about the context:
> - I am using storm 0.9.4.
> - Some metrics are defined by defaut (eg emit-count.default) some
> other are customed.
> - I am sure that my topology is still running.
> - According to Storm UI, the number of messages emiting by my
> bolts is increasing. In other words, metrics are still computing
> and send somewhere.
> - I try two instances of LoggingMetricsConsumer: in file
> (backtype.storm.metric.LoggingMetricsConsumer) and in graphite
> (com.verisign.storm.metrics.GraphiteMetricsConsumer).
> - I can reproduce it on 2 different machines.
> - No error are logged (worker, metric, supervisor).
> - I am using storm since 1 year and I never seen this problem.
>
> Thanks for your help.
>
> Denis
>
> ---
> L'absence de virus dans ce courrier électronique a été vérifiée
> par le logiciel antivirus Avast.
> https://www.avast.com/antivirus
>
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Re: RE : Problem when consuming storm metrics
Posted by Abhishek Agarwal <ab...@gmail.com>.
It would be a good idea to take thread dump when logging stops. That could
give some clues
Excuse typos
On Mar 19, 2016 9:33 PM, "Denis DEBARBIEUX" <dd...@norsys.fr> wrote:
> Yes, I do.
>
> Denis
> ------------------------------
> *De :* Abhishek Agarwal [abhishcool@gmail.com]
> *Date d'envoi :* vendredi 18 mars 2016 22:38
> *À :* user@storm.apache.org
> *Objet :* Re: Problem when consuming storm metrics
>
> Do you get this problem if you use only the LoggingMetricsConsumer?
>
> Excuse typos
> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX" <dd...@norsys.fr> wrote:
>
>> Dear all,
>>
>> I have a question about metrics and how they are consumed:
>> I launch my topology. During 3 or 4 minutes, I receive my metrics. Then
>> no more logs.
>> It's look like the process that is consuming the metrics has a dead lock
>> or is blocked by a time out.
>>
>> Here comes some information about the context:
>> - I am using storm 0.9.4.
>> - Some metrics are defined by defaut (eg emit-count.default) some other
>> are customed.
>> - I am sure that my topology is still running.
>> - According to Storm UI, the number of messages emiting by my bolts is
>> increasing. In other words, metrics are still computing and send somewhere.
>> - I try two instances of LoggingMetricsConsumer: in file
>> (backtype.storm.metric.LoggingMetricsConsumer) and in graphite
>> (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>> - I can reproduce it on 2 different machines.
>> - No error are logged (worker, metric, supervisor).
>> - I am using storm since 1 year and I never seen this problem.
>>
>> Thanks for your help.
>>
>> Denis
>>
>> ---
>> L'absence de virus dans ce courrier électronique a été vérifiée par le
>> logiciel antivirus Avast.
>> https://www.avast.com/antivirus
>>
>>
Re: Problem when consuming storm metrics
Posted by Abhishek Agarwal <ab...@gmail.com>.
Do you get this problem if you use only the LoggingMetricsConsumer?
Excuse typos
On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX" <dd...@norsys.fr> wrote:
> Dear all,
>
> I have a question about metrics and how they are consumed:
> I launch my topology. During 3 or 4 minutes, I receive my metrics. Then no
> more logs.
> It's look like the process that is consuming the metrics has a dead lock
> or is blocked by a time out.
>
> Here comes some information about the context:
> - I am using storm 0.9.4.
> - Some metrics are defined by defaut (eg emit-count.default) some other
> are customed.
> - I am sure that my topology is still running.
> - According to Storm UI, the number of messages emiting by my bolts is
> increasing. In other words, metrics are still computing and send somewhere.
> - I try two instances of LoggingMetricsConsumer: in file
> (backtype.storm.metric.LoggingMetricsConsumer) and in graphite
> (com.verisign.storm.metrics.GraphiteMetricsConsumer).
> - I can reproduce it on 2 different machines.
> - No error are logged (worker, metric, supervisor).
> - I am using storm since 1 year and I never seen this problem.
>
> Thanks for your help.
>
> Denis
>
> ---
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> https://www.avast.com/antivirus
>
>