You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Sreeram <sr...@gmail.com> on 2016/03/18 10:37:16 UTC
How to implement Hbase atomic increment from a storm bolt
Hi,
I am looking for suggestions from community on implementing HBase atomic
increment in a idempotent manner from inside a storm bolt.
Replay of the storm bolt results in a double increment.
Any suggestion on the right approach to be taken is welcome.
Thank you.
Regards,
Sreeram
Re: Problem when consuming storm metrics
Posted by Denis DEBARBIEUX <dd...@norsys.fr>.
Dear all,
I hope that this issue is now fixed.
I rewrote all my code related to metrics and now it is working.
- All my code is now based on ReducedMetric
- I removed all my static code.
Thanks a lot for your patience and your help.
Denis
On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX" <ddebarbieux@norsys.fr
<ma...@norsys.fr>> wrote:
Dear all,
I have a question about metrics and how they are consumed:
I launch my topology. During 3 or 4 minutes, I receive my metrics.
Then no more logs.
It's look like the process that is consuming the metrics has a dead
lock or is blocked by a time out.
Here comes some information about the context:
- I am using storm 0.9.4.
- Some metrics are defined by defaut (eg emit-count.default) some
other are customed.
- I am sure that my topology is still running.
- According to Storm UI, the number of messages emiting by my
bolts is increasing. In other words, metrics are still computing and
send somewhere.
- I try two instances of LoggingMetricsConsumer: in file
(backtype.storm.metric.LoggingMetricsConsumer) and in graphite
(com.verisign.storm.metrics.GraphiteMetricsConsumer).
- I can reproduce it on 2 different machines.
- No error are logged (worker, metric, supervisor).
- I am using storm since 1 year and I never seen this problem.
Thanks for your help.
Denis
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Re: Problem when consuming storm metrics
Posted by Denis DEBARBIEUX <dd...@norsys.fr>.
Dear all,
Please find enclose the full thread dump. I hope that you can find some
promishing information.
I did the following experiments:
- Reproduce the problem with the ExclamationTopology (storm starter).
But I fail.
- Activate acking (there is no acking in my toplogy). But I still have
the problem.
Sorry for the delay between my two posts.
Thanks for our help.
Denis
Le 11/04/2016 13:47, Abhishek Agarwal a écrit :
> I wanted to look at bolt stack specifically. With the names removed,
> it is difficult to figure out. can you tell me what are the
> bolts/spouts busy in? Are they very slow? The timer which publishes
> metric tick is blocked because the receive queue of some executor is
> full.
>
> Some related details are captured here -
> https://issues.apache.org/jira/browse/STORM-972
>
> On Mon, Apr 11, 2016 at 2:28 PM, Denis DEBARBIEUX
> <ddebarbieux@norsys.fr <ma...@norsys.fr>> wrote:
>
> Jungtaek,
>
> Thanks for your help.
>
> Please find enclose the jstack file:
> - I remove by hand all name related to my project (spout, bolt...).
> - The running jdk was openJDK (not an oracle JDK).
>
> I also increase the buffer size
> (topology.executor.receive.buffer.size and
> topology.executor.send.buffer.size) to 32768 and lauch my
> topology on 2 workers. Then the dead lock occurs after 3 days. At
> this moment 24577 points have been send.
>
> Please feel free to contact me if you need further information.
>
> Denis
>
> Le 29/03/2016 11:11, Jungtaek Lim a écrit :
>> Denis,
>>
>> Could you share your jstack output which contains locked information?
>> And I'm not sure but you may hit STORM-1027
>> <https://issues.apache.org/jira/browse/STORM-1027> which is
>> resolved at 0.10.0 and 0.9.6.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>> 2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX
>> <ddebarbieux@norsys.fr <ma...@norsys.fr>> 님이 작성:
>>
>> Hi all,
>>
>> I profile my code and I see that the thread
>> <BoldName>MetricConsumer is locked and never unlocked. It is
>> why my metrics are never published.
>>
>> More over, the problem is related to buffer size:
>> * topology.executor.receive.buffer.size=16 and
>> topology.executor.send.buffer.size=16 implies a blocking
>> after 4 minutes.
>> * with a size=512, I am able to run my test for more than 1 hour.
>>
>>
>> Thanks for your help.
>>
>> Denis
>>
>> Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>>>
>>> Do you get this problem if you use only the
>>> LoggingMetricsConsumer?
>>>
>>> Excuse typos
>>>
>>> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX"
>>> <ddebarbieux@norsys.fr <ma...@norsys.fr>> wrote:
>>>
>>> Dear all,
>>>
>>> I have a question about metrics and how they are consumed:
>>> I launch my topology. During 3 or 4 minutes, I receive
>>> my metrics. Then no more logs.
>>> It's look like the process that is consuming the metrics
>>> has a dead lock or is blocked by a time out.
>>>
>>> Here comes some information about the context:
>>> - I am using storm 0.9.4.
>>> - Some metrics are defined by defaut (eg
>>> emit-count.default) some other are customed.
>>> - I am sure that my topology is still running.
>>> - According to Storm UI, the number of messages emiting
>>> by my bolts is increasing. In other words, metrics are
>>> still computing and send somewhere.
>>> - I try two instances of LoggingMetricsConsumer: in
>>> file (backtype.storm.metric.LoggingMetricsConsumer) and
>>> in graphite
>>> (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>>> - I can reproduce it on 2 different machines.
>>> - No error are logged (worker, metric, supervisor).
>>> - I am using storm since 1 year and I never seen this
>>> problem.
>>>
>>> Thanks for your help.
>>>
>>> Denis
>>>
>>> ---
>>> L'absence de virus dans ce courrier électronique a été
>>> vérifiée par le logiciel antivirus Avast.
>>> https://www.avast.com/antivirus
>>>
>>
>>
>>
>> ------------------------------------------------------------------------
>> Avast logo
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>
>>
>> L'absence de virus dans ce courrier électronique a été
>> vérifiée par le logiciel antivirus Avast.
>>
>> www.avast.com <http://www.avast.com>
>>
>>
>
>
>
> ------------------------------------------------------------------------
> Avast logo
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
> L'absence de virus dans ce courrier électronique a été vérifiée
> par le logiciel antivirus Avast.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
>
>
>
> --
> Regards,
> Abhishek Agarwal
>
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Re: Problem when consuming storm metrics
Posted by Abhishek Agarwal <ab...@gmail.com>.
I wanted to look at bolt stack specifically. With the names removed, it is
difficult to figure out. can you tell me what are the bolts/spouts busy in?
Are they very slow? The timer which publishes metric tick is blocked
because the receive queue of some executor is full.
Some related details are captured here -
https://issues.apache.org/jira/browse/STORM-972
On Mon, Apr 11, 2016 at 2:28 PM, Denis DEBARBIEUX <dd...@norsys.fr>
wrote:
> Jungtaek,
>
> Thanks for your help.
>
> Please find enclose the jstack file:
> - I remove by hand all name related to my project (spout, bolt...).
> - The running jdk was openJDK (not an oracle JDK).
>
> I also increase the buffer size (topology.executor.receive.buffer.size and
> topology.executor.send.buffer.size) to 32768 and lauch my topology on 2
> workers. Then the dead lock occurs after 3 days. At this moment 24577
> points have been send.
>
> Please feel free to contact me if you need further information.
>
> Denis
>
> Le 29/03/2016 11:11, Jungtaek Lim a écrit :
>
> Denis,
>
> Could you share your jstack output which contains locked information?
> And I'm not sure but you may hit STORM-1027
> <https://issues.apache.org/jira/browse/STORM-1027> which is resolved at
> 0.10.0 and 0.9.6.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX < <dd...@norsys.fr>
> ddebarbieux@norsys.fr> 님이 작성:
>
>> Hi all,
>>
>> I profile my code and I see that the thread <BoldName>MetricConsumer is
>> locked and never unlocked. It is why my metrics are never published.
>>
>> More over, the problem is related to buffer size:
>> * topology.executor.receive.buffer.size=16 and
>> topology.executor.send.buffer.size=16 implies a blocking after 4 minutes.
>> * with a size=512, I am able to run my test for more than 1 hour.
>>
>>
>> Thanks for your help.
>>
>> Denis
>>
>> Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>>
>> Do you get this problem if you use only the LoggingMetricsConsumer?
>>
>> Excuse typos
>> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX" <dd...@norsys.fr>
>> wrote:
>>
>>> Dear all,
>>>
>>> I have a question about metrics and how they are consumed:
>>> I launch my topology. During 3 or 4 minutes, I receive my metrics. Then
>>> no more logs.
>>> It's look like the process that is consuming the metrics has a dead lock
>>> or is blocked by a time out.
>>>
>>> Here comes some information about the context:
>>> - I am using storm 0.9.4.
>>> - Some metrics are defined by defaut (eg emit-count.default) some other
>>> are customed.
>>> - I am sure that my topology is still running.
>>> - According to Storm UI, the number of messages emiting by my bolts is
>>> increasing. In other words, metrics are still computing and send somewhere.
>>> - I try two instances of LoggingMetricsConsumer: in file
>>> (backtype.storm.metric.LoggingMetricsConsumer) and in graphite
>>> (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>>> - I can reproduce it on 2 different machines.
>>> - No error are logged (worker, metric, supervisor).
>>> - I am using storm since 1 year and I never seen this problem.
>>>
>>> Thanks for your help.
>>>
>>> Denis
>>>
>>> ---
>>> L'absence de virus dans ce courrier électronique a été vérifiée par le
>>> logiciel antivirus Avast.
>>> https://www.avast.com/antivirus
>>>
>>>
>>
>>
>> ------------------------------
>> [image: Avast logo]
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>
>> L'absence de virus dans ce courrier électronique a été vérifiée par le
>> logiciel antivirus Avast.
>>
>>
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>> www.avast.com
>>
>>
>
>
> ------------------------------
> [image: Avast logo]
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
--
Regards,
Abhishek Agarwal
Re: Problem when consuming storm metrics
Posted by Denis DEBARBIEUX <dd...@norsys.fr>.
Jungtaek,
Thanks for your help.
Please find enclose the jstack file:
- I remove by hand all name related to my project (spout, bolt...).
- The running jdk was openJDK (not an oracle JDK).
I also increase the buffer size (topology.executor.receive.buffer.size
and topology.executor.send.buffer.size) to 32768 and lauch my topology
on 2 workers. Then the dead lock occurs after 3 days. At this moment
24577 points have been send.
Please feel free to contact me if you need further information.
Denis
Le 29/03/2016 11:11, Jungtaek Lim a écrit :
> Denis,
>
> Could you share your jstack output which contains locked information?
> And I'm not sure but you may hit STORM-1027
> <https://issues.apache.org/jira/browse/STORM-1027> which is resolved
> at 0.10.0 and 0.9.6.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX
> <ddebarbieux@norsys.fr <ma...@norsys.fr>> 님이 작성:
>
> Hi all,
>
> I profile my code and I see that the thread
> <BoldName>MetricConsumer is locked and never unlocked. It is why
> my metrics are never published.
>
> More over, the problem is related to buffer size:
> * topology.executor.receive.buffer.size=16 and
> topology.executor.send.buffer.size=16 implies a blocking after 4
> minutes.
> * with a size=512, I am able to run my test for more than 1 hour.
>
>
> Thanks for your help.
>
> Denis
>
> Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>>
>> Do you get this problem if you use only the LoggingMetricsConsumer?
>>
>> Excuse typos
>>
>> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX"
>> <ddebarbieux@norsys.fr <ma...@norsys.fr>> wrote:
>>
>> Dear all,
>>
>> I have a question about metrics and how they are consumed:
>> I launch my topology. During 3 or 4 minutes, I receive my
>> metrics. Then no more logs.
>> It's look like the process that is consuming the metrics has
>> a dead lock or is blocked by a time out.
>>
>> Here comes some information about the context:
>> - I am using storm 0.9.4.
>> - Some metrics are defined by defaut (eg emit-count.default)
>> some other are customed.
>> - I am sure that my topology is still running.
>> - According to Storm UI, the number of messages emiting by
>> my bolts is increasing. In other words, metrics are still
>> computing and send somewhere.
>> - I try two instances of LoggingMetricsConsumer: in file
>> (backtype.storm.metric.LoggingMetricsConsumer) and in
>> graphite (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>> - I can reproduce it on 2 different machines.
>> - No error are logged (worker, metric, supervisor).
>> - I am using storm since 1 year and I never seen this problem.
>>
>> Thanks for your help.
>>
>> Denis
>>
>> ---
>> L'absence de virus dans ce courrier électronique a été
>> vérifiée par le logiciel antivirus Avast.
>> https://www.avast.com/antivirus
>>
>
>
>
> ------------------------------------------------------------------------
> Avast logo
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
> L'absence de virus dans ce courrier électronique a été vérifiée
> par le logiciel antivirus Avast.
>
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Re: Problem when consuming storm metrics
Posted by Denis DEBARBIEUX <dd...@norsys.fr>.
Hi,
Thanks for you help.
I did many tests with version 0.9.6 but I did not find any improvement.
Denis
Le 29/03/2016 11:11, Jungtaek Lim a écrit :
> Denis,
>
> Could you share your jstack output which contains locked information?
> And I'm not sure but you may hit STORM-1027
> <https://issues.apache.org/jira/browse/STORM-1027> which is resolved
> at 0.10.0 and 0.9.6.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX
> <ddebarbieux@norsys.fr <ma...@norsys.fr>> 님이 작성:
>
> Hi all,
>
> I profile my code and I see that the thread
> <BoldName>MetricConsumer is locked and never unlocked. It is why
> my metrics are never published.
>
> More over, the problem is related to buffer size:
> * topology.executor.receive.buffer.size=16 and
> topology.executor.send.buffer.size=16 implies a blocking after 4
> minutes.
> * with a size=512, I am able to run my test for more than 1 hour.
>
>
> Thanks for your help.
>
> Denis
>
> Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>>
>> Do you get this problem if you use only the LoggingMetricsConsumer?
>>
>> Excuse typos
>>
>> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX"
>> <ddebarbieux@norsys.fr <ma...@norsys.fr>> wrote:
>>
>> Dear all,
>>
>> I have a question about metrics and how they are consumed:
>> I launch my topology. During 3 or 4 minutes, I receive my
>> metrics. Then no more logs.
>> It's look like the process that is consuming the metrics has
>> a dead lock or is blocked by a time out.
>>
>> Here comes some information about the context:
>> - I am using storm 0.9.4.
>> - Some metrics are defined by defaut (eg emit-count.default)
>> some other are customed.
>> - I am sure that my topology is still running.
>> - According to Storm UI, the number of messages emiting by
>> my bolts is increasing. In other words, metrics are still
>> computing and send somewhere.
>> - I try two instances of LoggingMetricsConsumer: in file
>> (backtype.storm.metric.LoggingMetricsConsumer) and in
>> graphite (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>> - I can reproduce it on 2 different machines.
>> - No error are logged (worker, metric, supervisor).
>> - I am using storm since 1 year and I never seen this problem.
>>
>> Thanks for your help.
>>
>> Denis
>>
>> ---
>> L'absence de virus dans ce courrier électronique a été
>> vérifiée par le logiciel antivirus Avast.
>> https://www.avast.com/antivirus
>>
>
>
>
> ------------------------------------------------------------------------
> Avast logo
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
> L'absence de virus dans ce courrier électronique a été vérifiée
> par le logiciel antivirus Avast.
>
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Re: Problem when consuming storm metrics
Posted by Jungtaek Lim <ka...@gmail.com>.
Denis,
Could you share your jstack output which contains locked information?
And I'm not sure but you may hit STORM-1027
<https://issues.apache.org/jira/browse/STORM-1027> which is resolved at
0.10.0 and 0.9.6.
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX <dd...@norsys.fr>님이 작성:
> Hi all,
>
> I profile my code and I see that the thread <BoldName>MetricConsumer is
> locked and never unlocked. It is why my metrics are never published.
>
> More over, the problem is related to buffer size:
> * topology.executor.receive.buffer.size=16 and
> topology.executor.send.buffer.size=16 implies a blocking after 4 minutes.
> * with a size=512, I am able to run my test for more than 1 hour.
>
>
> Thanks for your help.
>
> Denis
>
> Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>
> Do you get this problem if you use only the LoggingMetricsConsumer?
>
> Excuse typos
> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX" <dd...@norsys.fr> wrote:
>
>> Dear all,
>>
>> I have a question about metrics and how they are consumed:
>> I launch my topology. During 3 or 4 minutes, I receive my metrics. Then
>> no more logs.
>> It's look like the process that is consuming the metrics has a dead lock
>> or is blocked by a time out.
>>
>> Here comes some information about the context:
>> - I am using storm 0.9.4.
>> - Some metrics are defined by defaut (eg emit-count.default) some other
>> are customed.
>> - I am sure that my topology is still running.
>> - According to Storm UI, the number of messages emiting by my bolts is
>> increasing. In other words, metrics are still computing and send somewhere.
>> - I try two instances of LoggingMetricsConsumer: in file
>> (backtype.storm.metric.LoggingMetricsConsumer) and in graphite
>> (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>> - I can reproduce it on 2 different machines.
>> - No error are logged (worker, metric, supervisor).
>> - I am using storm since 1 year and I never seen this problem.
>>
>> Thanks for your help.
>>
>> Denis
>>
>> ---
>> L'absence de virus dans ce courrier électronique a été vérifiée par le
>> logiciel antivirus Avast.
>> https://www.avast.com/antivirus
>>
>>
>
>
> ------------------------------
> [image: Avast logo]
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
>
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
Re: Problem when consuming storm metrics
Posted by Denis DEBARBIEUX <dd...@norsys.fr>.
Hi all,
I profile my code and I see that the thread <BoldName>MetricConsumer is
locked and never unlocked. It is why my metrics are never published.
More over, the problem is related to buffer size:
* topology.executor.receive.buffer.size=16 and
topology.executor.send.buffer.size=16 implies a blocking after 4 minutes.
* with a size=512, I am able to run my test for more than 1 hour.
Thanks for your help.
Denis
Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>
> Do you get this problem if you use only the LoggingMetricsConsumer?
>
> Excuse typos
>
> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX" <ddebarbieux@norsys.fr
> <ma...@norsys.fr>> wrote:
>
> Dear all,
>
> I have a question about metrics and how they are consumed:
> I launch my topology. During 3 or 4 minutes, I receive my metrics.
> Then no more logs.
> It's look like the process that is consuming the metrics has a
> dead lock or is blocked by a time out.
>
> Here comes some information about the context:
> - I am using storm 0.9.4.
> - Some metrics are defined by defaut (eg emit-count.default) some
> other are customed.
> - I am sure that my topology is still running.
> - According to Storm UI, the number of messages emiting by my
> bolts is increasing. In other words, metrics are still computing
> and send somewhere.
> - I try two instances of LoggingMetricsConsumer: in file
> (backtype.storm.metric.LoggingMetricsConsumer) and in graphite
> (com.verisign.storm.metrics.GraphiteMetricsConsumer).
> - I can reproduce it on 2 different machines.
> - No error are logged (worker, metric, supervisor).
> - I am using storm since 1 year and I never seen this problem.
>
> Thanks for your help.
>
> Denis
>
> ---
> L'absence de virus dans ce courrier électronique a été vérifiée
> par le logiciel antivirus Avast.
> https://www.avast.com/antivirus
>
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Re: RE : Problem when consuming storm metrics
Posted by Abhishek Agarwal <ab...@gmail.com>.
It would be a good idea to take thread dump when logging stops. That could
give some clues
Excuse typos
On Mar 19, 2016 9:33 PM, "Denis DEBARBIEUX" <dd...@norsys.fr> wrote:
> Yes, I do.
>
> Denis
> ------------------------------
> *De :* Abhishek Agarwal [abhishcool@gmail.com]
> *Date d'envoi :* vendredi 18 mars 2016 22:38
> *À :* user@storm.apache.org
> *Objet :* Re: Problem when consuming storm metrics
>
> Do you get this problem if you use only the LoggingMetricsConsumer?
>
> Excuse typos
> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX" <dd...@norsys.fr> wrote:
>
>> Dear all,
>>
>> I have a question about metrics and how they are consumed:
>> I launch my topology. During 3 or 4 minutes, I receive my metrics. Then
>> no more logs.
>> It's look like the process that is consuming the metrics has a dead lock
>> or is blocked by a time out.
>>
>> Here comes some information about the context:
>> - I am using storm 0.9.4.
>> - Some metrics are defined by defaut (eg emit-count.default) some other
>> are customed.
>> - I am sure that my topology is still running.
>> - According to Storm UI, the number of messages emiting by my bolts is
>> increasing. In other words, metrics are still computing and send somewhere.
>> - I try two instances of LoggingMetricsConsumer: in file
>> (backtype.storm.metric.LoggingMetricsConsumer) and in graphite
>> (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>> - I can reproduce it on 2 different machines.
>> - No error are logged (worker, metric, supervisor).
>> - I am using storm since 1 year and I never seen this problem.
>>
>> Thanks for your help.
>>
>> Denis
>>
>> ---
>> L'absence de virus dans ce courrier électronique a été vérifiée par le
>> logiciel antivirus Avast.
>> https://www.avast.com/antivirus
>>
>>
Re: Problem when consuming storm metrics
Posted by Abhishek Agarwal <ab...@gmail.com>.
Do you get this problem if you use only the LoggingMetricsConsumer?
Excuse typos
On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX" <dd...@norsys.fr> wrote:
> Dear all,
>
> I have a question about metrics and how they are consumed:
> I launch my topology. During 3 or 4 minutes, I receive my metrics. Then no
> more logs.
> It's look like the process that is consuming the metrics has a dead lock
> or is blocked by a time out.
>
> Here comes some information about the context:
> - I am using storm 0.9.4.
> - Some metrics are defined by defaut (eg emit-count.default) some other
> are customed.
> - I am sure that my topology is still running.
> - According to Storm UI, the number of messages emiting by my bolts is
> increasing. In other words, metrics are still computing and send somewhere.
> - I try two instances of LoggingMetricsConsumer: in file
> (backtype.storm.metric.LoggingMetricsConsumer) and in graphite
> (com.verisign.storm.metrics.GraphiteMetricsConsumer).
> - I can reproduce it on 2 different machines.
> - No error are logged (worker, metric, supervisor).
> - I am using storm since 1 year and I never seen this problem.
>
> Thanks for your help.
>
> Denis
>
> ---
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> https://www.avast.com/antivirus
>
>
Problem when consuming storm metrics
Posted by Denis DEBARBIEUX <dd...@norsys.fr>.
Dear all,
I have a question about metrics and how they are consumed:
I launch my topology. During 3 or 4 minutes, I receive my metrics. Then
no more logs.
It's look like the process that is consuming the metrics has a dead lock
or is blocked by a time out.
Here comes some information about the context:
- I am using storm 0.9.4.
- Some metrics are defined by defaut (eg emit-count.default) some
other are customed.
- I am sure that my topology is still running.
- According to Storm UI, the number of messages emiting by my bolts is
increasing. In other words, metrics are still computing and send somewhere.
- I try two instances of LoggingMetricsConsumer: in file
(backtype.storm.metric.LoggingMetricsConsumer) and in graphite
(com.verisign.storm.metrics.GraphiteMetricsConsumer).
- I can reproduce it on 2 different machines.
- No error are logged (worker, metric, supervisor).
- I am using storm since 1 year and I never seen this problem.
Thanks for your help.
Denis
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Re: How to implement Hbase atomic increment from a storm bolt
Posted by Sreeram <sr...@gmail.com>.
Dear Alexander,
Thank you very much for firing up some neurons ... very much helpful :)
Regards,
Sreeram
On Fri, Mar 18, 2016 at 8:43 PM, Alexander T <mi...@gmail.com>
wrote:
> Hi Sreeram,
>
> Some ideas: One way to tolerate replays would be to use some type of CRDT
> counter (https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type).
> Poor mans CRDT could be constructed if you have a strict ordering of your
> messages through a monotonically increasing counter. Save the message id
> when you update the count and only update if your message id is larger than
> the last saved. This could also be done sloppily by using timestamps. If
> you don't want to store the extra state in HBase you could utilize a
> stateful bolt to store it.
>
> Best regards,
>
> Alexander
>
> On Fri, Mar 18, 2016 at 10:37 AM, Sreeram <sr...@gmail.com> wrote:
>
>> Hi,
>>
>> I am looking for suggestions from community on implementing HBase atomic
>> increment in a idempotent manner from inside a storm bolt.
>>
>> Replay of the storm bolt results in a double increment.
>>
>> Any suggestion on the right approach to be taken is welcome.
>>
>> Thank you.
>>
>> Regards,
>> Sreeram
>>
>
>
Re: How to implement Hbase atomic increment from a storm bolt
Posted by Alexander T <mi...@gmail.com>.
Hi Sreeram,
Some ideas: One way to tolerate replays would be to use some type of CRDT
counter (https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type).
Poor mans CRDT could be constructed if you have a strict ordering of your
messages through a monotonically increasing counter. Save the message id
when you update the count and only update if your message id is larger than
the last saved. This could also be done sloppily by using timestamps. If
you don't want to store the extra state in HBase you could utilize a
stateful bolt to store it.
Best regards,
Alexander
On Fri, Mar 18, 2016 at 10:37 AM, Sreeram <sr...@gmail.com> wrote:
> Hi,
>
> I am looking for suggestions from community on implementing HBase atomic
> increment in a idempotent manner from inside a storm bolt.
>
> Replay of the storm bolt results in a double increment.
>
> Any suggestion on the right approach to be taken is welcome.
>
> Thank you.
>
> Regards,
> Sreeram
>