You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Shaik Asifullah <s....@gmail.com> on 2018/08/21 08:18:21 UTC
Monitoring Apache Storm Trident Topology for Kafka Consumer Lags
Hi,
I am using Apache Storm Trident (1.2.1) for micro-batching. The main idea
of my topology is to modify messages from Kafka a little and push it to
Database. I am using KafkaSpoutOpaque for one time semantics. Can someone
please suggest regarding monitoring the topology for Kafka Consumer Lags. I
realised that I cannot be using storm-kafka-monitor since offsets for
Trident topologies are maintained in Storm Zookeeper rather than Kafka
Zookeeper. Is there anyway to solve this?
Thanks,
Shaik Asifullah
Re: Monitoring Apache Storm Trident Topology for Kafka Consumer Lags
Posted by Joshua Martell <jo...@gmail.com>.
Yes, that sounds right. In the UI, you'll need to click "Show System
Stats" as metrics consumer bolts are hidden by default. The log will have
lots of detailed information including the metrics that power the UI.
Docs:
V1 http://storm.apache.org/releases/1.2.2/Metrics.html
V2 http://storm.apache.org/releases/1.2.2/metrics_v2.html
Joshua
On Tue, Aug 21, 2018 at 8:15 AM Shaik Asifullah <s....@gmail.com>
wrote:
> Hi Joshua,
>
> Thanks a lot for your help. If I could configure my topology with conf.registerMetricsConsumer(org.apache.storm.metrics.LoggingMetricsConsumer.class,
> 1); Will it work ? If not, can you please give me a little more
> understanding on enabling this since I am very new to Trident.
>
> Best regards,
> Shaik Asifuallah
>
> On Tue, Aug 21, 2018 at 6:12 PM Joshua Martell <jo...@gmail.com>
> wrote:
>
>> As I recall, the Kafka spout reported metrics that could be seen if you
>> attach a LoggingMetricsConsumer. Once you find them and determine they meet
>> your needs, you’ll want to report them somewhere that you can visualize and
>> alert.
>>
>> Joshua
>> On Tue, Aug 21, 2018 at 3:18 AM Shaik Asifullah <s....@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I am using Apache Storm Trident (1.2.1) for micro-batching. The main
>>> idea of my topology is to modify messages from Kafka a little and push it
>>> to Database. I am using KafkaSpoutOpaque for one time semantics. Can
>>> someone please suggest regarding monitoring the topology for Kafka Consumer
>>> Lags. I realised that I cannot be using storm-kafka-monitor since offsets
>>> for Trident topologies are maintained in Storm Zookeeper rather than Kafka
>>> Zookeeper. Is there anyway to solve this?
>>>
>>> Thanks,
>>> Shaik Asifullah
>>>
>>
Re: Monitoring Apache Storm Trident Topology for Kafka Consumer Lags
Posted by Shaik Asifullah <s....@gmail.com>.
Hi Joshua,
Thanks a lot for your help. If I could configure my topology with
conf.registerMetricsConsumer(org.apache.storm.metrics.LoggingMetricsConsumer.class,
1); Will it work ? If not, can you please give me a little more
understanding on enabling this since I am very new to Trident.
Best regards,
Shaik Asifuallah
On Tue, Aug 21, 2018 at 6:12 PM Joshua Martell <jo...@gmail.com>
wrote:
> As I recall, the Kafka spout reported metrics that could be seen if you
> attach a LoggingMetricsConsumer. Once you find them and determine they meet
> your needs, you’ll want to report them somewhere that you can visualize and
> alert.
>
> Joshua
> On Tue, Aug 21, 2018 at 3:18 AM Shaik Asifullah <s....@gmail.com>
> wrote:
>
>> Hi,
>> I am using Apache Storm Trident (1.2.1) for micro-batching. The main idea
>> of my topology is to modify messages from Kafka a little and push it to
>> Database. I am using KafkaSpoutOpaque for one time semantics. Can someone
>> please suggest regarding monitoring the topology for Kafka Consumer Lags. I
>> realised that I cannot be using storm-kafka-monitor since offsets for
>> Trident topologies are maintained in Storm Zookeeper rather than Kafka
>> Zookeeper. Is there anyway to solve this?
>>
>> Thanks,
>> Shaik Asifullah
>>
>
Re: Monitoring Apache Storm Trident Topology for Kafka Consumer Lags
Posted by Joshua Martell <jo...@gmail.com>.
As I recall, the Kafka spout reported metrics that could be seen if you
attach a LoggingMetricsConsumer. Once you find them and determine they meet
your needs, you’ll want to report them somewhere that you can visualize and
alert.
Joshua
On Tue, Aug 21, 2018 at 3:18 AM Shaik Asifullah <s....@gmail.com>
wrote:
> Hi,
> I am using Apache Storm Trident (1.2.1) for micro-batching. The main idea
> of my topology is to modify messages from Kafka a little and push it to
> Database. I am using KafkaSpoutOpaque for one time semantics. Can someone
> please suggest regarding monitoring the topology for Kafka Consumer Lags. I
> realised that I cannot be using storm-kafka-monitor since offsets for
> Trident topologies are maintained in Storm Zookeeper rather than Kafka
> Zookeeper. Is there anyway to solve this?
>
> Thanks,
> Shaik Asifullah
>