You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Igor Kuzmenko <f1...@gmail.com> on 2017/02/01 09:01:54 UTC
Re: Kafka monitor unable to get offset lag
Yes, topology process data and works fine.
I couldn't find any exceptions in storm logs. access-web-ui.log contains
only these lines
2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
10.35.63.14 url: http://master001.s:8744/api/v1/topology/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
On Tue, Jan 31, 2017 at 5:19 PM, Priyank Shah <ps...@hortonworks.com> wrote:
> Hi Igor,
>
> When you say topology is working fine do you mean you see data flowing?
> Can you try to look up the logs for ui server and paste relevant lines here
> if any?
>
> Priyank
>
> Sent from my iPhone
>
> On Jan 31, 2017, at 4:34 AM, Igor Kuzmenko <f1...@gmail.com> wrote:
>
> I've launched topology with new kafka spout. Topology by it self working
> fine, but looking at storm UI I see kafka-monitor exception:
> *Unable to get offset lags for kafka. Reason:
> org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired
> while fetching topic metadata*
>
> Maybe I forgot to configure something, but then how topology reads
> messages?
>
>
Re: Kafka monitor unable to get offset lag
Posted by Priyank Shah <ps...@hortonworks.com>.
Hi Igor,
By secured setup I meant how is the kafka cluster setup. What port is the broker listening on?
Looked at the code. Not sure why is it throwing that error in UI. The only difference in debug output and command you run on command line is that –t argument has a trailing comma. Code should handle it.
Can you try running the below two commands on command line and paste output for each ?
/usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor -t gtp -g test -b dn-06.ru:6667<http://dn-06.ru:6667/>,dn-10.ru:6667<http://dn-10.ru:6667/>
/usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor -t gtp, -g test -b dn-06.ru:6667<http://dn-06.ru:6667/>,dn-10.ru:6667<http://dn-10.ru:6667/>
From: Igor Kuzmenko <f1...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Wednesday, February 1, 2017 at 11:53 AM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Re: Kafka monitor unable to get offset lag
Yes running from command line gives me json with topic lag.
What is secured setup?
On Wed, Feb 1, 2017 at 10:28 PM, Priyank Shah <ps...@hortonworks.com>> wrote:
Hi Igor,
Just to clarify, are you saying that running storm-kafka-monitor from command line with the arguments you pasted returns the spout lags correctly ? Also, can you confirm if you are running in a secured setup or not ?
From: Igor Kuzmenko <f1...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Wednesday, February 1, 2017 at 5:05 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Kafka monitor unable to get offset lag
Found in logs with debug level:
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] spout classname: org.apache.storm.kafka.spout.KafkaSpout
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] json configuration: {config.bootstrap.servers=dn-06.ru:6667<http://dn-06.ru:6667>,dn-10.ru:6667<http://dn-10.ru:6667>, config.topics=gtp,, config.groupid=test, topology.tasks=1}
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] Command to run: [/usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor, -t, gtp,, -g, test, -b, dn-06.ru:6667<http://dn-06.ru:6667>,dn-10.ru:6667<http://dn-10.ru:6667>]
2017-02-01 14:07:36.338 o.a.s.u.TopologySpoutLag [DEBUG] JSON parsing failed, assuming message as error message: Unable to get offset lags for kafka. Reason: org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired while fetching topic metadata
Running command /usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor -t gtp -g test -b dn-06.ru:6667<http://dn-06.ru:6667>,dn-10.ru:6667<http://dn-10.ru:6667> returns correct json with spout lags. There's no timeout command executing in a second.
On Wed, Feb 1, 2017 at 12:25 PM, Spico Florin <sp...@gmail.com>> wrote:
Hello!
You can check how your topic is consumed and its health via the scripts:
./kafka-consumer-groups.sh --new-consumer --bootstrap-server <your-broker>:6667 --list
./kafka-consumer-groups.sh --new-consumer --bootstrap-server <your-broker>:6667 --describe --group <your-group>
I hope it helps.
Florin
On Wed, Feb 1, 2017 at 11:01 AM, Igor Kuzmenko <f1...@gmail.com>> wrote:
Yes, topology process data and works fine.
I couldn't find any exceptions in storm logs. access-web-ui.log contains only these lines
2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 10.35.63.14 url: http://master001.s:8744/api/v1/topology/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
On Tue, Jan 31, 2017 at 5:19 PM, Priyank Shah <ps...@hortonworks.com>> wrote:
Hi Igor,
When you say topology is working fine do you mean you see data flowing? Can you try to look up the logs for ui server and paste relevant lines here if any?
Priyank
Sent from my iPhone
On Jan 31, 2017, at 4:34 AM, Igor Kuzmenko <f1...@gmail.com>> wrote:
I've launched topology with new kafka spout. Topology by it self working fine, but looking at storm UI I see kafka-monitor exception:
Unable to get offset lags for kafka. Reason: org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired while fetching topic metadata
Maybe I forgot to configure something, but then how topology reads messages?
Re: Kafka monitor unable to get offset lag
Posted by Igor Kuzmenko <f1...@gmail.com>.
Yes running from command line gives me json with topic lag.
What is secured setup?
On Wed, Feb 1, 2017 at 10:28 PM, Priyank Shah <ps...@hortonworks.com> wrote:
> Hi Igor,
>
>
>
> Just to clarify, are you saying that running storm-kafka-monitor from
> command line with the arguments you pasted returns the spout lags correctly
> ? Also, can you confirm if you are running in a secured setup or not ?
>
>
>
> *From: *Igor Kuzmenko <f1...@gmail.com>
> *Reply-To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Date: *Wednesday, February 1, 2017 at 5:05 AM
> *To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Subject: *Re: Kafka monitor unable to get offset lag
>
>
>
> Found in logs with debug level:
>
> 2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] spout classname:
> org.apache.storm.kafka.spout.KafkaSpout
>
> 2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] json
> configuration: {config.bootstrap.servers=dn-06.ru:6667,dn-10.ru:6667,
> config.topics=gtp,, config.groupid=test, topology.tasks=1}
>
> 2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] Command to run:
> [/usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor, -t, gtp,, -g, test,
> -b, dn-06.ru:6667,dn-10.ru:6667]
>
> 2017-02-01 14:07:36.338 o.a.s.u.TopologySpoutLag [DEBUG] JSON parsing
> failed, assuming message as error message: Unable to get offset lags for
> kafka. Reason: org.apache.kafka.shaded.common.errors.TimeoutException:
> Timeout expired while fetching topic metadata
>
>
>
> Running command /usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor -t
> gtp -g test -b dn-06.ru:6667,dn-10.ru:6667 returns correct json with
> spout lags. There's no timeout command executing in a second.
>
>
>
>
>
>
>
> On Wed, Feb 1, 2017 at 12:25 PM, Spico Florin <sp...@gmail.com>
> wrote:
>
> Hello!
>
> You can check how your topic is consumed and its health via the scripts:
>
> ./kafka-consumer-groups.sh --new-consumer --bootstrap-server
> <your-broker>:6667 --list
>
> ./kafka-consumer-groups.sh --new-consumer --bootstrap-server
> <your-broker>:6667 --describe --group <your-group>
>
>
>
> I hope it helps.
>
> Florin
>
>
>
> On Wed, Feb 1, 2017 at 11:01 AM, Igor Kuzmenko <f1...@gmail.com> wrote:
>
> Yes, topology process data and works fine.
>
> I couldn't find any exceptions in storm logs. access-web-ui.log contains
> only these lines
>
>
>
> 2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
>
> 2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
>
> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/topology/summary
> principal:
>
> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
>
> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
>
>
>
> On Tue, Jan 31, 2017 at 5:19 PM, Priyank Shah <ps...@hortonworks.com>
> wrote:
>
> Hi Igor,
>
>
>
> When you say topology is working fine do you mean you see data flowing?
> Can you try to look up the logs for ui server and paste relevant lines here
> if any?
>
>
>
> Priyank
>
> Sent from my iPhone
>
>
> On Jan 31, 2017, at 4:34 AM, Igor Kuzmenko <f1...@gmail.com> wrote:
>
> I've launched topology with new kafka spout. Topology by it self working
> fine, but looking at storm UI I see kafka-monitor exception:
>
> *Unable to get offset lags for kafka. Reason:
> org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired
> while fetching topic metadata*
>
>
>
> Maybe I forgot to configure something, but then how topology reads
> messages?
>
>
>
>
>
>
>
Re: Kafka monitor unable to get offset lag
Posted by Priyank Shah <ps...@hortonworks.com>.
Hi Igor,
Just to clarify, are you saying that running storm-kafka-monitor from command line with the arguments you pasted returns the spout lags correctly ? Also, can you confirm if you are running in a secured setup or not ?
From: Igor Kuzmenko <f1...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Wednesday, February 1, 2017 at 5:05 AM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Re: Kafka monitor unable to get offset lag
Found in logs with debug level:
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] spout classname: org.apache.storm.kafka.spout.KafkaSpout
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] json configuration: {config.bootstrap.servers=dn-06.ru:6667<http://dn-06.ru:6667>,dn-10.ru:6667<http://dn-10.ru:6667>, config.topics=gtp,, config.groupid=test, topology.tasks=1}
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] Command to run: [/usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor, -t, gtp,, -g, test, -b, dn-06.ru:6667<http://dn-06.ru:6667>,dn-10.ru:6667<http://dn-10.ru:6667>]
2017-02-01 14:07:36.338 o.a.s.u.TopologySpoutLag [DEBUG] JSON parsing failed, assuming message as error message: Unable to get offset lags for kafka. Reason: org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired while fetching topic metadata
Running command /usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor -t gtp -g test -b dn-06.ru:6667<http://dn-06.ru:6667>,dn-10.ru:6667<http://dn-10.ru:6667> returns correct json with spout lags. There's no timeout command executing in a second.
On Wed, Feb 1, 2017 at 12:25 PM, Spico Florin <sp...@gmail.com>> wrote:
Hello!
You can check how your topic is consumed and its health via the scripts:
./kafka-consumer-groups.sh --new-consumer --bootstrap-server <your-broker>:6667 --list
./kafka-consumer-groups.sh --new-consumer --bootstrap-server <your-broker>:6667 --describe --group <your-group>
I hope it helps.
Florin
On Wed, Feb 1, 2017 at 11:01 AM, Igor Kuzmenko <f1...@gmail.com>> wrote:
Yes, topology process data and works fine.
I couldn't find any exceptions in storm logs. access-web-ui.log contains only these lines
2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 10.35.63.14 url: http://master001.s:8744/api/v1/topology/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
On Tue, Jan 31, 2017 at 5:19 PM, Priyank Shah <ps...@hortonworks.com>> wrote:
Hi Igor,
When you say topology is working fine do you mean you see data flowing? Can you try to look up the logs for ui server and paste relevant lines here if any?
Priyank
Sent from my iPhone
On Jan 31, 2017, at 4:34 AM, Igor Kuzmenko <f1...@gmail.com>> wrote:
I've launched topology with new kafka spout. Topology by it self working fine, but looking at storm UI I see kafka-monitor exception:
Unable to get offset lags for kafka. Reason: org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired while fetching topic metadata
Maybe I forgot to configure something, but then how topology reads messages?
Re: Kafka monitor unable to get offset lag
Posted by Igor Kuzmenko <f1...@gmail.com>.
Found in logs with debug level:
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] spout classname:
org.apache.storm.kafka.spout.KafkaSpout
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] json
configuration: {config.bootstrap.servers=dn-06.ru:6667,dn-10.ru:6667,
config.topics=gtp,, config.groupid=test, topology.tasks=1}
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] Command to run:
[/usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor, -t, gtp,, -g, test,
-b, dn-06.ru:6667,dn-10.ru:6667]
2017-02-01 14:07:36.338 o.a.s.u.TopologySpoutLag [DEBUG] JSON parsing
failed, assuming message as error message: Unable to get offset lags for
kafka. Reason: org.apache.kafka.shaded.common.errors.TimeoutException:
Timeout expired while fetching topic metadata
Running command /usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor -t gtp
-g test -b dn-06.ru:6667,dn-10.ru:6667 returns correct json with spout
lags. There's no timeout command executing in a second.
On Wed, Feb 1, 2017 at 12:25 PM, Spico Florin <sp...@gmail.com> wrote:
> Hello!
> You can check how your topic is consumed and its health via the scripts:
> ./kafka-consumer-groups.sh --new-consumer --bootstrap-server
> <your-broker>:6667 --list
> ./kafka-consumer-groups.sh --new-consumer --bootstrap-server
> <your-broker>:6667 --describe --group <your-group>
>
> I hope it helps.
> Florin
>
> On Wed, Feb 1, 2017 at 11:01 AM, Igor Kuzmenko <f1...@gmail.com> wrote:
>
>> Yes, topology process data and works fine.
>> I couldn't find any exceptions in storm logs. access-web-ui.log
>> contains only these lines
>>
>> 2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
>> 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary
>> principal:
>> 2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
>> 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
>> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
>> 10.35.63.14 url: http://master001.s:8744/api/v1/topology/summary
>> principal:
>> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
>> 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary
>> principal:
>> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
>> 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
>>
>> On Tue, Jan 31, 2017 at 5:19 PM, Priyank Shah <ps...@hortonworks.com>
>> wrote:
>>
>>> Hi Igor,
>>>
>>> When you say topology is working fine do you mean you see data flowing?
>>> Can you try to look up the logs for ui server and paste relevant lines here
>>> if any?
>>>
>>> Priyank
>>>
>>> Sent from my iPhone
>>>
>>> On Jan 31, 2017, at 4:34 AM, Igor Kuzmenko <f1...@gmail.com> wrote:
>>>
>>> I've launched topology with new kafka spout. Topology by it self working
>>> fine, but looking at storm UI I see kafka-monitor exception:
>>> *Unable to get offset lags for kafka. Reason:
>>> org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired
>>> while fetching topic metadata*
>>>
>>> Maybe I forgot to configure something, but then how topology reads
>>> messages?
>>>
>>>
>>
>
Re: Kafka monitor unable to get offset lag
Posted by Spico Florin <sp...@gmail.com>.
Hello!
You can check how your topic is consumed and its health via the scripts:
./kafka-consumer-groups.sh --new-consumer --bootstrap-server
<your-broker>:6667 --list
./kafka-consumer-groups.sh --new-consumer --bootstrap-server
<your-broker>:6667 --describe --group <your-group>
I hope it helps.
Florin
On Wed, Feb 1, 2017 at 11:01 AM, Igor Kuzmenko <f1...@gmail.com> wrote:
> Yes, topology process data and works fine.
> I couldn't find any exceptions in storm logs. access-web-ui.log contains
> only these lines
>
> 2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
> 2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/topology/summary
> principal:
> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
>
> On Tue, Jan 31, 2017 at 5:19 PM, Priyank Shah <ps...@hortonworks.com>
> wrote:
>
>> Hi Igor,
>>
>> When you say topology is working fine do you mean you see data flowing?
>> Can you try to look up the logs for ui server and paste relevant lines here
>> if any?
>>
>> Priyank
>>
>> Sent from my iPhone
>>
>> On Jan 31, 2017, at 4:34 AM, Igor Kuzmenko <f1...@gmail.com> wrote:
>>
>> I've launched topology with new kafka spout. Topology by it self working
>> fine, but looking at storm UI I see kafka-monitor exception:
>> *Unable to get offset lags for kafka. Reason:
>> org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired
>> while fetching topic metadata*
>>
>> Maybe I forgot to configure something, but then how topology reads
>> messages?
>>
>>
>