You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Alexandre Vermeerbergen <av...@gmail.com> on 2022/05/20 15:08:02 UTC
Missing "Kafka Spouts Lag" in Storm UI 2.4.0
Hello,
We have been using Storm in production since quite a long time, with
latest Storm 1.2.x.
After a very long hesitation, we decided to give a try to Storm 2.x,
first without rebuilding our big jars.
The very good surprise for us is that everything seems to work.
However, I found one major regression: with Storm 1.2.X UI we heavily
rely on "Kafka Spout lag" table to check if we have some issues with
our Spout on some partition.
However, with Storm 2.4.x, the "Kafka Spout lag" table is no longer visible.
Are we missing something?
Was "Kafka Spout Lag" dropped out of Storm 2.x (I don't see such
information in Storm 2.x release notes...)
I hope that we missing something trivial and that bringing back the
Kafka lag in Storm UI isn't much complicated.
Any ideas ?
Kind regards,
Alexandre
Re: Missing "Kafka Spouts Lag" in Storm UI 2.4.0
Posted by Bipin Prasad <bi...@apache.org>.
StormKafkaOffsetLagUtil is in external/storm-kafka-monitor module.
Looks like this jar file is not in the classpath.
Do you mind raising a storm jira for this issue at https://issues.apache.org/ under Apache Storm project - https://issues.apache.org/jira/projects/STORM and add the details of your setting and usage.
Thanks
On 2022/05/20 17:09:58 Alexandre Vermeerbergen wrote:
> Looks like in storm/bin directory the storm-kafka-monitor command is
> missing in Storm 2.4.0 binary distribution
>
> I tried to copy storm-kafka-monitor from our Storm 1.2.x installation,
> but the shell fails with ClassNotFound:
>
> [root@ip-172-31-44-210 bin]# ./storm-kafka-monitor
> Error: Could not find or load main class
> org.apache.storm.kafka.monitor.KafkaOffsetLagUtil
> Caused by: java.lang.ClassNotFoundException:
> org.apache.storm.kafka.monitor.KafkaOffsetLagUtil
>
> So I guess more is needed to restore Kafka lag display in Storm UI 2.4.0.
>
> Anyone having this Kafka lag display working with 2.4.0 ?
>
> Kind regards,
> Alexandre
>
> Le ven. 20 mai 2022 à 17:08, Alexandre Vermeerbergen
> <av...@gmail.com> a écrit :
> >
> > Hello,
> >
> > We have been using Storm in production since quite a long time, with
> > latest Storm 1.2.x.
> >
> > After a very long hesitation, we decided to give a try to Storm 2.x,
> > first without rebuilding our big jars.
> >
> > The very good surprise for us is that everything seems to work.
> >
> > However, I found one major regression: with Storm 1.2.X UI we heavily
> > rely on "Kafka Spout lag" table to check if we have some issues with
> > our Spout on some partition.
> >
> > However, with Storm 2.4.x, the "Kafka Spout lag" table is no longer visible.
> >
> > Are we missing something?
> >
> > Was "Kafka Spout Lag" dropped out of Storm 2.x (I don't see such
> > information in Storm 2.x release notes...)
> >
> > I hope that we missing something trivial and that bringing back the
> > Kafka lag in Storm UI isn't much complicated.
> >
> > Any ideas ?
> >
> > Kind regards,
> > Alexandre
>
Re: Missing "Kafka Spouts Lag" in Storm UI 2.4.0
Posted by Alexandre Vermeerbergen <av...@gmail.com>.
Looks like in storm/bin directory the storm-kafka-monitor command is
missing in Storm 2.4.0 binary distribution
I tried to copy storm-kafka-monitor from our Storm 1.2.x installation,
but the shell fails with ClassNotFound:
[root@ip-172-31-44-210 bin]# ./storm-kafka-monitor
Error: Could not find or load main class
org.apache.storm.kafka.monitor.KafkaOffsetLagUtil
Caused by: java.lang.ClassNotFoundException:
org.apache.storm.kafka.monitor.KafkaOffsetLagUtil
So I guess more is needed to restore Kafka lag display in Storm UI 2.4.0.
Anyone having this Kafka lag display working with 2.4.0 ?
Kind regards,
Alexandre
Le ven. 20 mai 2022 à 17:08, Alexandre Vermeerbergen
<av...@gmail.com> a écrit :
>
> Hello,
>
> We have been using Storm in production since quite a long time, with
> latest Storm 1.2.x.
>
> After a very long hesitation, we decided to give a try to Storm 2.x,
> first without rebuilding our big jars.
>
> The very good surprise for us is that everything seems to work.
>
> However, I found one major regression: with Storm 1.2.X UI we heavily
> rely on "Kafka Spout lag" table to check if we have some issues with
> our Spout on some partition.
>
> However, with Storm 2.4.x, the "Kafka Spout lag" table is no longer visible.
>
> Are we missing something?
>
> Was "Kafka Spout Lag" dropped out of Storm 2.x (I don't see such
> information in Storm 2.x release notes...)
>
> I hope that we missing something trivial and that bringing back the
> Kafka lag in Storm UI isn't much complicated.
>
> Any ideas ?
>
> Kind regards,
> Alexandre
Re: Missing "Kafka Spouts Lag" in Storm UI 2.4.0
Posted by Bipin Prasad <bi...@yahoo.com>.
Can you provide a few more details. (1) What Kafka component are you using (2) Topology components and there classes (3) Are you making API call? like to "/api/v1/topology/<topologyId>/lag" or looking in Storm UI itself. (4) Config value for "ui.disable.spout.lag.monitoring"
On Friday, May 20, 2022, 08:08:31 AM PDT, Alexandre Vermeerbergen <av...@gmail.com> wrote:
Hello,
We have been using Storm in production since quite a long time, with
latest Storm 1.2.x.
After a very long hesitation, we decided to give a try to Storm 2.x,
first without rebuilding our big jars.
The very good surprise for us is that everything seems to work.
However, I found one major regression: with Storm 1.2.X UI we heavily
rely on "Kafka Spout lag" table to check if we have some issues with
our Spout on some partition.
However, with Storm 2.4.x, the "Kafka Spout lag" table is no longer visible.
Are we missing something?
Was "Kafka Spout Lag" dropped out of Storm 2.x (I don't see such
information in Storm 2.x release notes...)
I hope that we missing something trivial and that bringing back the
Kafka lag in Storm UI isn't much complicated.
Any ideas ?
Kind regards,
Alexandre