[jira] [Commented] (STORM-2600) Improve or replace storm-kafka-monitor

["Shyne Qin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249242#comment-16249242 ] 

Shyne Qin commented on STORM-2600:
----------------------------------

Hi，my dear friends,I have a problem,I want the kafka spout offsets lag to display in the storm ui,but I don't know how to make it happen. I am not a programer,I am an Operation & Maintenance Engineer,if you have a  document,please send an email to me (lichenglongqi@outlook.com) ,thank you very muck

> Improve or replace storm-kafka-monitor
> --------------------------------------
>
>                 Key: STORM-2600
>                 URL: https://issues.apache.org/jira/browse/STORM-2600
>             Project: Apache Storm
>          Issue Type: Improvement
>          Components: storm-kafka-monitor
>    Affects Versions: 2.0.0
>            Reporter: Stig Rohde Døssing
>            Priority: Minor
>
> The storm-kafka-monitor module, which is used by Storm UI to show offset lag for topologies with Kafka spouts, has some shortcomings:
> * The Storm UI integration code doesn't seem to be able to support topic subscriptions that change after topology submission. The UI code (https://github.com/apache/storm/blob/64e29f365c9b5d3e15b33f33ab64e200345333e4/storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java#L91) gets the topic list it should request offset lag for via the spout's getComponentConfiguration method, as far as I can tell through this call https://github.com/apache/storm/blob/9e31509d47c4e91c1009f55c7ccf321d7d7e63aa/storm-client/src/jvm/org/apache/storm/topology/TopologyBuilder.java#L541. It seems like the component configuration is intended to be static once the topology has started running. This prevents us from showing the right topic list for subscriptions that are not known at submission time, which is currently the case for Pattern subscriptions. The topic list for that type of subscription isn't known until the spout has started the KafkaConsumer in {{ISpout.open()}}. I don't see a way to fix this, unless there is some way to update the component configuration when the subscription changes.
> * The jar is installed along with the cluster, and depends on the Kafka version specified in Storm's root POM. Kafka guarantees backwards compatible client-server communication for one release only, so there's a potential coupling between Storm cluster version and Kafka version. If users want to update the Kafka version in storm-kafka-monitor, they have to rebuild that module and replace the jar in their Storm install.
> * The UI integration uses the storm-kafka-monitor Bash script to start the monitoring code, in order to avoid a dependency between storm-core and storm-kafka-monitor. This prevents the UI integration from working on Windows. We could supply a Windows script as well, but then we'd need to keep the two in sync.
> I am wondering if these problems could be solved by implementing offset lag monitoring via the metrics system instead. The spout could periodically seek to the log end offset and submit a metric for how far behind the committed offset is, then seek back to where it left off.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)