You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by vesense <gi...@git.apache.org> on 2016/08/31 15:12:32 UTC
[GitHub] storm pull request #1665: STORM-2074: fix storm-kafka-monitor NPE bug
GitHub user vesense opened a pull request:
https://github.com/apache/storm/pull/1665
STORM-2074: fix storm-kafka-monitor NPE bug
https://issues.apache.org/jira/browse/STORM-2074
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vesense/storm STORM-2074
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1665.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1665
----
commit e61e7738328913ec429742cd2b3cb22a39eb263f
Author: Xin Wang <be...@163.com>
Date: 2016-08-31T15:11:28Z
STORM-2074: fix storm-kafka-monitor NPE bug
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on the issue:
https://github.com/apache/storm/pull/1665
Thanks @priyank5485 @abellina I think I have addressed your comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/1665#discussion_r77106883
--- Diff: external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java ---
@@ -89,6 +89,10 @@ public static void main (String args[]) {
printUsageAndExit(options, OPTION_ZK_SERVERS_LONG + " and " + OPTION_ZK_COMMITTED_NODE_LONG + " are required with " +
OPTION_OLD_CONSUMER_LONG);
}
+ String zkNode = commandLine.getOptionValue(OPTION_ZK_COMMITTED_NODE_LONG);
--- End diff --
`zkNode == null` is not unnecessary since `hasArg` is set to true. and I will remove.
but `zkNode.length() <= 1` is necessary because zkPath should start with `/`, so the valid path length should `> 1`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/storm/pull/1665#discussion_r77107294
--- Diff: external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java ---
@@ -373,16 +377,20 @@ private static Options buildOptions () {
curatorFramework.start();
String partitionPrefix = "partition_";
String zkPath = oldKafkaSpoutOffsetQuery.getZkPath();
- if (!zkPath.endsWith("/")) {
- zkPath += "/";
+ if (zkPath.endsWith("/")) {
+ zkPath = zkPath.substring(0, zkPath.length()-1);
+ }
+ if (curatorFramework.checkExists().forPath(zkPath) == null) {
+ System.out.printf("--zk-node '%s' is not exists.%n", zkPath);
--- End diff --
%n? You mean.. println? Also small nit: "is not exists" => "does not exist"
This is also using tabs so the indentation is off. Why not use spaces (4 spaces)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on the issue:
https://github.com/apache/storm/pull/1665
this should be merged into 1.x-branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/1665#discussion_r77109600
--- Diff: external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java ---
@@ -373,16 +377,20 @@ private static Options buildOptions () {
curatorFramework.start();
String partitionPrefix = "partition_";
String zkPath = oldKafkaSpoutOffsetQuery.getZkPath();
- if (!zkPath.endsWith("/")) {
- zkPath += "/";
+ if (zkPath.endsWith("/")) {
+ zkPath = zkPath.substring(0, zkPath.length()-1);
+ }
+ if (curatorFramework.checkExists().forPath(zkPath) == null) {
+ System.out.printf("--zk-node '%s' is not exists.%n", zkPath);
--- End diff --
Will fix. Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1665
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by priyank5485 <gi...@git.apache.org>.
Github user priyank5485 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1665#discussion_r77034491
--- Diff: external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java ---
@@ -373,16 +377,20 @@ private static Options buildOptions () {
curatorFramework.start();
String partitionPrefix = "partition_";
String zkPath = oldKafkaSpoutOffsetQuery.getZkPath();
- if (!zkPath.endsWith("/")) {
- zkPath += "/";
+ if (zkPath.endsWith("/")) {
--- End diff --
Is there a reason for removing trailing / here and then appending in the loop below? The loop already has a check for curatorFramework.checkExists().forPath(path). May be we can just print and exit therE? Also, can you call the printUsageAndExit method here to be consistent? Or is there a reason you could not use it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by priyank5485 <gi...@git.apache.org>.
Github user priyank5485 commented on the issue:
https://github.com/apache/storm/pull/1665
@vesense Not sure if you saw my comment about exiting with a code of 1 in getOldConsumerOffsetsFromZk. Can you check that comment and address it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on the issue:
https://github.com/apache/storm/pull/1665
@priyank5485 Updated. Can you take a look again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on the issue:
https://github.com/apache/storm/pull/1665
@HeartSaVioR Can we merge this in? and, it's very easy to cherry-pick to 1.x-branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by priyank5485 <gi...@git.apache.org>.
Github user priyank5485 commented on the issue:
https://github.com/apache/storm/pull/1665
+1 (non-binding)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by priyank5485 <gi...@git.apache.org>.
Github user priyank5485 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1665#discussion_r77237088
--- Diff: external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java ---
@@ -89,6 +89,10 @@ public static void main (String args[]) {
printUsageAndExit(options, OPTION_ZK_SERVERS_LONG + " and " + OPTION_ZK_COMMITTED_NODE_LONG + " are required with " +
OPTION_OLD_CONSUMER_LONG);
}
+ String zkNode = commandLine.getOptionValue(OPTION_ZK_COMMITTED_NODE_LONG);
--- End diff --
I am not sure if just checking the length <= 1 is really that useful. As you mentioned that it should start with / may be we add check satisfying all the conditions for a valid zookeeper node name? Or just remove any check? We already have a check to see if node exists later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/1665#discussion_r77310159
--- Diff: external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java ---
@@ -89,6 +89,10 @@ public static void main (String args[]) {
printUsageAndExit(options, OPTION_ZK_SERVERS_LONG + " and " + OPTION_ZK_COMMITTED_NODE_LONG + " are required with " +
OPTION_OLD_CONSUMER_LONG);
}
+ String zkNode = commandLine.getOptionValue(OPTION_ZK_COMMITTED_NODE_LONG);
--- End diff --
we can just remove the check since `curatorFramework.checkExists().forPath(zkPath)` will do the same check for zkpath. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by priyank5485 <gi...@git.apache.org>.
Github user priyank5485 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1665#discussion_r77033989
--- Diff: external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java ---
@@ -89,6 +89,10 @@ public static void main (String args[]) {
printUsageAndExit(options, OPTION_ZK_SERVERS_LONG + " and " + OPTION_ZK_COMMITTED_NODE_LONG + " are required with " +
OPTION_OLD_CONSUMER_LONG);
}
+ String zkNode = commandLine.getOptionValue(OPTION_ZK_COMMITTED_NODE_LONG);
--- End diff --
While adding OPTION_ZK_NODE_COMMITTED_NODE_LONG to options it is added using the method options.addOption(String opt, String longOpt, boolean hasArg, String description) The third argument is set to true in the call which specifies that the argument must have a value. Does not that take care of null or empty value? Do we need to have this explicit check here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by priyank5485 <gi...@git.apache.org>.
Github user priyank5485 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1665#discussion_r77241785
--- Diff: external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java ---
@@ -373,16 +377,20 @@ private static Options buildOptions () {
curatorFramework.start();
String partitionPrefix = "partition_";
String zkPath = oldKafkaSpoutOffsetQuery.getZkPath();
- if (!zkPath.endsWith("/")) {
- zkPath += "/";
+ if (zkPath.endsWith("/")) {
--- End diff --
This is called from TopologySpoutLag using ShellUtils to display the results in UI. It is expected that KafkaOffsetLagUtil will always exit with code 0, unless there is a command line options error. TopologySpout guarantees that it will never call with wrong command line arguments. For any other runtime errors it will still exit with code 0, printing out the error condition on System.out. Reason is we want the error message to be displayed in UI. That is why there is a catch all block in the main method. If exit code is not 0, ShellUtils will throw an exception and there will be no meaningful message printed on storm UI. Instead of exiting here, can you throw an exception with a message like "zk node does not exist" and let the main method catch it. If you want to test out what I mentioned you can build a quick topology with kafka spout and look at storm ui.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on the issue:
https://github.com/apache/storm/pull/1665
Oh, I missed it. Will update. Thanks @priyank5485
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1665
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/1665#discussion_r77107687
--- Diff: external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java ---
@@ -373,16 +377,20 @@ private static Options buildOptions () {
curatorFramework.start();
String partitionPrefix = "partition_";
String zkPath = oldKafkaSpoutOffsetQuery.getZkPath();
- if (!zkPath.endsWith("/")) {
- zkPath += "/";
+ if (zkPath.endsWith("/")) {
--- End diff --
I think it is better to check the variable before use it. and the zkPath must not ends with `/`.
the `printUsageAndExit` method needs argument `Options` which is not available in `getOldConsumerOffsetsFromZk`. so I use `exit` directly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on the issue:
https://github.com/apache/storm/pull/1665
@HeartSaVioR OK, have a good time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1665: STORM-2074: fix storm-kafka-monitor NPE bug
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1665
I'd like to wait for @priyank5485 and @abellina to finish reviewing. I'm now reviewing and testing Supervisor V2 and it could take some times.
FYI: I'm on vacation on next week, and will be traveling in several days (not too long but not available for that moment).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---