You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by HeartSaVioR <gi...@git.apache.org> on 2016/07/06 14:10:18 UTC
[GitHub] storm pull request #1541: STORM-1945 Fix NPE bugs on topology spout lag for ...
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/storm/pull/1541
STORM-1945 Fix NPE bugs on topology spout lag for storm-kafka-monitor (1.x)
PR for master: #1540
* Fix several spots which are throwing NPE on TopologySpoutLag
* Use a trick to get classname of Spout without requiring storm-kafka and storm-kafka-client
* Move storm-kafka-monitor jar to toollib directory
* since jars in extlib are added to worker classpath which we don't want for this jar
@priyank5485 @harshach Please take a look and comment. Thanks!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HeartSaVioR/storm STORM-1945-1.x
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1541.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1541
----
commit 224f542d539406297f84947aed33a404b73798d7
Author: Jungtaek Lim <ka...@gmail.com>
Date: 2016-07-06T13:49:46Z
STORM-1945 Fix NPE bugs on topology spout lag for storm-kafka-monitor
* Fix several spots which are throwing NPE on TopologySpoutLag
* Use a trick to get classname of Spout without requiring storm-kafka and storm-kafka-client
* Move storm-kafka-monitor jar to toollib directory
* since jars in extlib are added to worker classpath which we don't want for this jar
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1541: STORM-1945 Fix NPE bugs on topology spout lag for storm-k...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the issue:
https://github.com/apache/storm/pull/1541
+1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1541: STORM-1945 Fix NPE bugs on topology spout lag for storm-k...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1541
It would be better for changing JSON response to file a new issue and make it out of scope for this PR, since [STORM-1945](https://issues.apache.org/jira/browse/STORM-1945) is a bugfix which is marked as critical.
Filed new issue for changing JSON response format: [STORM-1950](https://issues.apache.org/jira/browse/STORM-1950)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1541: STORM-1945 Fix NPE bugs on topology spout lag for ...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/1541#discussion_r69841403
--- Diff: storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java ---
@@ -43,26 +43,44 @@
public static List<Map<String, Object>> lag (StormTopology stormTopology, Map topologyConf) {
List<Map<String, Object>> result = new ArrayList<>();
Map<String, SpoutSpec> spouts = stormTopology.get_spouts();
- Object object = null;
+ String className = null;
for (Map.Entry<String, SpoutSpec> spout: spouts.entrySet()) {
try {
SpoutSpec spoutSpec = spout.getValue();
ComponentObject componentObject = spoutSpec.get_spout_object();
- object = Utils.getSetComponentObject(componentObject);
--- End diff --
The problem starts here. For many situation we can't deserialize Spout object since UI doesn't have dependency for that Spout, for example, storm-kafka or storm-kafka-client.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1541: STORM-1945 Fix NPE bugs on topology spout lag for ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1541
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1541: STORM-1945 Fix NPE bugs on topology spout lag for ...
Posted by abhishekagarwal87 <gi...@git.apache.org>.
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1541#discussion_r69857098
--- Diff: storm-dist/binary/src/main/assembly/binary.xml ---
@@ -340,10 +340,10 @@
</fileSet>
<!-- $STORM_HOME/extlib -->
<fileSet>
- <directory>${project.basedir}/../../external/storm-kafka-monitor/target</directory>
+ <directory></directory>
--- End diff --
I see. Since storm-kafka-monitor was being copied always, this section was not needed anymore. Thanks for clarifying.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1541: STORM-1945 Fix NPE bugs on topology spout lag for ...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/1541#discussion_r69854113
--- Diff: storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java ---
@@ -43,26 +43,44 @@
public static List<Map<String, Object>> lag (StormTopology stormTopology, Map topologyConf) {
List<Map<String, Object>> result = new ArrayList<>();
Map<String, SpoutSpec> spouts = stormTopology.get_spouts();
- Object object = null;
+ String className = null;
for (Map.Entry<String, SpoutSpec> spout: spouts.entrySet()) {
try {
SpoutSpec spoutSpec = spout.getValue();
ComponentObject componentObject = spoutSpec.get_spout_object();
- object = Utils.getSetComponentObject(componentObject);
--- End diff --
@harshach Only task calls this method, no need to deserialize spout / bolt objects on daemons. Now UI is an exception case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1541: STORM-1945 Fix NPE bugs on topology spout lag for storm-k...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the issue:
https://github.com/apache/storm/pull/1541
@HeartSaVioR while we are here can we fix the JSON response as well.
currently what we have is this
[{"spoutLagResult":"[{\"topic\":\"topic\",\"partition\":0,\"consumerCommittedOffset\":1175610,\"logHeadOffset\":5634192,\"lag\":4458582},{\"topic\":\"topic\",\"partition\":1,\"consumerCommittedOffset\":1251325,\"logHeadOffset\":5634192,\"lag\":4382867}]","spoutId":"spout","spoutType":"KAFKA"}]
we should be returning an array instead lets have something like this
{ "spoutId" : "spout"
"spoutType" : "KAFKA"
"spoutLagResult" : [
"topic": // key is topic name
["partition0": {\"consumerCommittedOffset\":1175610,\"logHeadOffset\":5634192,\"lag\":4458582},
"partition2: {\"consumerCommittedOffset\":1175610,\"logHeadOffset\":5634192,\"lag\":4458582},
],
"topic2" :
["partition0": {\"consumerCommittedOffset\":1175610,\"logHeadOffset\":5634192,\"lag\":4458582},
"partition2: {\"consumerCommittedOffset\":1175610,\"logHeadOffset\":5634192,\"lag\":4458582},
]
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1541: STORM-1945 Fix NPE bugs on topology spout lag for ...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/1541#discussion_r69851558
--- Diff: storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java ---
@@ -43,26 +43,44 @@
public static List<Map<String, Object>> lag (StormTopology stormTopology, Map topologyConf) {
List<Map<String, Object>> result = new ArrayList<>();
Map<String, SpoutSpec> spouts = stormTopology.get_spouts();
- Object object = null;
+ String className = null;
for (Map.Entry<String, SpoutSpec> spout: spouts.entrySet()) {
try {
SpoutSpec spoutSpec = spout.getValue();
ComponentObject componentObject = spoutSpec.get_spout_object();
- object = Utils.getSetComponentObject(componentObject);
--- End diff --
@HeartSaVioR how do we deserialize for supervisors or others they don't have any dependency on spouts or any other component.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1541: STORM-1945 Fix NPE bugs on topology spout lag for storm-k...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1541
@harshach
I guess you mean this:
```
[
{
"spoutId":"spout",
"spoutType":"KAFKA",
"spoutLagResult":{
"topic":{
"partition0":{
"consumerCommittedOffset":1175610,
"logHeadOffset":5634192,
"lag":4458582
},
"partition2":{
"consumerCommittedOffset":1175610,
"logHeadOffset":5634192,
"lag":4458582
}
},
"topic2":{
"partition0":{
"consumerCommittedOffset":1175610,
"logHeadOffset":5634192,
"lag":4458582
},
"partition2":{
"consumerCommittedOffset":1175610,
"logHeadOffset":5634192,
"lag":4458582
}
}
}
}
]
```
I agree this is better for removing duplicated thing and also easy to find by path. I'll update it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1541: STORM-1945 Fix NPE bugs on topology spout lag for ...
Posted by abhishekagarwal87 <gi...@git.apache.org>.
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1541#discussion_r69855772
--- Diff: storm-dist/binary/src/main/assembly/binary.xml ---
@@ -340,10 +340,10 @@
</fileSet>
<!-- $STORM_HOME/extlib -->
<fileSet>
- <directory>${project.basedir}/../../external/storm-kafka-monitor/target</directory>
+ <directory></directory>
--- End diff --
This change is not clear. Directory change from extlib to toollib is happening in other section. Shouldn't this be removed entirely
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1541: STORM-1945 Fix NPE bugs on topology spout lag for ...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/1541#discussion_r69855945
--- Diff: storm-dist/binary/src/main/assembly/binary.xml ---
@@ -340,10 +340,10 @@
</fileSet>
<!-- $STORM_HOME/extlib -->
<fileSet>
- <directory>${project.basedir}/../../external/storm-kafka-monitor/target</directory>
+ <directory></directory>
--- End diff --
It's needed to create empty directory "extlib".
https://github.com/apache/storm/blob/1.0.x-branch/storm-dist/binary/src/main/assembly/binary.xml#L335
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---