You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Kamal B <bk...@gmail.com> on 2016/08/15 08:28:20 UTC
Storm Development Assistance
Hi All,
I am newbie to Storm Community and intend to learn Storm and willing to
contribute to Storm development efforts.
I have done some initial ground work in terms of going through the
contribution documents (
https://storm.apache.org/contribute/Contributing-to-Storm.html). I have
also setup development environment on IntelliJ and trying to browse through
the codebase.
As a next step, I picked up a newbie task to work on. I am trying to work on
https://issues.apache.org/jira/browse/STORM-1985
Few questions on this tasks:
1. Do we need to create a standalone tool or add this feature to an
existing tool.
2. Please confirm if this feature is only applicable to HA Nimbus.
I am tried to come up with pseudo code to verify solution approach.
---------------------------------------------------------------------------------------------
Iterator<String> getCorruptTopologyList(boolean shouldKillCorrupt) {
Map conf = ConfigUtil.readStormConfig();
ClientBlobStore clientBlobStore = Utils.getClientBlobStore(conf);
Iterator<String> blobStoreTopologyIds = clientBlobStore.listKeys();
Set<String> activeTopologyIds = new HashSet<>(Zookeeper.getChildren(zk,
conf.get (Config.STORM_ZOOKEEPER_ROOT) + ClusterUtils.STORMS_SUBTREE,
false));
Set<String> corruptTopologies =
differenceInTopologies(activeTopologyIds.iterator(),
blobStoreTopologyIds);
if(shouldKillCorrupt) {
Client client = NimbusClient.getConfiguredClient(conf).getClient();
Iterstor iter = corruptTopologies.iterator();
while(iter.hasNext()) {
client.killTopology(iter.next())
}
}
return corruptTopologies.iterator();
}
--------------------------------------
I appreciate if someone answer queries and guide me on my approach for
this task.
Thanks,
--Kamal
Re: Storm Development Assistance
Posted by Kamal B <bk...@gmail.com>.
Hi Bobby,
Thanks for your reply. I would use JIRA for any further discussion on this
task.
Regards,
--Kamal
On Mon, Aug 15, 2016 at 6:50 PM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:
> My guess is that this is going to be a new tool. I think we want to start
> looking into supporting a storm admin tool that will give users that have
> access to super user credentials access to directly examine/modify the
> state of a running storm cluster, zookeeper, blobstore, and local state. I
> don't expect you to do all of that work, but at least starting a place for
> a storm admin command in my opinion is the best place for this.
> I'll add this to the JIRA.
> - Bobby
>
> On Monday, August 15, 2016 3:29 AM, Kamal B <bk...@gmail.com>
> wrote:
>
>
> Hi All,
>
> I am newbie to Storm Community and intend to learn Storm and willing to
> contribute to Storm development efforts.
>
> I have done some initial ground work in terms of going through the
> contribution documents (
> https://storm.apache.org/contribute/Contributing-to-Storm.html). I have
> also setup development environment on IntelliJ and trying to browse through
> the codebase.
>
> As a next step, I picked up a newbie task to work on. I am trying to work
> on
> https://issues.apache.org/jira/browse/STORM-1985
>
> Few questions on this tasks:
> 1. Do we need to create a standalone tool or add this feature to an
> existing tool.
> 2. Please confirm if this feature is only applicable to HA Nimbus.
>
> I am tried to come up with pseudo code to verify solution approach.
>
> ------------------------------------------------------------
> ---------------------------------
> Iterator<String> getCorruptTopologyList(boolean shouldKillCorrupt) {
> Map conf = ConfigUtil.readStormConfig();
> ClientBlobStore clientBlobStore = Utils.getClientBlobStore(conf);
> Iterator<String> blobStoreTopologyIds = clientBlobStore.listKeys();
> Set<String> activeTopologyIds = new HashSet<>(Zookeeper.getChildren(zk,
> conf.get (Config.STORM_ZOOKEEPER_ROOT) + ClusterUtils.STORMS_SUBTREE,
> false));
> Set<String> corruptTopologies =
> differenceInTopologies(activeTopologyIds.iterator(),
> blobStoreTopologyIds);
>
> if(shouldKillCorrupt) {
> Client client = NimbusClient.getConfiguredClient(conf).getClient();
> Iterstor iter = corruptTopologies.iterator();
> while(iter.hasNext()) {
> client.killTopology(iter.next())
> }
> }
> return corruptTopologies.iterator();
> }
> --------------------------------------
> I appreciate if someone answer queries and guide me on my approach for
> this task.
>
> Thanks,
> --Kamal
>
>
>
>
Re: Storm Development Assistance
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
My guess is that this is going to be a new tool. I think we want to start looking into supporting a storm admin tool that will give users that have access to super user credentials access to directly examine/modify the state of a running storm cluster, zookeeper, blobstore, and local state. I don't expect you to do all of that work, but at least starting a place for a storm admin command in my opinion is the best place for this.
I'll add this to the JIRA.
- Bobby
On Monday, August 15, 2016 3:29 AM, Kamal B <bk...@gmail.com> wrote:
Hi All,
I am newbie to Storm Community and intend to learn Storm and willing to
contribute to Storm development efforts.
I have done some initial ground work in terms of going through the
contribution documents (
https://storm.apache.org/contribute/Contributing-to-Storm.html). I have
also setup development environment on IntelliJ and trying to browse through
the codebase.
As a next step, I picked up a newbie task to work on. I am trying to work on
https://issues.apache.org/jira/browse/STORM-1985
Few questions on this tasks:
1. Do we need to create a standalone tool or add this feature to an
existing tool.
2. Please confirm if this feature is only applicable to HA Nimbus.
I am tried to come up with pseudo code to verify solution approach.
---------------------------------------------------------------------------------------------
Iterator<String> getCorruptTopologyList(boolean shouldKillCorrupt) {
Map conf = ConfigUtil.readStormConfig();
ClientBlobStore clientBlobStore = Utils.getClientBlobStore(conf);
Iterator<String> blobStoreTopologyIds = clientBlobStore.listKeys();
Set<String> activeTopologyIds = new HashSet<>(Zookeeper.getChildren(zk,
conf.get (Config.STORM_ZOOKEEPER_ROOT) + ClusterUtils.STORMS_SUBTREE,
false));
Set<String> corruptTopologies =
differenceInTopologies(activeTopologyIds.iterator(),
blobStoreTopologyIds);
if(shouldKillCorrupt) {
Client client = NimbusClient.getConfiguredClient(conf).getClient();
Iterstor iter = corruptTopologies.iterator();
while(iter.hasNext()) {
client.killTopology(iter.next())
}
}
return corruptTopologies.iterator();
}
--------------------------------------
I appreciate if someone answer queries and guide me on my approach for
this task.
Thanks,
--Kamal