You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by harshach <gi...@git.apache.org> on 2015/03/12 21:59:15 UTC
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
GitHub user harshach opened a pull request:
https://github.com/apache/storm/pull/464
STORM-615. Add REST API to upload topology.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/harshach/incubator-storm STORM-615-V2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/464.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #464
----
commit d18508e6def0000a4d790002d2512c2905aa6a76
Author: Sriharsha Chintalapani <ma...@harsha.io>
Date: 2015-03-12T20:58:31Z
STORM-615. Add REST API to upload topology.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88083784
@HeartSaVioR can you atleast try with a single node cluster before going and testing with docker
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88292492
@HeartSaVioR can you try with the latest PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-87911469
@harshach Maybe there could be broken configuration. I'll fix it and retry. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/464#discussion_r27486303
--- Diff: storm-core/src/clj/backtype/storm/ui/core.clj ---
@@ -503,6 +505,41 @@
(hashmap-to-persistent bolts))
spout-comp-summs bolt-comp-summs window id))))
+(defn validate-tplg-submit-params [params]
+ (let [tplg-jar-file (params :topologyJar)
+ tplg-config (if (not-nil? (params :topologyConfig)) (from-json (params :topologyConfig)))]
+ (cond
+ (nil? tplg-jar-file) {:valid false :error "missing topology jar file"}
+ (nil? tplg-config) {:valid false :error "missing topology config"}
+ (nil? (tplg-config "topologyMainClass")) {:valid false :error "topologyMainClass missing in topologyConfig"}
+ :else {:valid true})))
+
+(defn run-tplg-submit-cmd [tplg-jar-file tplg-config]
+ (let [tplg-main-class (if (not-nil? tplg-config) (trim (tplg-config "topologyMainClass")))
+ tplg-main-class-args (if (not-nil? tplg-config) (clojure.string/join " " (tplg-config "topologyMainClassArgs")))
+ tplg-jvm-opts (if (not-nil? tplg-config) (clojure.string/join " " (tplg-config "topologyJvmOpts")))
+ storm-home (System/getProperty "storm.home")
+ storm-log-dir (if (not-nil? (*STORM-CONF* "storm.log.dir")) (*STORM-CONF* "storm.log.dir")
+ (str storm-home file-path-separator "logs"))
+ storm-libs (str storm-home file-path-separator "lib" file-path-separator "*")
--- End diff --
@harshach
Replacing this line with ```storm-libs (clojure.string/join ":" [(str storm-home file-path-separator "lib" file-path-separator "*") (str storm-home file-path-separator "conf")])``` make it works.
But it's just a workaround cause users may set STORM_CONF_DIR.
```
if STORM_CONF_DIR == None:
CLUSTER_CONF_DIR = os.path.join(STORM_DIR, "conf")
else:
CLUSTER_CONF_DIR = STORM_CONF_DIR
if (not os.path.isfile(os.path.join(USER_CONF_DIR, "storm.yaml"))):
USER_CONF_DIR = CLUSTER_CONF_DIR
```
USER_CONF_DIR will become one of classpath. storm-libs should follow this rule, except applying ~/.storm directory cause this is cluster-side operation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by salimane <gi...@git.apache.org>.
Github user salimane commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-201204396
@harshach i can't seem to find the upload rest api in the rest api docs anymore. Has anything changed ?
Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88120666
@harshach Maybe you can reproduce this issue by doing some modification to STORM_DIR/conf/storm.yaml and do uploadTopology, and see topology's configuration.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88119894
@harshach I make it works with some fix, but it's just a workaround so some more works should be taken place. Or if we ensure that UI node has python, it's better to call storm.py instead of JVM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88302906
@harshach It works! Great! +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-87903618
@HeartSaVioR I didn't tried with the docker. But looking at your error from your earlier request it doesn't look storm UI is not able to reach the nimbus. Did you try logging into storm UI docker instance and tried to deploy a topology?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-87891309
@HeartSaVioR not sure I might be looking at some other file. I downloaded the file linked and ran a single node cluster don't see any issue with uploadTopology. Can you give it try again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-86966516
@HeartSaVioR thanks for trying it out. Did you try the curl command in the docs to upload the topology? antiForgery there for to deter malicious api calls from cross site scripting. All you need is to ask the api for a token using /api/v1/token and store it in a cookie and for any subsequent calls you need to pass this token in x-csrf-header in http headers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88130360
@harshach Btw, I'm sorry to being confused due to my short English.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88122113
@HeartSaVioR "UI node has python, it's better to call storm.py instead of JVM." . Can you elaborate on this without python installed on the UI node you won't be able to start storm UI daemon itself.
What you need to do to make it work can you add those details
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88117340
@HeartSaVioR do give details on you docker setup. I'll try to reproduce the issue. I can't reproduce in the vagrant setup or single node.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-87098232
@harshach Oh, I'm sorry I missed token api.
Maybe it's better to modify documentation about ```Cross site request forgery (CSRF) prevention in POST requests``` to call token api to get antiForgery.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88304330
Btw, I've uploaded new image "heartsavior/storm:WIP-STORM-615-V2" and also push https://github.com/heartsavior/storm-docker/tree/WIP-STORM-615-V2 to apply new image.
You can play it with my docker & fig.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-85670825
I am +1, I think in order to make it work in secure mode we will need to use the doAs feature .We can ask the users to use the StormSubmitter's submitTopology method that accepts a doAs Param and ensure that they pass their userName.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88151079
@HeartSaVioR Thanks for the details. You did find a valid bug if the storm ui is not running on the same host as nimbus the same issue you are seeing happening as it finds the default.yaml nimbus.host and port not the one in storm.home/conf/storm.yaml. Let me send the updated pr.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-87892365
@harshach OK. I'll try again.
Btw, could you try uploading with multi-nodes cluster? Maybe wurstmeister docker seems fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-86873291
When we're trying to upload first Topology with this API, I'm afraid we can't get antiForgery since no topology is submitted.
(Actually I'm confused about antiForgery. Does Storm allow getting antiForgery from Topology A and use it with Topology B?)
Please correct me if I'm wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-87623121
I just build dist from your branch, and modifies wurstmeister docker to apply your branch during build.
Binary dist : https://www.dropbox.com/s/ldm9t76cm43zqdf/apache-storm-0.10.0-SNAPSHOT.tar.gz?dl=0
I gave a try with using a token and ring-session (cookie) but failed.
```
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
Date: Mon, 30 Mar 2015 10:10:31 GMT
Cache-Control: no-cache, no-store
Content-Type: application/json;charset=utf-8
Content-Length: 55
Server: Jetty(7.6.13.v20130916)
{"error":"Failed to deploy topology","status":"failed"}
```
Here's UI log.
```
2015-03-30 10:10:42 b.s.u.core [INFO] topology cmd {:exit 1, :out "434 [main] INFO backtype.storm.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -5707587195506149327:-9132844115790108814\n439 [main] INFO backtype.storm.security.auth.AuthUtils - Got AutoCreds []\n450 [main] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5]\n", :err "Exception in thread \"main\" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused\n\tat backtype.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:59)\n\tat backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:51)\n\tat backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103)\n\tat backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:72)\n\tat backtype.storm.utils.NimbusClient.<init>(NimbusC
lient.java:62)\n\tat backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:46)\n\tat backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225)\n\tat backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)\n\tat backtype.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:307)\n\tat backtype.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:288)\n\tat storm.starter.WordCountTopology.main(WordCountTopology.java:94)\nCaused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused\n\tat org.apache.thrift7.transport.TSocket.open(TSocket.java:187)\n\tat org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)\n\tat backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:102)\n\tat backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48)\n\t... 9 more\nCaused by: java.net.ConnectE
xception: Connection refused\n\tat java.net.PlainSocketImpl.socketConnect(Native Method)\n\tat java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)\n\tat java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)\n\tat java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)\n\tat java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)\n\tat java.net.Socket.connect(Socket.java:579)\n\tat org.apache.thrift7.transport.TSocket.open(TSocket.java:182)\n\t... 12 more\n"}
2015-03-30 10:11:14 b.s.thrift [INFO] Connecting to Nimbus at 172.17.0.62:6627 as user:
...
2015-03-30 10:11:14 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5]
...
```
Seems like UI server can reach Nimbus Thrift port, I've confirmed with telnet.
Does 0.10.x run into security mode as default? I didn't modify configuration file from 0.9.3.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/storm/pull/464#discussion_r27066660
--- Diff: storm-core/src/clj/backtype/storm/ui/core.clj ---
@@ -982,7 +1019,23 @@
(.killTopologyWithOpts nimbus name options)
(log-message "Killing topology '" name "' with wait time: " wait-time " secs")))
(json-response (topology-op-response id "kill") (m "callback")))
-
+ (POST "/api/v1/uploadTopology" [:as {:keys [cookies servlet-request]} id & params]
+ (let [valid-tplg (validate-tplg-submit-params params)
+ valid (valid-tplg :valid)]
+ (log-message "valid topology " valid-tplg)
+ (if (= valid true)
+ (let [tplg-file-data (params :topologyJar)
+ tplg-temp-file (tplg-file-data :tempfile)
--- End diff --
I am guessing these things get set as part of multipart wrapper magic?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach closed the pull request at:
https://github.com/apache/storm/pull/464
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88126334
@HeartSaVioR Are you saying java command is not in path. I would like to know whats the error you are facing. Its not able to find the storm.conf.file and unable to connect to nimbus.host? . Let me try changing it to storm.py and see if that fixes the issue and also you are using https://github.com/wurstmeister/storm-docker . Can it reproduce the issue if I use the dropbox link you provided?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-86993357
@HeartSaVioR @Parth-Brahmbhatt it looks like anti-forgery requires users to store the ring-session id and along with passing x-csrf-token in headers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88137143
@HeartSaVioR As I said earlier if you can test it in single node installation and see the same issue. I can look at it. It looks like you are hacking around docker setup and I am not sure how much of that it contributing to the issue you are seeing. If you want to test it in multi node env I provided Vagrant instructions above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-87889571
@harshach Harsha, I re-download my link and extract it, and can't find two clojure files from lib directory.
```
$ find . -name "clojure*"
./lib/clojure-1.6.0.jar
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88043963
@harshach
Maybe we can check this.
I'm running Storm with STORM_HOME/conf/storm.yaml, and I don't make ~/.storm directory.
```
root 14 2.5 7.2 3264920 149328 ? Sl 09:53 1:33 java -server -Dstorm.options= -Dstorm.home=/opt/apache-storm-0.10.0-SNAPSHOT -Dstorm.log.dir=/opt/apache-storm-0.10.0-SNAPSHOT/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/apache-storm-0.10.0-SNAPSHOT/lib/cheshire-5.3.1.jar:...
```
When we submit Topology by API, classpath is ```storm-libs (str storm-home file-path-separator "lib" file-path-separator "*")```, which is only STORM_HOME/lib/*, and jar file itself.
But storm.py finds effective configuration directory and add it to classpath, which is different from current implementation.
```
if STORM_CONF_DIR == None:
CLUSTER_CONF_DIR = os.path.join(STORM_DIR, "conf")
else:
CLUSTER_CONF_DIR = STORM_CONF_DIR
if (not os.path.isfile(os.path.join(USER_CONF_DIR, "storm.yaml"))):
USER_CONF_DIR = CLUSTER_CONF_DIR
```
If we need to use cluster-wide effective configuration while submitting topology, we should include effective configuration directory.
Btw, other APIs (which use ```with-nimbus``` macro) seems to run fine. I'm curious it could be represented to ```with-nimbus```.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/464#discussion_r27067015
--- Diff: storm-core/src/clj/backtype/storm/ui/core.clj ---
@@ -982,7 +1019,23 @@
(.killTopologyWithOpts nimbus name options)
(log-message "Killing topology '" name "' with wait time: " wait-time " secs")))
(json-response (topology-op-response id "kill") (m "callback")))
-
+ (POST "/api/v1/uploadTopology" [:as {:keys [cookies servlet-request]} id & params]
+ (let [valid-tplg (validate-tplg-submit-params params)
+ valid (valid-tplg :valid)]
+ (log-message "valid topology " valid-tplg)
+ (if (= valid true)
+ (let [tplg-file-data (params :topologyJar)
+ tplg-temp-file (tplg-file-data :tempfile)
--- End diff --
@Parth-Brahmbhatt yep its being handled by multi-part
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88083643
@HeartSaVioR I am not sure I follow. The examples I gave and the tests I ran I never created ~/.storm/storm.yaml. I am not sure if you are testing it right. Once the jar is uploaded storm ui does a shell command for storm jar and it effectively finds the STORM_HOME/conf/storm.yaml as the cluster config .
If storm.conf.file is not passed it will look under STORM_HOME/conf/storm.yaml
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-87874852
@HeartSaVioR Thanks for testing it out. I ran few tests with your dropbox link. For starters your apache-storm-0.10-SNAPSHOT contains two different versions of clojure libs under lib folder. Delete clojure-1.5.1.jar and started a single node cluster.
curl -i --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:8080/api/v1/token
The above call stores the ring-session id in the cookiejar.txt
curl -i --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt -X POST -H 'x-csrf-token: jz8utSXi0PM1timC5phUelnMaYxcRGba8PEHWL1yU2WEQL59zwijyP4GjWv/uhDWUJAD7tS50vyF8R4E' -F topologyConfig='{"topologyMainClass": "storm.starter.WordCountTopology", "topologyMainClassArgs": ["wordcount1"]}' -F topologyJar=@examples/storm-starter/storm-starter-topologies-0.10.0-SNAPSHOT.jar http://localhost:8080/api/v1/uploadTopology
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
Date: Mon, 30 Mar 2015 23:18:45 GMT
Cache-Control: no-cache, no-store
Content-Type: application/json;charset=utf-8
Content-Length: 20
Server: Jetty(7.6.13.v20130916)
{"status":"success"}

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88254339
@harshach Thanks for understanding. :)
Just calling ```storm.py jar``` may work well, but still need to test more.
I hope I can see this feature ASAP in a released version as I wanna use control features of topology.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-87910582
@HeartSaVioR I did tested multi-node cluster before opening a PR. Ran another test with your apache-storm.tar.gz using the vagrant setup.
https://github.com/harshach/storm-vagrant
you can drop in the apache-storm.tar.gz . Make sure bin/storm has execute permissions the link you've doesn't and convert it into zip file.
vagrant up will create multinode storm cluster(1 zk, 1 nimbus, 2 supervisors)
curl -i --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt http://nimbus.witzend.com:8080/api/v1/token ◉ ◼◼◼◼◼◼◼◼◻◻
HTTP/1.1 200 OK
Date: Tue, 31 Mar 2015 02:46:05 GMT
Set-Cookie: ring-session=64ce2e4d-d62b-4e45-8cd0-4c778948d7b5;Path=/
Cache-Control: no-cache, no-store
Content-Type: application/json;charset=utf-8
Content-Length: 104
Server: Jetty(7.6.13.v20130916)
{"antiForgeryToken": "wMYtcaNy8yA90jacPXw+BK4vE0w6c748shhtvr6HYeR1lhpFsO+1k2CSIMq+GUqpTnuwwn8/tDUbPyEK"}
Here is the output of uploadTopology
curl -i --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt -X POST -H 'x-csrf-token: wMYtcaNy8yA90jacPXw+BK4vE0w6c748shhtvr6HYeR1lhpFsO+1k2CSIMq+GUqpTnuwwn8/tDUbPyEK' -F topologyConfig='{"topologyMainClass": "storm.starter.WordCountTopology", "topologyMainClassArgs": ["wordcount1"]}' -F topologyJar=@examples/storm-starter/storm-starter-topologies-0.10.0-SNAPSHOT.jar http://nimbus.witzend.com:8080/api/v1/uploadTopology
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
Date: Tue, 31 Mar 2015 02:48:38 GMT
Cache-Control: no-cache, no-store
Content-Type: application/json;charset=utf-8
Content-Length: 20
Server: Jetty(7.6.13.v20130916)
{"status":"success"}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/464
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by harshach <gi...@git.apache.org>.
GitHub user harshach reopened a pull request:
https://github.com/apache/storm/pull/464
STORM-615. Add REST API to upload topology.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/harshach/incubator-storm STORM-615-V2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/464.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #464
----
commit d18508e6def0000a4d790002d2512c2905aa6a76
Author: Sriharsha Chintalapani <ma...@harsha.io>
Date: 2015-03-12T20:58:31Z
STORM-615. Add REST API to upload topology.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88123831
@harshach Oh, sorry I was mistaken. Then how about calling ```storm.py jar ~~~``` instead of java?
storm.py takes care of bootstrapping phase, and add proper classpaths if it needs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88140228
Important point is that in docker setup it writes Nimbus IP and PORT to storm.yaml.
```
root@433d02f34915:/opt/apache-storm-0.10.0-SNAPSHOT/conf# cat storm.yaml
storm.zookeeper.servers:
- "172.17.0.18"
nimbus.host: "172.17.0.19"
nimbus.thrift.port: 6627
drpc.servers:
- "172.17.0.19"
drpc.port: 3772
drpc.invocations.port: 3773
storm.local.hostname: 172.17.0.21
```
I'm thinking it may raise an issue if storm.yaml is differ from defaults.yaml.
TopologySubmitter could use default values so if cluster is set up to be no problem with default config, we can't reproduce this issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88039031
@harshach
I'm seeing failed message again. I ran a ```storm jar``` from UI container's shell and it works well.
UI shows topology information nicely, which may tell connection between UI and Nimbus Thrift is valid.
Killing topology with API also works well.
Below is error logs from UI container.
```
2015-03-31 09:54:29 b.s.u.core [INFO] topology cmd {:exit 1, :out "428 [main] INFO backtype.storm.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -5597653866246168988:-6838346593417745091\n432 [main] INFO backtype.storm.security.auth.AuthUtils - Got AutoCreds []\n442 [main] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5]\n", :err "Exception in thread \"main\" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused\n\tat backtype.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:59)\n\tat backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:51)\n\tat backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103)\n\tat backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:72)\n\tat backtype.storm.utils.NimbusClient.<init>(NimbusC
lient.java:62)\n\tat backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:46)\n\tat backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225)\n\tat backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)\n\tat backtype.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:307)\n\tat backtype.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:288)\n\tat storm.starter.WordCountTopology.main(WordCountTopology.java:94)\nCaused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused\n\tat org.apache.thrift7.transport.TSocket.open(TSocket.java:187)\n\tat org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)\n\tat backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:102)\n\tat backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48)\n\t... 9 more\nCaused by: java.net.ConnectE
xception: Connection refused\n\tat java.net.PlainSocketImpl.socketConnect(Native Method)\n\tat java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)\n\tat java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)\n\tat java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)\n\tat java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)\n\tat java.net.Socket.connect(Socket.java:579)\n\tat org.apache.thrift7.transport.TSocket.open(TSocket.java:182)\n\t... 12 more\n"}
```
I tried it into UI container shell and failed again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88138647
@harshach @HeartSaVioR FYI, I tested this with upmerge to master and it works fine on my mac. No configs needed to be added, no storm.yaml created. Worked fine out of the box on a single host setup.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88129861
@harshach
Let me elaborate current issue.
1. UI process is fine. UI can communicate with Nimbus.
2. Please see full stack trace. Exception occurred from storm.starter.WordCountTopology.main, which implies root cause is ```TopologySubmitter cannot connect Nimbus thrift port```. We already know UI can communicate with Nimbus.
2015-03-30 10:10:42 b.s.u.core [INFO] topology cmd {:exit 1, :out "434 [main] INFO backtype.storm.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -5707587195506149327:-9132844115790108814\n439 [main] INFO backtype.storm.security.auth.AuthUtils - Got AutoCreds []\n450 [main] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5]\n", :err "Exception in thread \"main\" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused\n\tat backtype.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:59)\n\tat backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:51)\n\tat backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103)\n\tat backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:72)\n\tat backtype.storm.utils.NimbusClient.<init>(NimbusC
lient.java:62)\n\tat backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:46)\n\tat backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225)\n\tat backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)\n\tat backtype.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:307)\n\tat backtype.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:288)\n\tat storm.starter.WordCountTopology.main(WordCountTopology.java:94)\nCaused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused\n\tat org.apache.thrift7.transport.TSocket.open(TSocket.java:187)\n\tat org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)\n\tat backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:102)\n\tat backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48)\n\t... 9 more\nCaused by: java.net.ConnectE
xception: Connection refused\n\tat java.net.PlainSocketImpl.socketConnect(Native Method)\n\tat java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)\n\tat java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)\n\tat java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)\n\tat java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)\n\tat java.net.Socket.connect(Socket.java:579)\n\tat org.apache.thrift7.transport.TSocket.open(TSocket.java:182)\n\t... 12 more\n"}
3. Adding ```(log-message (str storm-libs ":" tplg-jar-file))``` before ```(log-message "topology cmd " tplg-cmd-response)``` helps debugging what classpath is applied with TopologySubmitter. Without my patch it just prints out ```/opt/apache-storm-0.10.0-SNAPSHOT/lib/*:/tmp/storm-starter-topologies-0.10.0-SNAPSHOT.jar```, which is not same as ```storm jar```.
4. Seems like TopologySubmitter cannot see storm.conf during uploadTopology. It's another way to reproduce.
I'll try to make docker image, but I'm having hard time to reflect new binary dist to docker image, so I manually connect UI container by ssh and replace storm-core jar file to continue.
My patched storm-core is https://www.dropbox.com/s/r7jb0w5uai5azjr/storm-core-0.10.0-SNAPSHOT.jar?dl=0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88056652
@harshach Oh, it would be very hard to submit topology directly (from jetty), so your approach is enough reasonable. Maybe loading cluster-wide configuration directory would be enough. Same location as UI uses)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-615. Add REST API to upload topology.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/464#issuecomment-88138160
@harshach OK, I'd like to reproduce current issue with Vagrant setup. Could you explain instruction?
Btw, I've uploaded my docker setup to https://github.com/HeartSaVioR/storm-docker/tree/WIP-STORM-615-V2
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---