You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by abhishekagarwal87 <gi...@git.apache.org> on 2016/02/11 20:05:46 UTC
[GitHub] storm pull request: STORM-1248: port backtype.storm.messaging.load...
GitHub user abhishekagarwal87 opened a pull request:
https://github.com/apache/storm/pull/1097
STORM-1248: port backtype.storm.messaging.loader to java
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/abhishekagarwal87/storm thrift
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1097.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1097
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1248: port backtype.storm.messaging.load...
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1097#discussion_r52656904
--- Diff: storm-core/src/clj/org/apache/storm/daemon/worker.clj ---
@@ -487,11 +486,12 @@
)))))
(defn register-callbacks [worker]
- (log-message "Registering IConnectionCallbacks for " (:assignment-id worker) ":" (:port worker))
- (msg-loader/register-callback (:transfer-local-fn worker)
- (:receiver worker)
- (:storm-conf worker)
- (worker-context worker)))
+ (let [transfer-local-fn (:transfer-local-fn worker)
+ ^IConnection socket (:receiver worker)]
+ (log-message "Registering IConnectionCallbacks for " (:assignment-id worker) ":" (:port worker))
+ (.registerRecv socket (DeserializingConnectionCallback. (:storm-conf worker)
--- End diff --
Could we just inline `DeserializingConnectionCallback`? It's only used here and it'd be only a couple lines of Clojure. We're going to have to get rid of Java code that references Clojure like `DeserializingConnectionCallback` does anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1248: port backtype.storm.messaging.load...
Posted by abhishekagarwal87 <gi...@git.apache.org>.
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1097#discussion_r52658173
--- Diff: storm-core/src/clj/org/apache/storm/daemon/worker.clj ---
@@ -487,11 +486,12 @@
)))))
(defn register-callbacks [worker]
- (log-message "Registering IConnectionCallbacks for " (:assignment-id worker) ":" (:port worker))
- (msg-loader/register-callback (:transfer-local-fn worker)
- (:receiver worker)
- (:storm-conf worker)
- (worker-context worker)))
+ (let [transfer-local-fn (:transfer-local-fn worker)
+ ^IConnection socket (:receiver worker)]
+ (log-message "Registering IConnectionCallbacks for " (:assignment-id worker) ":" (:port worker))
+ (.registerRecv socket (DeserializingConnectionCallback. (:storm-conf worker)
--- End diff --
It wouldn't be that simple I suppose. There is a thread local KryoTupleDeserializer in the callback. I don't know how that will be implemented in clojure. I might end up porting more java code to clojure than other way around.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1248: port backtype.storm.messaging.load...
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1097#discussion_r52662537
--- Diff: storm-core/src/clj/org/apache/storm/daemon/worker.clj ---
@@ -487,11 +486,12 @@
)))))
(defn register-callbacks [worker]
- (log-message "Registering IConnectionCallbacks for " (:assignment-id worker) ":" (:port worker))
- (msg-loader/register-callback (:transfer-local-fn worker)
- (:receiver worker)
- (:storm-conf worker)
- (worker-context worker)))
+ (let [transfer-local-fn (:transfer-local-fn worker)
+ ^IConnection socket (:receiver worker)]
+ (log-message "Registering IConnectionCallbacks for " (:assignment-id worker) ":" (:port worker))
+ (.registerRecv socket (DeserializingConnectionCallback. (:storm-conf worker)
--- End diff --
@abhishekagarwal87 That's fine with me. It's true we'd have to translate it back to java, even if inlined.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1248: port backtype.storm.messaging.load...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/1097#issuecomment-183061460
+1 pending travis
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1248: port backtype.storm.messaging.load...
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1097#discussion_r52662362
--- Diff: storm-core/src/clj/org/apache/storm/daemon/worker.clj ---
@@ -487,11 +486,12 @@
)))))
(defn register-callbacks [worker]
- (log-message "Registering IConnectionCallbacks for " (:assignment-id worker) ":" (:port worker))
- (msg-loader/register-callback (:transfer-local-fn worker)
- (:receiver worker)
- (:storm-conf worker)
- (worker-context worker)))
+ (let [transfer-local-fn (:transfer-local-fn worker)
+ ^IConnection socket (:receiver worker)]
+ (log-message "Registering IConnectionCallbacks for " (:assignment-id worker) ":" (:port worker))
+ (.registerRecv socket (DeserializingConnectionCallback. (:storm-conf worker)
--- End diff --
@abhishekagarwal87 That's fine with me. It's true we'd have to translate it back to java, even if inlined.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1248: port backtype.storm.messaging.load...
Posted by abhishekagarwal87 <gi...@git.apache.org>.
Github user abhishekagarwal87 commented on the pull request:
https://github.com/apache/storm/pull/1097#issuecomment-183201855
Travis check passed. Ready for merge
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1248: port backtype.storm.messaging.load...
Posted by abhishekagarwal87 <gi...@git.apache.org>.
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1097#discussion_r52654035
--- Diff: storm-core/src/clj/org/apache/storm/daemon/worker.clj ---
@@ -461,11 +460,11 @@
)))))
(defn register-callbacks [worker]
- (log-message "Registering IConnectionCallbacks for " (:assignment-id worker) ":" (:port worker))
- (msg-loader/register-callback (:transfer-local-fn worker)
- (:receiver worker)
- (:storm-conf worker)
- (worker-context worker)))
+ (let [transfer-local-fn (:transfer-local-fn worker) ^IConnection socket (:receiver worker)]
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1248: port backtype.storm.messaging.load...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1097
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1248: port backtype.storm.messaging.load...
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1097#discussion_r52660183
--- Diff: storm-core/src/clj/org/apache/storm/daemon/worker.clj ---
@@ -487,11 +486,12 @@
)))))
(defn register-callbacks [worker]
- (log-message "Registering IConnectionCallbacks for " (:assignment-id worker) ":" (:port worker))
- (msg-loader/register-callback (:transfer-local-fn worker)
- (:receiver worker)
- (:storm-conf worker)
- (worker-context worker)))
+ (let [transfer-local-fn (:transfer-local-fn worker)
+ ^IConnection socket (:receiver worker)]
+ (log-message "Registering IConnectionCallbacks for " (:assignment-id worker) ":" (:port worker))
+ (.registerRecv socket (DeserializingConnectionCallback. (:storm-conf worker)
--- End diff --
```
(reify ThreadLocal
(initialValue [this] (KryoTupleDeserializer. (:storm-conf worker) (worker-context worker)))
```
That's it for the thread-local.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1248: port backtype.storm.messaging.load...
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on the pull request:
https://github.com/apache/storm/pull/1097#issuecomment-183056056
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1248: port backtype.storm.messaging.load...
Posted by abhishekagarwal87 <gi...@git.apache.org>.
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1097#discussion_r52662027
--- Diff: storm-core/src/clj/org/apache/storm/daemon/worker.clj ---
@@ -487,11 +486,12 @@
)))))
(defn register-callbacks [worker]
- (log-message "Registering IConnectionCallbacks for " (:assignment-id worker) ":" (:port worker))
- (msg-loader/register-callback (:transfer-local-fn worker)
- (:receiver worker)
- (:storm-conf worker)
- (worker-context worker)))
+ (let [transfer-local-fn (:transfer-local-fn worker)
+ ^IConnection socket (:receiver worker)]
+ (log-message "Registering IConnectionCallbacks for " (:assignment-id worker) ":" (:port worker))
+ (.registerRecv socket (DeserializingConnectionCallback. (:storm-conf worker)
--- End diff --
I am still not very convinced. I will port DeserializationCallBack to clojure, then we will again port it back to java. IMO it will be better to make it inline later, if needed.
@revans2 Since you wrote DeserializationCallback, was there any particular reason for writing it outside worker.clj e.g. intending to use it in other places?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1248: port backtype.storm.messaging.load...
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1097#discussion_r52651786
--- Diff: storm-core/src/clj/org/apache/storm/daemon/worker.clj ---
@@ -461,11 +460,11 @@
)))))
(defn register-callbacks [worker]
- (log-message "Registering IConnectionCallbacks for " (:assignment-id worker) ":" (:port worker))
- (msg-loader/register-callback (:transfer-local-fn worker)
- (:receiver worker)
- (:storm-conf worker)
- (worker-context worker)))
+ (let [transfer-local-fn (:transfer-local-fn worker) ^IConnection socket (:receiver worker)]
--- End diff --
`let` bindings should be one per line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---