You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by knusbaum <gi...@git.apache.org> on 2014/06/27 20:39:20 UTC
[GitHub] incubator-storm pull request: [STORM-375] Smarter downloading of a...
GitHub user knusbaum opened a pull request:
https://github.com/apache/incubator-storm/pull/167
[STORM-375] Smarter downloading of assignments by supervisors and workers
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/knusbaum/incubator-storm community-zk-assignments
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-storm/pull/167.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #167
----
commit b27b8c4fa015f19db6a087c3a565a703b39dc517
Author: Kyle Nusbaum <kn...@yahoo-inc.com>
Date: 2014-06-26T21:13:56Z
Demo conservative ZK assignments (lots of logging)
commit 41e5b91a71a609818c8ca8b0291ea292710dd468
Author: Kyle Nusbaum <kn...@yahoo-inc.com>
Date: 2014-06-27T17:52:52Z
Added smart assignment-refreshing for workers.
commit 516b33317d49c8687984a77de0764f0b25333e22
Author: Kyle Nusbaum <kn...@yahoo-inc.com>
Date: 2014-06-27T17:54:05Z
Merge remote-tracking branch 'community/master' into community-zk-assignments
commit 310e09e1a594ef44785e7aeb4b2fd5e4299a303b
Author: Kyle Nusbaum <kn...@yahoo-inc.com>
Date: 2014-06-26T21:40:29Z
Cleaned up and ready for pull.
commit 6d560cb6b7c1b120f30d026a50042c768ab90692
Author: Kyle Nusbaum <kn...@yahoo-inc.com>
Date: 2014-06-27T18:29:03Z
Cleanup for pull
commit f4e6c1933ff3f2e8ded52b9ceed389362ad41796
Author: Kyle Nusbaum <kn...@yahoo-inc.com>
Date: 2014-06-27T18:31:07Z
One more bit of cleanup
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: [STORM-375] Smarter downloading of a...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-storm/pull/167
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: [STORM-375] Smarter downloading of a...
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on the pull request:
https://github.com/apache/incubator-storm/pull/167#issuecomment-48786418
Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: [STORM-375] Smarter downloading of a...
Posted by nathanmarz <gi...@git.apache.org>.
Github user nathanmarz commented on the pull request:
https://github.com/apache/incubator-storm/pull/167#issuecomment-48548791
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: [STORM-375] Smarter downloading of a...
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on the pull request:
https://github.com/apache/incubator-storm/pull/167#issuecomment-48431035
@nathanmarz I believe this is ready for another look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: [STORM-375] Smarter downloading of a...
Posted by nathanmarz <gi...@git.apache.org>.
Github user nathanmarz commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/167#discussion_r14314682
--- Diff: storm-core/src/clj/backtype/storm/daemon/supervisor.clj ---
@@ -297,7 +307,10 @@
^ISupervisor isupervisor (:isupervisor supervisor)
^LocalState local-state (:local-state supervisor)
sync-callback (fn [& ignored] (.add event-manager this))
- assignments-snapshot (assignments-snapshot storm-cluster-state sync-callback)
+ assignment-versions (.get local-state LS-ASSIGNMENT-VERSIONS)
--- End diff --
This shouldn't be stored in the local state as there is no need to. Just keep this information purely in memory. It's ok if the supervisor has to re-download everything on death because deaths are rare. Persisted state should be minimized as a general rule.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: [STORM-375] Smarter downloading of a...
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the pull request:
https://github.com/apache/incubator-storm/pull/167#issuecomment-48556385
Looks good to me.
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: [STORM-375] Smarter downloading of a...
Posted by nathanmarz <gi...@git.apache.org>.
Github user nathanmarz commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/167#discussion_r14314628
--- Diff: storm-core/src/clj/backtype/storm/daemon/worker.clj ---
@@ -174,11 +175,13 @@
)
:timer-name timer-name))
+(def assignment-versions (atom {}))
--- End diff --
This shouldn't be global but part of the worker data
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: [STORM-375] Smarter downloading of a...
Posted by xiaokang <gi...@git.apache.org>.
Github user xiaokang commented on the pull request:
https://github.com/apache/incubator-storm/pull/167#issuecomment-49539879
Good idea to reduce ZK read load. We will try it in busy cluster. Thanks @knusbaum!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: [STORM-375] Smarter downloading of a...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/incubator-storm/pull/167#issuecomment-48783596
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: [STORM-375] Smarter downloading of a...
Posted by nathanmarz <gi...@git.apache.org>.
Github user nathanmarz commented on the pull request:
https://github.com/apache/incubator-storm/pull/167#issuecomment-47398962
Good idea, definitely want to merge this in. I made a few comments on some things that need to change in the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---