You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by sr...@apache.org on 2015/11/04 21:05:31 UTC
[1/3] storm git commit: Fixed incubation URLs
Repository: storm
Updated Branches:
refs/heads/master f3ed08b9f -> 4a57e500b
Fixed incubation URLs
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/acca5720
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/acca5720
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/acca5720
Branch: refs/heads/master
Commit: acca57205c731a48b5285bfe85c5ea55e6222108
Parents: c65b07b
Author: Julien Nioche <ju...@digitalpebble.com>
Authored: Thu Oct 29 12:37:18 2015 +0000
Committer: Julien Nioche <ju...@digitalpebble.com>
Committed: Thu Oct 29 12:37:18 2015 +0000
----------------------------------------------------------------------
DISCLAIMER | 10 ------
bin/storm.py | 14 ++++----
.../Message-passing-implementation.md | 34 ++++++++++----------
3 files changed, 24 insertions(+), 34 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/acca5720/DISCLAIMER
----------------------------------------------------------------------
diff --git a/DISCLAIMER b/DISCLAIMER
deleted file mode 100644
index 8638904..0000000
--- a/DISCLAIMER
+++ /dev/null
@@ -1,10 +0,0 @@
-Apache Storm is an effort undergoing incubation at the Apache Software
-Foundation (ASF), sponsored by the Apache Incubator PMC.
-
-Incubation is required of all newly accepted projects until a further review
-indicates that the infrastructure, communications, and decision making process
-have stabilized in a manner consistent with other successful ASF projects.
-
-While incubation status is not necessarily a reflection of the completeness
-or stability of the code, it does indicate that the project has yet to be
-fully endorsed by the ASF.
http://git-wip-us.apache.org/repos/asf/storm/blob/acca5720/bin/storm.py
----------------------------------------------------------------------
diff --git a/bin/storm.py b/bin/storm.py
index 1b9617e..e91b1e9 100755
--- a/bin/storm.py
+++ b/bin/storm.py
@@ -218,7 +218,7 @@ def jar(jarfile, klass, *args):
Runs the main method of class with the specified arguments.
The storm jars and configs in ~/.storm are put on the classpath.
The process is configured so that StormSubmitter
- (http://storm.incubator.apache.org/apidocs/backtype/storm/StormSubmitter.html)
+ (http://storm.apache.org/apidocs/backtype/storm/StormSubmitter.html)
will upload the jar at topology-jar-path when the topology is submitted.
"""
exec_storm_class(
@@ -432,7 +432,7 @@ def nimbus(klass="backtype.storm.daemon.nimbus"):
supervision with a tool like daemontools or monit.
See Setting up a Storm cluster for more information.
- (http://storm.incubator.apache.org/documentation/Setting-up-a-Storm-cluster)
+ (http://storm.apache.org/documentation/Setting-up-a-Storm-cluster)
"""
cppaths = [CLUSTER_CONF_DIR]
jvmopts = parse_args(confvalue("nimbus.childopts", cppaths)) + [
@@ -453,7 +453,7 @@ def supervisor(klass="backtype.storm.daemon.supervisor"):
under supervision with a tool like daemontools or monit.
See Setting up a Storm cluster for more information.
- (http://storm.incubator.apache.org/documentation/Setting-up-a-Storm-cluster)
+ (http://storm.apache.org/documentation/Setting-up-a-Storm-cluster)
"""
cppaths = [CLUSTER_CONF_DIR]
jvmopts = parse_args(confvalue("supervisor.childopts", cppaths)) + [
@@ -475,7 +475,7 @@ def ui():
should be run under supervision with a tool like daemontools or monit.
See Setting up a Storm cluster for more information.
- (http://storm.incubator.apache.org/documentation/Setting-up-a-Storm-cluster)
+ (http://storm.apache.org/documentation/Setting-up-a-Storm-cluster)
"""
cppaths = [CLUSTER_CONF_DIR]
jvmopts = parse_args(confvalue("ui.childopts", cppaths)) + [
@@ -497,7 +497,7 @@ def logviewer():
tool like daemontools or monit.
See Setting up a Storm cluster for more information.
- (http://storm.incubator.apache.org/documentation/Setting-up-a-Storm-cluster)
+ (http://storm.apache.org/documentation/Setting-up-a-Storm-cluster)
"""
cppaths = [CLUSTER_CONF_DIR]
jvmopts = parse_args(confvalue("logviewer.childopts", cppaths)) + [
@@ -518,7 +518,7 @@ def drpc():
with a tool like daemontools or monit.
See Distributed RPC for more information.
- (http://storm.incubator.apache.org/documentation/Distributed-RPC)
+ (http://storm.apache.org/documentation/Distributed-RPC)
"""
cppaths = [CLUSTER_CONF_DIR]
jvmopts = parse_args(confvalue("drpc.childopts", cppaths)) + [
@@ -585,7 +585,7 @@ def print_commands():
"""Print all client commands and link to documentation"""
print("Commands:\n\t" + "\n\t".join(sorted(COMMANDS.keys())))
print("\nHelp: \n\thelp \n\thelp <command>")
- print("\nDocumentation for the storm client can be found at http://storm.incubator.apache.org/documentation/Command-line-client.html\n")
+ print("\nDocumentation for the storm client can be found at http://storm.apache.org/documentation/Command-line-client.html\n")
print("Configs can be overridden using one or more -c flags, e.g. \"storm list -c nimbus.host=nimbus.mycompany.com\"\n")
def print_usage(command=None):
http://git-wip-us.apache.org/repos/asf/storm/blob/acca5720/docs/documentation/Message-passing-implementation.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Message-passing-implementation.md b/docs/documentation/Message-passing-implementation.md
index e17fd3f..e498224 100644
--- a/docs/documentation/Message-passing-implementation.md
+++ b/docs/documentation/Message-passing-implementation.md
@@ -8,23 +8,23 @@ documentation: true
This page walks through how emitting and transferring tuples works in Storm.
- Worker is responsible for message transfer
- - `refresh-connections` is called every "task.refresh.poll.secs" or whenever assignment in ZK changes. It manages connections to other workers and maintains a mapping from task -> worker [code](https://github.com/apache/incubator-storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L123)
- - Provides a "transfer function" that is used by tasks to send tuples to other tasks. The transfer function takes in a task id and a tuple, and it serializes the tuple and puts it onto a "transfer queue". There is a single transfer queue for each worker. [code](https://github.com/apache/incubator-storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L56)
- - The serializer is thread-safe [code](https://github.com/apache/incubator-storm/blob/0.7.1/src/jvm/backtype/storm/serialization/KryoTupleSerializer.java#L26)
- - The worker has a single thread which drains the transfer queue and sends the messages to other workers [code](https://github.com/apache/incubator-storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L185)
- - Message sending happens through this protocol: [code](https://github.com/apache/incubator-storm/blob/0.7.1/src/clj/backtype/storm/messaging/protocol.clj)
- - The implementation for distributed mode uses ZeroMQ [code](https://github.com/apache/incubator-storm/blob/0.7.1/src/clj/backtype/storm/messaging/zmq.clj)
- - The implementation for local mode uses in memory Java queues (so that it's easy to use Storm locally without needing to get ZeroMQ installed) [code](https://github.com/apache/incubator-storm/blob/0.7.1/src/clj/backtype/storm/messaging/local.clj)
+ - `refresh-connections` is called every "task.refresh.poll.secs" or whenever assignment in ZK changes. It manages connections to other workers and maintains a mapping from task -> worker [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L123)
+ - Provides a "transfer function" that is used by tasks to send tuples to other tasks. The transfer function takes in a task id and a tuple, and it serializes the tuple and puts it onto a "transfer queue". There is a single transfer queue for each worker. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L56)
+ - The serializer is thread-safe [code](https://github.com/apache/storm/blob/0.7.1/src/jvm/backtype/storm/serialization/KryoTupleSerializer.java#L26)
+ - The worker has a single thread which drains the transfer queue and sends the messages to other workers [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L185)
+ - Message sending happens through this protocol: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/messaging/protocol.clj)
+ - The implementation for distributed mode uses ZeroMQ [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/messaging/zmq.clj)
+ - The implementation for local mode uses in memory Java queues (so that it's easy to use Storm locally without needing to get ZeroMQ installed) [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/messaging/local.clj)
- Receiving messages in tasks works differently in local mode and distributed mode
- - In local mode, the tuple is sent directly to an in-memory queue for the receiving task [code](https://github.com/apache/incubator-storm/blob/master/src/clj/backtype/storm/messaging/local.clj#L21)
- - In distributed mode, each worker listens on a single TCP port for incoming messages and then routes those messages in-memory to tasks. The TCP port is called a "virtual port", because it receives [task id, message] and then routes it to the actual task. [code](https://github.com/apache/incubator-storm/blob/master/src/clj/backtype/storm/daemon/worker.clj#L204)
- - The virtual port implementation is here: [code](https://github.com/apache/incubator-storm/blob/master/src/clj/zilch/virtual_port.clj)
- - Tasks listen on an in-memory ZeroMQ port for messages from the virtual port [code](https://github.com/apache/incubator-storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L201)
- - Bolts listen here: [code](https://github.com/apache/incubator-storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L489)
- - Spouts listen here: [code](https://github.com/apache/incubator-storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L382)
+ - In local mode, the tuple is sent directly to an in-memory queue for the receiving task [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/messaging/local.clj#L21)
+ - In distributed mode, each worker listens on a single TCP port for incoming messages and then routes those messages in-memory to tasks. The TCP port is called a "virtual port", because it receives [task id, message] and then routes it to the actual task. [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/worker.clj#L204)
+ - The virtual port implementation is here: [code](https://github.com/apache/storm/blob/master/src/clj/zilch/virtual_port.clj)
+ - Tasks listen on an in-memory ZeroMQ port for messages from the virtual port [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L201)
+ - Bolts listen here: [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L489)
+ - Spouts listen here: [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L382)
- Tasks are responsible for message routing. A tuple is emitted either to a direct stream (where the task id is specified) or a regular stream. In direct streams, the message is only sent if that bolt subscribes to that direct stream. In regular streams, the stream grouping functions are used to determine the task ids to send the tuple to.
- - Tasks have a routing map from {stream id} -> {component id} -> {stream grouping function} [code](https://github.com/apache/incubator-storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L198)
- - The "tasks-fn" returns the task ids to send the tuples to for either regular stream emit or direct stream emit [code](https://github.com/apache/incubator-storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L207)
+ - Tasks have a routing map from {stream id} -> {component id} -> {stream grouping function} [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L198)
+ - The "tasks-fn" returns the task ids to send the tuples to for either regular stream emit or direct stream emit [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L207)
- After getting the output task ids, bolts and spouts use the transfer-fn provided by the worker to actually transfer the tuples
- - Bolt transfer code here: [code](https://github.com/apache/incubator-storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L429)
- - Spout transfer code here: [code](https://github.com/apache/incubator-storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L329)
+ - Bolt transfer code here: [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L429)
+ - Spout transfer code here: [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L329)
[3/3] storm git commit: Merge branch 'master' of
https://github.com/DigitalPebble/storm
Posted by sr...@apache.org.
Merge branch 'master' of https://github.com/DigitalPebble/storm
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/4a57e500
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/4a57e500
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/4a57e500
Branch: refs/heads/master
Commit: 4a57e500b3e0c3f3256e776357d1e1de1c7f5e49
Parents: f3ed08b 698fff6
Author: Sriharsha Chintalapani <ha...@hortonworks.com>
Authored: Wed Nov 4 12:04:59 2015 -0800
Committer: Sriharsha Chintalapani <ha...@hortonworks.com>
Committed: Wed Nov 4 12:04:59 2015 -0800
----------------------------------------------------------------------
DISCLAIMER | 10 ------
bin/storm.py | 14 ++++----
.../Message-passing-implementation.md | 34 ++++++++++----------
3 files changed, 24 insertions(+), 34 deletions(-)
----------------------------------------------------------------------
[2/3] storm git commit: Message-passing-implementation : fix URLs by
directing them to the branch 0.7.1 instead of master
Posted by sr...@apache.org.
Message-passing-implementation : fix URLs by directing them to the branch 0.7.1 instead of master
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/698fff6f
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/698fff6f
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/698fff6f
Branch: refs/heads/master
Commit: 698fff6f7426c30694adbefac07cfe390030cf05
Parents: acca572
Author: Julien Nioche <ju...@digitalpebble.com>
Authored: Thu Oct 29 14:53:08 2015 +0000
Committer: Julien Nioche <ju...@digitalpebble.com>
Committed: Thu Oct 29 14:53:08 2015 +0000
----------------------------------------------------------------------
.../Message-passing-implementation.md | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/698fff6f/docs/documentation/Message-passing-implementation.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Message-passing-implementation.md b/docs/documentation/Message-passing-implementation.md
index e498224..a17f66a 100644
--- a/docs/documentation/Message-passing-implementation.md
+++ b/docs/documentation/Message-passing-implementation.md
@@ -16,15 +16,15 @@ This page walks through how emitting and transferring tuples works in Storm.
- The implementation for distributed mode uses ZeroMQ [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/messaging/zmq.clj)
- The implementation for local mode uses in memory Java queues (so that it's easy to use Storm locally without needing to get ZeroMQ installed) [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/messaging/local.clj)
- Receiving messages in tasks works differently in local mode and distributed mode
- - In local mode, the tuple is sent directly to an in-memory queue for the receiving task [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/messaging/local.clj#L21)
- - In distributed mode, each worker listens on a single TCP port for incoming messages and then routes those messages in-memory to tasks. The TCP port is called a "virtual port", because it receives [task id, message] and then routes it to the actual task. [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/worker.clj#L204)
- - The virtual port implementation is here: [code](https://github.com/apache/storm/blob/master/src/clj/zilch/virtual_port.clj)
- - Tasks listen on an in-memory ZeroMQ port for messages from the virtual port [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L201)
- - Bolts listen here: [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L489)
- - Spouts listen here: [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L382)
+ - In local mode, the tuple is sent directly to an in-memory queue for the receiving task [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/messaging/local.clj#L21)
+ - In distributed mode, each worker listens on a single TCP port for incoming messages and then routes those messages in-memory to tasks. The TCP port is called a "virtual port", because it receives [task id, message] and then routes it to the actual task. [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L204)
+ - The virtual port implementation is here: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/zilch/virtual_port.clj)
+ - Tasks listen on an in-memory ZeroMQ port for messages from the virtual port [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L201)
+ - Bolts listen here: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L489)
+ - Spouts listen here: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L382)
- Tasks are responsible for message routing. A tuple is emitted either to a direct stream (where the task id is specified) or a regular stream. In direct streams, the message is only sent if that bolt subscribes to that direct stream. In regular streams, the stream grouping functions are used to determine the task ids to send the tuple to.
- - Tasks have a routing map from {stream id} -> {component id} -> {stream grouping function} [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L198)
- - The "tasks-fn" returns the task ids to send the tuples to for either regular stream emit or direct stream emit [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L207)
+ - Tasks have a routing map from {stream id} -> {component id} -> {stream grouping function} [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L198)
+ - The "tasks-fn" returns the task ids to send the tuples to for either regular stream emit or direct stream emit [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L207)
- After getting the output task ids, bolts and spouts use the transfer-fn provided by the worker to actually transfer the tuples
- - Bolt transfer code here: [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L429)
- - Spout transfer code here: [code](https://github.com/apache/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L329)
+ - Bolt transfer code here: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L429)
+ - Spout transfer code here: [code](https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L329)