You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by pt...@apache.org on 2014/11/14 19:28:33 UTC
[1/6] storm git commit: STORM-549: "topology.enable.message.timeouts"
does nothing
Repository: storm
Updated Branches:
refs/heads/master 413a6952c -> c2fefbcc1
STORM-549: "topology.enable.message.timeouts" does nothing
The ability to turn off message timeouts doesn't work.
The value is a true/false value that is represented as
a Java Boolean. The check for the truthiness used the
clojure not function. The idea was, turn off message
timeouts if "topology.enable.message.timeouts" is not true.
Unfortunately this won't worth with clojure's definition of
truthiness as:
(not (Boolean. false))
evaluates to true.
This commit updates the logic to compare the Java Boolean
"topology.enable.message.timeouts" to false using the
= function which will work.
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/0505d474
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/0505d474
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/0505d474
Branch: refs/heads/master
Commit: 0505d47471170861ad303dc530a8d8de4e730c10
Parents: baf7e2c
Author: Sean T Allen <se...@monkeysnatchbanana.com>
Authored: Sat Nov 8 09:41:29 2014 -0500
Committer: Sean T Allen <se...@monkeysnatchbanana.com>
Committed: Sat Nov 8 09:41:29 2014 -0500
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/daemon/executor.clj | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/0505d474/storm-core/src/clj/backtype/storm/daemon/executor.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/daemon/executor.clj b/storm-core/src/clj/backtype/storm/daemon/executor.clj
index f60aa80..91b3128 100644
--- a/storm-core/src/clj/backtype/storm/daemon/executor.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/executor.clj
@@ -301,7 +301,7 @@
context (:worker-context executor-data)]
(when tick-time-secs
(if (or (system-id? (:component-id executor-data))
- (and (not (storm-conf TOPOLOGY-ENABLE-MESSAGE-TIMEOUTS))
+ (and (= false (storm-conf TOPOLOGY-ENABLE-MESSAGE-TIMEOUTS))
(= :spout (:type executor-data))))
(log-message "Timeouts disabled for executor " (:component-id executor-data) ":" (:executor-id executor-data))
(schedule-recurring
[6/6] storm git commit: Merge branch 'master' of
https://git-wip-us.apache.org/repos/asf/storm
Posted by pt...@apache.org.
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/storm
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/c2fefbcc
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/c2fefbcc
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/c2fefbcc
Branch: refs/heads/master
Commit: c2fefbcc1bcc022303ce690a35206cce92421fb0
Parents: b606bff 413a695
Author: P. Taylor Goetz <pt...@gmail.com>
Authored: Fri Nov 14 13:28:06 2014 -0500
Committer: P. Taylor Goetz <pt...@gmail.com>
Committed: Fri Nov 14 13:28:06 2014 -0500
----------------------------------------------------------------------
README.markdown | 1 +
SECURITY.md | 356 ++-
STORM-UI-REST-API.md | 17 +-
bin/storm | 15 +-
conf/defaults.yaml | 45 +-
conf/jaas_digest.conf | 8 +-
conf/jaas_kerberos.conf | 15 +
.../src/jvm/storm/starter/util/StormRunner.java | 3 +-
logback/worker.xml | 41 +
pom.xml | 33 +-
storm-core/pom.xml | 299 +-
.../src/clj/backtype/storm/LocalCluster.clj | 4 +
storm-core/src/clj/backtype/storm/LocalDRPC.clj | 4 +-
.../src/clj/backtype/storm/MockAutoCred.clj | 58 +
storm-core/src/clj/backtype/storm/bootstrap.clj | 5 +-
storm-core/src/clj/backtype/storm/cluster.clj | 167 +-
.../storm/command/upload_credentials.clj | 35 +
storm-core/src/clj/backtype/storm/config.clj | 40 +-
.../src/clj/backtype/storm/daemon/common.clj | 18 +-
.../src/clj/backtype/storm/daemon/drpc.clj | 174 +-
.../src/clj/backtype/storm/daemon/executor.clj | 119 +-
.../src/clj/backtype/storm/daemon/logviewer.clj | 271 +-
.../src/clj/backtype/storm/daemon/nimbus.clj | 245 +-
.../clj/backtype/storm/daemon/supervisor.clj | 173 +-
.../src/clj/backtype/storm/daemon/task.clj | 7 +-
.../src/clj/backtype/storm/daemon/worker.clj | 97 +-
storm-core/src/clj/backtype/storm/testing.clj | 52 +-
storm-core/src/clj/backtype/storm/testing4j.clj | 2 +-
storm-core/src/clj/backtype/storm/thrift.clj | 28 +-
storm-core/src/clj/backtype/storm/ui/core.clj | 169 +-
.../src/clj/backtype/storm/ui/helpers.clj | 69 +-
storm-core/src/clj/backtype/storm/util.clj | 89 +-
storm-core/src/clj/backtype/storm/zookeeper.clj | 19 +-
storm-core/src/clj/storm/trident/testing.clj | 1 +
.../src/dev/drpc-simple-acl-test-scenario.yaml | 11 +
storm-core/src/jvm/backtype/storm/Config.java | 537 +++-
.../jvm/backtype/storm/ConfigValidation.java | 156 +-
.../src/jvm/backtype/storm/Constants.java | 3 +-
.../backtype/storm/ICredentialsListener.java | 32 +
.../src/jvm/backtype/storm/ILocalCluster.java | 2 +
.../src/jvm/backtype/storm/StormSubmitter.java | 164 +-
.../storm/drpc/DRPCInvocationsClient.java | 91 +-
.../src/jvm/backtype/storm/drpc/DRPCSpout.java | 100 +-
.../jvm/backtype/storm/drpc/ReturnResults.java | 35 +-
.../storm/generated/AuthorizationException.java | 345 ++
.../backtype/storm/generated/Credentials.java | 390 +++
.../storm/generated/DistributedRPC.java | 110 +-
.../generated/DistributedRPCInvocations.java | 352 +-
.../jvm/backtype/storm/generated/Nimbus.java | 3006 +++++++++++++++++-
.../backtype/storm/generated/SubmitOptions.java | 98 +-
.../backtype/storm/generated/TopologyInfo.java | 192 +-
.../storm/generated/TopologySummary.java | 192 +-
.../backtype/storm/messaging/netty/Client.java | 31 +-
.../storm/messaging/netty/ControlMessage.java | 4 +-
.../storm/messaging/netty/MessageDecoder.java | 32 +-
.../storm/messaging/netty/MessageEncoder.java | 4 +
.../storm/messaging/netty/SaslMessageToken.java | 99 +
.../storm/messaging/netty/SaslNettyClient.java | 166 +
.../messaging/netty/SaslNettyClientState.java | 31 +
.../storm/messaging/netty/SaslNettyServer.java | 165 +
.../messaging/netty/SaslNettyServerState.java | 31 +
.../messaging/netty/SaslStormClientHandler.java | 158 +
.../netty/SaslStormServerAuthorizeHandler.java | 83 +
.../messaging/netty/SaslStormServerHandler.java | 155 +
.../storm/messaging/netty/SaslUtils.java | 74 +
.../backtype/storm/messaging/netty/Server.java | 27 +-
.../netty/StormClientPipelineFactory.java | 12 +-
.../netty/StormServerPipelineFactory.java | 20 +-
.../jvm/backtype/storm/scheduler/Cluster.java | 13 +
.../scheduler/multitenant/DefaultPool.java | 219 ++
.../storm/scheduler/multitenant/FreePool.java | 125 +
.../scheduler/multitenant/IsolatedPool.java | 346 ++
.../multitenant/MultitenantScheduler.java | 98 +
.../storm/scheduler/multitenant/Node.java | 343 ++
.../storm/scheduler/multitenant/NodePool.java | 296 ++
.../storm/security/INimbusCredentialPlugin.java | 47 +
.../backtype/storm/security/auth/AuthUtils.java | 228 +-
.../auth/DefaultHttpCredentialsPlugin.java | 87 +
.../security/auth/DefaultPrincipalToLocal.java | 43 +
.../storm/security/auth/IAutoCredentials.java | 55 +
.../security/auth/ICredentialsRenewer.java | 41 +
.../auth/IGroupMappingServiceProvider.java | 42 +
.../security/auth/IHttpCredentialsPlugin.java | 50 +
.../storm/security/auth/IPrincipalToLocal.java | 41 +
.../storm/security/auth/ITransportPlugin.java | 14 +-
.../security/auth/KerberosPrincipalToLocal.java | 45 +
.../storm/security/auth/ReqContext.java | 10 +-
.../security/auth/SaslTransportPlugin.java | 44 +-
.../security/auth/ShellBasedGroupsMapping.java | 94 +
.../security/auth/SimpleTransportPlugin.java | 61 +-
.../security/auth/SingleUserPrincipal.java | 56 +
.../storm/security/auth/TBackoffConnect.java | 77 +
.../storm/security/auth/ThriftClient.java | 85 +-
.../security/auth/ThriftConnectionType.java | 77 +
.../storm/security/auth/ThriftServer.java | 19 +-
.../auth/authorizer/DRPCAuthorizerBase.java | 46 +
.../authorizer/DRPCSimpleACLAuthorizer.java | 157 +
.../auth/authorizer/DenyAuthorizer.java | 4 +-
.../auth/authorizer/NoopAuthorizer.java | 6 +-
.../auth/authorizer/SimpleACLAuthorizer.java | 131 +
.../authorizer/SimpleWhitelistAuthorizer.java | 70 +
.../auth/digest/DigestSaslTransportPlugin.java | 1 +
.../storm/security/auth/hadoop/AutoHDFS.java | 262 ++
.../storm/security/auth/kerberos/AutoTGT.java | 281 ++
.../auth/kerberos/AutoTGTKrb5LoginModule.java | 108 +
.../kerberos/AutoTGTKrb5LoginModuleTest.java | 44 +
.../auth/kerberos/ClientCallbackHandler.java | 104 +
.../kerberos/KerberosSaslTransportPlugin.java | 206 ++
.../auth/kerberos/ServerCallbackHandler.java | 86 +
.../auth/kerberos/jaas_kerberos_cluster.conf | 31 +
.../auth/kerberos/jaas_kerberos_launcher.conf | 12 +
.../testing/SingleUserSimpleTransport.java | 37 +
.../state/TestTransactionalState.java | 47 +
.../transactional/state/TransactionalState.java | 56 +-
.../jvm/backtype/storm/utils/DRPCClient.java | 63 +-
.../jvm/backtype/storm/utils/LocalState.java | 14 +-
.../jvm/backtype/storm/utils/NimbusClient.java | 11 +-
.../jvm/backtype/storm/utils/ShellUtils.java | 498 +++
.../src/jvm/backtype/storm/utils/TestUtils.java | 34 +
.../src/jvm/backtype/storm/utils/Utils.java | 130 +-
.../backtype/storm/utils/ZookeeperAuthInfo.java | 9 +-
.../storm/utils/ZookeeperServerCnxnFactory.java | 84 +
.../trident/drpc/ReturnResultsReducer.java | 13 +-
.../topology/state/TestTransactionalState.java | 47 +
.../topology/state/TransactionalState.java | 58 +-
.../src/native/worker-launcher/.autom4te.cfg | 42 +
.../worker-launcher/.deps/worker-launcher.Po | 1 +
.../src/native/worker-launcher/Makefile.am | 32 +
.../src/native/worker-launcher/configure.ac | 50 +
.../native/worker-launcher/impl/configuration.c | 340 ++
.../native/worker-launcher/impl/configuration.h | 45 +
.../src/native/worker-launcher/impl/main.c | 210 ++
.../worker-launcher/impl/worker-launcher.c | 779 +++++
.../worker-launcher/impl/worker-launcher.h | 129 +
.../worker-launcher/test/test-worker-launcher.c | 340 ++
storm-core/src/py/__init__.py | 16 +
storm-core/src/py/storm/DistributedRPC-remote | 18 +
storm-core/src/py/storm/DistributedRPC.py | 37 +-
.../py/storm/DistributedRPCInvocations-remote | 18 +
.../src/py/storm/DistributedRPCInvocations.py | 96 +-
storm-core/src/py/storm/Nimbus-remote | 25 +
storm-core/src/py/storm/Nimbus.py | 652 +++-
storm-core/src/py/storm/__init__.py | 16 +
storm-core/src/py/storm/constants.py | 16 +
storm-core/src/py/storm/ttypes.py | 1243 +++++++-
storm-core/src/storm.thrift | 58 +-
storm-core/src/ui/public/component.html | 21 +-
storm-core/src/ui/public/css/style.css | 9 +
storm-core/src/ui/public/index.html | 12 +-
storm-core/src/ui/public/js/script.js | 3 +-
.../public/templates/anti-forgery-template.html | 19 +
.../public/templates/index-page-template.html | 12 +
.../templates/topology-page-template.html | 23 +-
.../src/ui/public/templates/user-template.html | 25 +
storm-core/src/ui/public/topology.html | 21 +-
.../test/clj/backtype/storm/cluster_test.clj | 93 +-
.../test/clj/backtype/storm/config_test.clj | 11 +
.../test/clj/backtype/storm/drpc_test.clj | 14 +-
.../test/clj/backtype/storm/logviewer_test.clj | 187 ++
.../storm/messaging/netty_integration_test.clj | 4 +-
.../storm/messaging/netty_unit_test.clj | 4 +
.../test/clj/backtype/storm/nimbus_test.clj | 377 ++-
.../scheduler/multitenant_scheduler_test.clj | 831 +++++
.../storm/security/auth/AuthUtils_test.clj | 16 +-
.../auth/DefaultHttpCredentialsPlugin_test.clj | 40 +
.../storm/security/auth/ThriftClient_test.clj | 28 +-
.../storm/security/auth/ThriftServer_test.clj | 8 +-
.../backtype/storm/security/auth/auth_test.clj | 374 ++-
.../authorizer/DRPCSimpleACLAuthorizer_test.clj | 226 ++
.../security/auth/auto_login_module_test.clj | 91 +
.../storm/security/auth/drpc-auth-alice.jaas | 5 +
.../storm/security/auth/drpc-auth-bob.jaas | 5 +
.../storm/security/auth/drpc-auth-charlie.jaas | 5 +
.../storm/security/auth/drpc-auth-server.jaas | 6 +
.../storm/security/auth/drpc_auth_test.clj | 315 ++
.../storm/security/auth/nimbus_auth_test.clj | 181 ++
.../test/clj/backtype/storm/submitter_test.clj | 75 +
.../test/clj/backtype/storm/supervisor_test.clj | 176 +-
.../clj/backtype/storm/transactional_test.clj | 27 +-
.../utils/ZookeeperServerCnxnFactory_test.clj | 35 +
.../test/clj/backtype/storm/utils_test.clj | 58 +-
.../test/clj/storm/trident/state_test.clj | 25 +-
storm-dist/binary/src/main/assembly/binary.xml | 4 +
183 files changed, 21137 insertions(+), 1285 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/c2fefbcc/storm-core/src/clj/backtype/storm/daemon/executor.clj
----------------------------------------------------------------------
[3/6] storm git commit: Merge pull request #1 from
HeartSaVioR/SeanTAllen-master
Posted by pt...@apache.org.
Merge pull request #1 from HeartSaVioR/SeanTAllen-master
Add unit test for disable tuple timeout (adding to apache/storm#309)
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/cc633a94
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/cc633a94
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/cc633a94
Branch: refs/heads/master
Commit: cc633a9420962c3b93a396e4dabb978e4e373492
Parents: 0505d47 5c9d06b
Author: Sean T Allen <se...@monkeysnatchbanana.com>
Authored: Sat Nov 8 23:23:09 2014 -0500
Committer: Sean T Allen <se...@monkeysnatchbanana.com>
Committed: Sat Nov 8 23:23:09 2014 -0500
----------------------------------------------------------------------
.../test/clj/backtype/storm/testing4j_test.clj | 32 ++++++++++++++++++++
1 file changed, 32 insertions(+)
----------------------------------------------------------------------
[2/6] storm git commit: Add test for disable tuple timeout
Posted by pt...@apache.org.
Add test for disable tuple timeout
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/5c9d06bc
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/5c9d06bc
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/5c9d06bc
Branch: refs/heads/master
Commit: 5c9d06bcb280c5059497929d7910a9a8d07349ac
Parents: 0505d47
Author: Jungtaek Lim <ka...@gmail.com>
Authored: Sun Nov 9 12:12:13 2014 +0900
Committer: Jungtaek Lim <ka...@gmail.com>
Committed: Sun Nov 9 12:12:13 2014 +0900
----------------------------------------------------------------------
.../test/clj/backtype/storm/testing4j_test.clj | 32 ++++++++++++++++++++
1 file changed, 32 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/5c9d06bc/storm-core/test/clj/backtype/storm/testing4j_test.clj
----------------------------------------------------------------------
diff --git a/storm-core/test/clj/backtype/storm/testing4j_test.clj b/storm-core/test/clj/backtype/storm/testing4j_test.clj
index bb920af..36f3df6 100644
--- a/storm-core/test/clj/backtype/storm/testing4j_test.clj
+++ b/storm-core/test/clj/backtype/storm/testing4j_test.clj
@@ -157,6 +157,38 @@
(it/assert-failed tracker 2)
))))))
+(deftest test-disable-tuple-timeout
+ (let [daemon-conf (doto (Config.)
+ (.put TOPOLOGY-ENABLE-MESSAGE-TIMEOUTS false))
+ mk-cluster-param (doto (MkClusterParam.)
+ (.setDaemonConf daemon-conf))]
+ (Testing/withSimulatedTimeLocalCluster
+ mk-cluster-param
+ (reify TestJob
+ (^void run [this ^ILocalCluster cluster]
+ (let [feeder (feeder-spout ["field1"])
+ tracker (AckFailMapTracker.)
+ _ (.setAckFailDelegate feeder tracker)
+ topology (thrift/mk-topology
+ {"1" (thrift/mk-spout-spec feeder)}
+ {"2" (thrift/mk-bolt-spec {"1" :global} it/ack-every-other)})
+ storm-conf (doto (Config.)
+ (.put TOPOLOGY-MESSAGE-TIMEOUT-SECS 10)
+ (.put TOPOLOGY-ENABLE-MESSAGE-TIMEOUTS false))]
+ (.submitTopology cluster
+ "disable-timeout-tester"
+ storm-conf
+ topology)
+ (.feed feeder ["a"] 1)
+ (.feed feeder ["b"] 2)
+ (.feed feeder ["c"] 3)
+ (Testing/advanceClusterTime cluster (int 9))
+ (it/assert-acked tracker 1 3)
+ (is (not (.isFailed tracker 2)))
+ (Testing/advanceClusterTime cluster (int 12))
+ (is (not (.isFailed tracker 2)))
+ ))))))
+
(deftest test-test-tuple
(letlocals
;; test the one-param signature
[4/6] storm git commit: Merge branch 'master' of
github.com:SeanTAllen/storm
Posted by pt...@apache.org.
Merge branch 'master' of github.com:SeanTAllen/storm
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/03542ad8
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/03542ad8
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/03542ad8
Branch: refs/heads/master
Commit: 03542ad875cc195057c2aec83184947d91e2b1d8
Parents: 64f06c7 cc633a9
Author: P. Taylor Goetz <pt...@gmail.com>
Authored: Fri Nov 14 13:26:52 2014 -0500
Committer: P. Taylor Goetz <pt...@gmail.com>
Committed: Fri Nov 14 13:26:52 2014 -0500
----------------------------------------------------------------------
.../src/clj/backtype/storm/daemon/executor.clj | 2 +-
.../test/clj/backtype/storm/testing4j_test.clj | 32 ++++++++++++++++++++
2 files changed, 33 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/03542ad8/storm-core/src/clj/backtype/storm/daemon/executor.clj
----------------------------------------------------------------------
[5/6] storm git commit: STORM-549: update changelog
Posted by pt...@apache.org.
STORM-549: update changelog
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/b606bff8
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/b606bff8
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/b606bff8
Branch: refs/heads/master
Commit: b606bff8a3ed278db4376f13a0ad23d2dbc7c724
Parents: 03542ad
Author: P. Taylor Goetz <pt...@gmail.com>
Authored: Fri Nov 14 13:24:59 2014 -0500
Committer: P. Taylor Goetz <pt...@gmail.com>
Committed: Fri Nov 14 13:27:40 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/b606bff8/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bcdef34..b05f9ab 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
## 0.9.3-rc2
+ * STORM-549: "topology.enable.message.timeouts" does nothing
* STORM-546: Local hostname configuration ignored by executor
* STORM-492: Test timeout should be configurable
* STORM-540: Change default time format in logs to ISO8601 in order to include timezone