You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by sr...@apache.org on 2019/03/15 21:01:32 UTC
[storm] branch master updated (ca3a17a -> 68d2f49)
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git.
from ca3a17a Merge branch 'minor-update-maven-repository-to-https'
new 7191246 STORM-3353: Upgrade to Curator 4.2.0
new 18e24a3 Merge branch 'STORM-3353' of https://github.com/srdo/storm into asfgit-master
new b25c242 STORM-3312: Upgrade Guava to latest version where possible
new 65ce33c Merge branch 'STORM-3312' of https://github.com/srdo/storm into asfgit-master
new 60199bd STORM-3355: Use supervisor.worker.shutdown.sleep.secs to set worker suicide delay to allow users to configure how long they're willing to wait for orderly shutdown
new 68d2f49 Merge branch 'STORM-3355' of https://github.com/srdo/storm into asfgit-master
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
conf/defaults.yaml | 2 +-
examples/storm-hdfs-examples/pom.xml | 10 ++++++++++
examples/storm-hive-examples/pom.xml | 10 ++++++++++
examples/storm-mqtt-examples/pom.xml | 10 ++++++++++
examples/storm-pmml-examples/pom.xml | 10 ++++++++++
examples/storm-solr-examples/pom.xml | 10 ++++++++++
external/storm-autocreds/pom.xml | 1 -
external/storm-blobstore-migration/pom.xml | 11 +++++++++++
external/storm-elasticsearch/pom.xml | 1 -
external/storm-hdfs-blobstore/pom.xml | 11 ++++++++++-
external/storm-hdfs/pom.xml | 10 ++++++++++
external/storm-hive/pom.xml | 12 +++++++++++-
external/storm-jdbc/pom.xml | 1 -
external/storm-mqtt/pom.xml | 10 ++++++++++
external/storm-opentsdb/pom.xml | 1 -
external/storm-pmml/pom.xml | 10 ++++++++++
external/storm-redis/pom.xml | 1 -
external/storm-solr/pom.xml | 4 +++-
pom.xml | 6 +++---
shaded-deps/pom.xml | 10 +++-------
sql/storm-sql-core/pom.xml | 10 ++++++++++
sql/storm-sql-external/storm-sql-hdfs/pom.xml | 10 ++++++++++
sql/storm-sql-external/storm-sql-kafka/pom.xml | 10 ++++++++++
sql/storm-sql-runtime/pom.xml | 10 ++++++++++
storm-client/src/jvm/org/apache/storm/Config.java | 7 +++++++
.../src/jvm/org/apache/storm/daemon/worker/Worker.java | 4 +++-
storm-client/src/jvm/org/apache/storm/task/IBolt.java | 6 ++++--
.../src/main/java/org/apache/storm/DaemonConfig.java | 7 -------
.../main/java/org/apache/storm/daemon/supervisor/Slot.java | 2 +-
.../java/org/apache/storm/daemon/supervisor/Supervisor.java | 2 +-
30 files changed, 178 insertions(+), 31 deletions(-)
[storm] 01/06: STORM-3312: Upgrade Guava to latest version where
possible
Posted by sr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
commit b25c242813da80dd1c5ee1ef6aa33499a0c1c10b
Author: Stig Rohde Døssing <sr...@apache.org>
AuthorDate: Wed Jan 9 22:07:37 2019 +0100
STORM-3312: Upgrade Guava to latest version where possible
---
examples/storm-hdfs-examples/pom.xml | 10 ++++++++++
examples/storm-hive-examples/pom.xml | 10 ++++++++++
examples/storm-mqtt-examples/pom.xml | 10 ++++++++++
examples/storm-pmml-examples/pom.xml | 10 ++++++++++
examples/storm-solr-examples/pom.xml | 10 ++++++++++
external/storm-autocreds/pom.xml | 1 -
external/storm-blobstore-migration/pom.xml | 11 +++++++++++
external/storm-elasticsearch/pom.xml | 1 -
external/storm-hdfs-blobstore/pom.xml | 11 ++++++++++-
external/storm-hdfs/pom.xml | 10 ++++++++++
external/storm-hive/pom.xml | 12 +++++++++++-
external/storm-jdbc/pom.xml | 1 -
external/storm-mqtt/pom.xml | 10 ++++++++++
external/storm-opentsdb/pom.xml | 1 -
external/storm-pmml/pom.xml | 10 ++++++++++
external/storm-redis/pom.xml | 1 -
external/storm-solr/pom.xml | 4 +++-
pom.xml | 4 ++--
shaded-deps/pom.xml | 10 +++-------
sql/storm-sql-core/pom.xml | 10 ++++++++++
sql/storm-sql-external/storm-sql-hdfs/pom.xml | 10 ++++++++++
sql/storm-sql-external/storm-sql-kafka/pom.xml | 10 ++++++++++
sql/storm-sql-runtime/pom.xml | 10 ++++++++++
23 files changed, 160 insertions(+), 17 deletions(-)
diff --git a/examples/storm-hdfs-examples/pom.xml b/examples/storm-hdfs-examples/pom.xml
index b116a3f..6a2161b 100644
--- a/examples/storm-hdfs-examples/pom.xml
+++ b/examples/storm-hdfs-examples/pom.xml
@@ -26,6 +26,11 @@
</parent>
<artifactId>storm-hdfs-examples</artifactId>
+
+ <properties>
+ <!-- Required downgrade by hadoop-hdfs 2.6.1 -->
+ <guava.version>16.0.1</guava.version>
+ </properties>
<dependencies>
<dependency>
@@ -39,6 +44,11 @@
<artifactId>storm-hdfs</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
diff --git a/examples/storm-hive-examples/pom.xml b/examples/storm-hive-examples/pom.xml
index 696b56c..1a2b81a 100644
--- a/examples/storm-hive-examples/pom.xml
+++ b/examples/storm-hive-examples/pom.xml
@@ -26,6 +26,11 @@
</parent>
<artifactId>storm-hive-examples</artifactId>
+
+ <properties>
+ <!-- Required downgrade by hive-hcatalog-core 2.3.4 -->
+ <guava.version>16.0.1</guava.version>
+ </properties>
<dependencies>
<dependency>
@@ -39,6 +44,11 @@
<artifactId>storm-hive</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
diff --git a/examples/storm-mqtt-examples/pom.xml b/examples/storm-mqtt-examples/pom.xml
index 3bdb88b..27237dc 100644
--- a/examples/storm-mqtt-examples/pom.xml
+++ b/examples/storm-mqtt-examples/pom.xml
@@ -29,6 +29,11 @@
<version>2.0.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
+
+ <properties>
+ <!-- Required downgrade by activemq-broker 5.15.3 -->
+ <guava.version>16.0.1</guava.version>
+ </properties>
<dependencies>
<dependency>
@@ -64,6 +69,11 @@
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-kahadb-store</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
<plugins>
diff --git a/examples/storm-pmml-examples/pom.xml b/examples/storm-pmml-examples/pom.xml
index e46514c..9cdc7ef 100644
--- a/examples/storm-pmml-examples/pom.xml
+++ b/examples/storm-pmml-examples/pom.xml
@@ -29,6 +29,11 @@
<artifactId>storm-pmml-examples</artifactId>
<name>storm-pmml-examples</name>
+ <properties>
+ <!-- Required downgrade by pmml-evaluator 1.0.22 -->
+ <guava.version>16.0.1</guava.version>
+ </properties>
+
<dependencies>
<dependency>
<groupId>org.apache.storm</groupId>
@@ -45,6 +50,11 @@
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
diff --git a/examples/storm-solr-examples/pom.xml b/examples/storm-solr-examples/pom.xml
index cbb40c6..d2ae1dc 100644
--- a/examples/storm-solr-examples/pom.xml
+++ b/examples/storm-solr-examples/pom.xml
@@ -26,6 +26,11 @@
</parent>
<artifactId>storm-solr-examples</artifactId>
+
+ <properties>
+ <!-- Required downgrade by solr-core 5.5.5 -->
+ <guava.version>17.0</guava.version>
+ </properties>
<dependencies>
<dependency>
@@ -60,6 +65,11 @@
<artifactId>commons-logging</artifactId>
<version>1.1.3</version>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
diff --git a/external/storm-autocreds/pom.xml b/external/storm-autocreds/pom.xml
index d819145..a3e580e 100644
--- a/external/storm-autocreds/pom.xml
+++ b/external/storm-autocreds/pom.xml
@@ -26,7 +26,6 @@
<artifactId>storm-autocreds</artifactId>
-
<dependencies>
<dependency>
<groupId>org.apache.storm</groupId>
diff --git a/external/storm-blobstore-migration/pom.xml b/external/storm-blobstore-migration/pom.xml
index c993f20..3c39a92 100644
--- a/external/storm-blobstore-migration/pom.xml
+++ b/external/storm-blobstore-migration/pom.xml
@@ -30,6 +30,12 @@ limitations under the License.
<name>blobstore-migrator</name>
<url>https://maven.apache.org</url>
+
+ <properties>
+ <!-- Required downgrade by hadoop-hdfs 2.8.5 -->
+ <guava.version>16.0.1</guava.version>
+ </properties>
+
<dependencies>
<dependency>
<groupId>org.apache.storm</groupId>
@@ -72,6 +78,11 @@ limitations under the License.
<groupId>org.apache.hadoop</groupId>
<version>${hadoop.version}</version>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
<plugins>
diff --git a/external/storm-elasticsearch/pom.xml b/external/storm-elasticsearch/pom.xml
index bbc9bf0..eae9177 100644
--- a/external/storm-elasticsearch/pom.xml
+++ b/external/storm-elasticsearch/pom.xml
@@ -113,7 +113,6 @@
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
- <version>18.0</version>
<scope>test</scope>
</dependency>
</dependencies>
diff --git a/external/storm-hdfs-blobstore/pom.xml b/external/storm-hdfs-blobstore/pom.xml
index 2d6504d..f234087 100644
--- a/external/storm-hdfs-blobstore/pom.xml
+++ b/external/storm-hdfs-blobstore/pom.xml
@@ -26,6 +26,11 @@
</parent>
<artifactId>storm-hdfs-blobstore</artifactId>
+
+ <properties>
+ <!-- Required downgrade by hadoop-hdfs 2.8.5 -->
+ <guava.version>16.0.1</guava.version>
+ </properties>
<developers>
<developer>
@@ -198,7 +203,11 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
-
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
<plugins>
diff --git a/external/storm-hdfs/pom.xml b/external/storm-hdfs/pom.xml
index b09f6d4..7b52ba1 100644
--- a/external/storm-hdfs/pom.xml
+++ b/external/storm-hdfs/pom.xml
@@ -26,6 +26,11 @@
</parent>
<artifactId>storm-hdfs</artifactId>
+
+ <properties>
+ <!-- Required downgrade by hadoop-hdfs 2.8.5 -->
+ <guava.version>16.0.1</guava.version>
+ </properties>
<developers>
<developer>
@@ -218,6 +223,11 @@
<artifactId>storm-autocreds</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
<plugins>
diff --git a/external/storm-hive/pom.xml b/external/storm-hive/pom.xml
index 2fd8ae5..95ea4c7 100644
--- a/external/storm-hive/pom.xml
+++ b/external/storm-hive/pom.xml
@@ -36,6 +36,11 @@
</developer>
</developers>
+ <properties>
+ <!-- Required downgrade by hive-hcatalog-core 2.3.4 -->
+ <guava.version>16.0.1</guava.version>
+ </properties>
+
<dependencies>
<dependency>
<groupId>org.apache.storm</groupId>
@@ -164,7 +169,12 @@
<groupId>org.apache.storm</groupId>
<artifactId>storm-autocreds</artifactId>
<version>${project.version}</version>
- </dependency>
+ </dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
<plugins>
diff --git a/external/storm-jdbc/pom.xml b/external/storm-jdbc/pom.xml
index 19c17d6..e950c05 100644
--- a/external/storm-jdbc/pom.xml
+++ b/external/storm-jdbc/pom.xml
@@ -54,7 +54,6 @@
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
- <version>17.0</version>
</dependency>
<dependency>
<groupId>com.zaxxer</groupId>
diff --git a/external/storm-mqtt/pom.xml b/external/storm-mqtt/pom.xml
index 3ee2d88..1ff7832 100644
--- a/external/storm-mqtt/pom.xml
+++ b/external/storm-mqtt/pom.xml
@@ -28,6 +28,11 @@
<version>2.0.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
+
+ <properties>
+ <!-- Required downgrade by activemq-broker 5.15.3 -->
+ <guava.version>16.0.1</guava.version>
+ </properties>
<repositories>
<repository>
@@ -87,6 +92,11 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
<plugins>
diff --git a/external/storm-opentsdb/pom.xml b/external/storm-opentsdb/pom.xml
index cbf4708..7ec697d 100644
--- a/external/storm-opentsdb/pom.xml
+++ b/external/storm-opentsdb/pom.xml
@@ -49,7 +49,6 @@
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
- <version>${guava.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
diff --git a/external/storm-pmml/pom.xml b/external/storm-pmml/pom.xml
index dd57158..4d1e091 100644
--- a/external/storm-pmml/pom.xml
+++ b/external/storm-pmml/pom.xml
@@ -30,6 +30,11 @@
<name>storm-pmml</name>
<packaging>jar</packaging>
+
+ <properties>
+ <!-- Required downgrade by pmml-evaluator 1.0.22 -->
+ <guava.version>16.0.1</guava.version>
+ </properties>
<developers>
<developer>
@@ -74,6 +79,11 @@
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
diff --git a/external/storm-redis/pom.xml b/external/storm-redis/pom.xml
index 5617237..c8f27a7 100644
--- a/external/storm-redis/pom.xml
+++ b/external/storm-redis/pom.xml
@@ -59,7 +59,6 @@
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
- <version>18.0</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
diff --git a/external/storm-solr/pom.xml b/external/storm-solr/pom.xml
index 6fca1c6..01a79bd 100644
--- a/external/storm-solr/pom.xml
+++ b/external/storm-solr/pom.xml
@@ -37,6 +37,8 @@
<properties>
<solr.version>5.5.5</solr.version>
+ <!-- Required downgrade by solr-core 5.5.5 -->
+ <guava.version>17.0</guava.version>
</properties>
<dependencies>
@@ -95,7 +97,7 @@
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
- <version>17.0</version>
+ <version>${guava.version}</version>
</dependency>
<!--test dependencies -->
<dependency>
diff --git a/pom.xml b/pom.xml
index fefd3e7..8f312a6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -276,8 +276,8 @@
<httpclient.version>4.5.6</httpclient.version>
<jctools.version>2.0.1</jctools.version>
<jgrapht.version>0.9.0</jgrapht.version>
- <guava.version>16.0.1</guava.version>
- <auto-service.version>1.0-rc3</auto-service.version>
+ <guava.version>27.0.1-jre</guava.version>
+ <auto-service.version>1.0-rc4</auto-service.version>
<netty.version>4.1.30.Final</netty.version>
<sysout-over-slf4j.version>1.0.2</sysout-over-slf4j.version>
<log4j-over-slf4j.version>1.6.6</log4j-over-slf4j.version>
diff --git a/shaded-deps/pom.xml b/shaded-deps/pom.xml
index 6208c33..1342d12 100644
--- a/shaded-deps/pom.xml
+++ b/shaded-deps/pom.xml
@@ -177,7 +177,7 @@
<minimizeJar>false</minimizeJar>
<artifactSet>
<includes>
- <include>com.google.guava:guava</include>
+ <include>com.google.guava:*</include>
<include>com.googlecode.json-simple:json-simple</include>
<include>commons-codec:commons-codec</include>
<include>commons-collections:commons-collections</include>
@@ -203,12 +203,8 @@
<shadedPattern>org.apache.storm.thrift</shadedPattern>
</relocation>
<relocation>
- <pattern>com.google.common</pattern>
- <shadedPattern>org.apache.storm.shade.com.google.common</shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.google.thirdparty</pattern>
- <shadedPattern>org.apache.storm.shade.com.google.thirdparty</shadedPattern>
+ <pattern>com.google</pattern>
+ <shadedPattern>org.apache.storm.shade.com.google</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.curator</pattern>
diff --git a/sql/storm-sql-core/pom.xml b/sql/storm-sql-core/pom.xml
index 006fc45..0b959e2 100644
--- a/sql/storm-sql-core/pom.xml
+++ b/sql/storm-sql-core/pom.xml
@@ -34,6 +34,11 @@
<email>ricetons@gmail.com</email>
</developer>
</developers>
+
+ <properties>
+ <!-- Required downgrade by calcite-core 1.14.0 -->
+ <guava.version>16.0.1</guava.version>
+ </properties>
<dependencies>
<dependency>
@@ -123,6 +128,11 @@
<version>4.1</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
<sourceDirectory>src/jvm</sourceDirectory>
diff --git a/sql/storm-sql-external/storm-sql-hdfs/pom.xml b/sql/storm-sql-external/storm-sql-hdfs/pom.xml
index 1c20d4c..9233e43 100644
--- a/sql/storm-sql-external/storm-sql-hdfs/pom.xml
+++ b/sql/storm-sql-external/storm-sql-hdfs/pom.xml
@@ -26,6 +26,11 @@
</parent>
<artifactId>storm-sql-hdfs</artifactId>
+
+ <properties>
+ <!-- Required downgrade by calcite-core 1.14.0 -->
+ <guava.version>16.0.1</guava.version>
+ </properties>
<developers>
<developer>
@@ -89,6 +94,11 @@
<groupId>org.hamcrest</groupId>
<artifactId>java-hamcrest</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
<sourceDirectory>src/jvm</sourceDirectory>
diff --git a/sql/storm-sql-external/storm-sql-kafka/pom.xml b/sql/storm-sql-external/storm-sql-kafka/pom.xml
index 6c47bae..29d7b07 100644
--- a/sql/storm-sql-external/storm-sql-kafka/pom.xml
+++ b/sql/storm-sql-external/storm-sql-kafka/pom.xml
@@ -26,6 +26,11 @@
</parent>
<artifactId>storm-sql-kafka</artifactId>
+
+ <properties>
+ <!-- Required downgrade by calcite-core 1.15.0 -->
+ <guava.version>16.0.1</guava.version>
+ </properties>
<developers>
<developer>
@@ -73,6 +78,11 @@
<groupId>org.hamcrest</groupId>
<artifactId>java-hamcrest</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
<sourceDirectory>src/jvm</sourceDirectory>
diff --git a/sql/storm-sql-runtime/pom.xml b/sql/storm-sql-runtime/pom.xml
index b579fec..633fd84 100644
--- a/sql/storm-sql-runtime/pom.xml
+++ b/sql/storm-sql-runtime/pom.xml
@@ -26,6 +26,11 @@
</parent>
<artifactId>storm-sql-runtime</artifactId>
+
+ <properties>
+ <!-- Required downgrade by calcite-core 1.15.0 -->
+ <guava.version>16.0.1</guava.version>
+ </properties>
<developers>
<developer>
@@ -115,6 +120,11 @@
<groupId>org.hamcrest</groupId>
<artifactId>java-hamcrest</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
</dependencies>
<build>
<sourceDirectory>src/jvm</sourceDirectory>
[storm] 06/06: Merge branch 'STORM-3355' of
https://github.com/srdo/storm into asfgit-master
Posted by sr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
commit 68d2f49e2bfd984b9d015bc7f40ae600a0031ca1
Merge: 65ce33c 60199bd
Author: Stig Rohde Døssing <sr...@apache.org>
AuthorDate: Fri Mar 15 22:00:58 2019 +0100
Merge branch 'STORM-3355' of https://github.com/srdo/storm into asfgit-master
conf/defaults.yaml | 2 +-
storm-client/src/jvm/org/apache/storm/Config.java | 7 +++++++
storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java | 4 +++-
storm-client/src/jvm/org/apache/storm/task/IBolt.java | 6 ++++--
storm-server/src/main/java/org/apache/storm/DaemonConfig.java | 7 -------
.../src/main/java/org/apache/storm/daemon/supervisor/Slot.java | 2 +-
.../main/java/org/apache/storm/daemon/supervisor/Supervisor.java | 2 +-
7 files changed, 17 insertions(+), 13 deletions(-)
[storm] 02/06: STORM-3353: Upgrade to Curator 4.2.0
Posted by sr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
commit 719124682e6a736fac5c4e089bed55f9d45e7b20
Author: Stig Rohde Døssing <sr...@apache.org>
AuthorDate: Mon Mar 11 17:57:17 2019 +0100
STORM-3353: Upgrade to Curator 4.2.0
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index fefd3e7..31715cf 100644
--- a/pom.xml
+++ b/pom.xml
@@ -266,7 +266,7 @@
<commons-fileupload.version>1.3.3</commons-fileupload.version>
<commons-codec.version>1.11</commons-codec.version>
<commons-cli.version>1.4</commons-cli.version>
- <curator.version>4.0.1</curator.version>
+ <curator.version>4.2.0</curator.version>
<curator-test.version>2.13.0</curator-test.version>
<json-simple.version>1.1</json-simple.version>
<jetty.version>9.4.14.v20181114</jetty.version>
[storm] 03/06: STORM-3355: Use
supervisor.worker.shutdown.sleep.secs to set worker suicide delay to allow
users to configure how long they're willing to wait for orderly shutdown
Posted by sr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
commit 60199bd7600a7d8b32cea59d5691c279e4032c0f
Author: Stig Rohde Døssing <sr...@apache.org>
AuthorDate: Tue Mar 12 08:50:08 2019 +0100
STORM-3355: Use supervisor.worker.shutdown.sleep.secs to set worker suicide delay to allow users to configure how long they're willing to wait for orderly shutdown
---
conf/defaults.yaml | 2 +-
storm-client/src/jvm/org/apache/storm/Config.java | 7 +++++++
storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java | 4 +++-
storm-client/src/jvm/org/apache/storm/task/IBolt.java | 6 ++++--
storm-server/src/main/java/org/apache/storm/DaemonConfig.java | 7 -------
.../src/main/java/org/apache/storm/daemon/supervisor/Slot.java | 2 +-
.../main/java/org/apache/storm/daemon/supervisor/Supervisor.java | 2 +-
7 files changed, 17 insertions(+), 13 deletions(-)
diff --git a/conf/defaults.yaml b/conf/defaults.yaml
index ec81d22..548f14c 100644
--- a/conf/defaults.yaml
+++ b/conf/defaults.yaml
@@ -161,7 +161,7 @@ supervisor.run.worker.as.user: false
supervisor.worker.start.timeout.secs: 120
#how long between heartbeats until supervisor considers that worker dead and tries to restart it
supervisor.worker.timeout.secs: 30
-#how many seconds to sleep for before shutting down threads on worker
+#How many seconds to allow for graceful worker shutdown when killing workers before resorting to force kill
supervisor.worker.shutdown.sleep.secs: 3
#how frequently the supervisor checks on the status of the processes it's monitoring and restarts if necessary
supervisor.monitor.frequency.secs: 3
diff --git a/storm-client/src/jvm/org/apache/storm/Config.java b/storm-client/src/jvm/org/apache/storm/Config.java
index a1aea4c..7c8d19e 100644
--- a/storm-client/src/jvm/org/apache/storm/Config.java
+++ b/storm-client/src/jvm/org/apache/storm/Config.java
@@ -1042,6 +1042,13 @@ public class Config extends HashMap<String, Object> {
@NotNull
public static final String SUPERVISOR_WORKER_TIMEOUT_SECS = "supervisor.worker.timeout.secs";
/**
+ * How many seconds to allow for graceful worker shutdown when killing workers before resorting to force kill.
+ * If a worker fails to shut down gracefully within this delay, it will either suicide or be forcibly killed by the supervisor.
+ */
+ @isInteger
+ @isPositiveNumber
+ public static final String SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS = "supervisor.worker.shutdown.sleep.secs";
+ /**
* A list of hosts of ZooKeeper servers used to manage the cluster.
*/
@isStringList
diff --git a/storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java b/storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java
index 048425e..0e7b6cb 100644
--- a/storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java
+++ b/storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java
@@ -127,7 +127,9 @@ public class Worker implements Shutdownable, DaemonCommon {
Worker worker = new Worker(conf, null, stormId, assignmentId, Integer.parseInt(supervisorPort),
Integer.parseInt(portStr), workerId);
worker.start();
- Utils.addShutdownHookWithForceKillIn1Sec(worker::shutdown);
+ int workerShutdownSleepSecs = ObjectReader.getInt(conf.get(Config.SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS));
+ LOG.info("Adding shutdown hook with kill in {} secs", workerShutdownSleepSecs);
+ Utils.addShutdownHookWithDelayedForceKill(worker::shutdown, workerShutdownSleepSecs);
}
public void start() throws Exception {
diff --git a/storm-client/src/jvm/org/apache/storm/task/IBolt.java b/storm-client/src/jvm/org/apache/storm/task/IBolt.java
index 21f1b8d..6e0d033 100644
--- a/storm-client/src/jvm/org/apache/storm/task/IBolt.java
+++ b/storm-client/src/jvm/org/apache/storm/task/IBolt.java
@@ -14,6 +14,7 @@ package org.apache.storm.task;
import java.io.Serializable;
import java.util.Map;
+import org.apache.storm.Config;
import org.apache.storm.tuple.Tuple;
/**
@@ -63,8 +64,9 @@ public interface IBolt extends Serializable {
void execute(Tuple input);
/**
- * Called when an IBolt is going to be shutdown. There is no guarantee that cleanup will be called, because the supervisor kill -9's
- * worker processes on the cluster.
+ * Called when an IBolt is going to be shutdown. Storm will make a best-effort attempt to call this if the worker shutdown is orderly.
+ * The {@link Config#SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS} setting controls how long orderly shutdown is allowed to take.
+ * There is no guarantee that cleanup will be called if shutdown is not orderly, or if the shutdown exceeds the time limit.
*
* The one context where cleanup is guaranteed to be called is when a topology is killed when running Storm in local mode.
*/
diff --git a/storm-server/src/main/java/org/apache/storm/DaemonConfig.java b/storm-server/src/main/java/org/apache/storm/DaemonConfig.java
index ae013de..8015c1d 100644
--- a/storm-server/src/main/java/org/apache/storm/DaemonConfig.java
+++ b/storm-server/src/main/java/org/apache/storm/DaemonConfig.java
@@ -773,13 +773,6 @@ public class DaemonConfig implements Validated {
public static final String SUPERVISOR_CHILDOPTS = "supervisor.childopts";
/**
- * How many seconds to sleep for before shutting down threads on worker.
- */
- @isInteger
- @isPositiveNumber
- public static final String SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS = "supervisor.worker.shutdown.sleep.secs";
-
- /**
* How long a worker can go without heartbeating during the initial launch before the supervisor tries to restart the worker process.
* This value override supervisor.worker.timeout.secs during launch because there is additional overhead to starting and configuring the
* JVM on launch.
diff --git a/storm-server/src/main/java/org/apache/storm/daemon/supervisor/Slot.java b/storm-server/src/main/java/org/apache/storm/daemon/supervisor/Slot.java
index 74c3e3d..4607862 100644
--- a/storm-server/src/main/java/org/apache/storm/daemon/supervisor/Slot.java
+++ b/storm-server/src/main/java/org/apache/storm/daemon/supervisor/Slot.java
@@ -95,7 +95,7 @@ public class Slot extends Thread implements AutoCloseable, BlobChangingCallback
this.staticState = new StaticState(localizer,
ObjectReader.getInt(conf.get(Config.SUPERVISOR_WORKER_TIMEOUT_SECS)) * 1000,
ObjectReader.getInt(conf.get(DaemonConfig.SUPERVISOR_WORKER_START_TIMEOUT_SECS)) * 1000,
- ObjectReader.getInt(conf.get(DaemonConfig.SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS)) * 1000,
+ ObjectReader.getInt(conf.get(Config.SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS)) * 1000,
ObjectReader.getInt(conf.get(DaemonConfig.SUPERVISOR_MONITOR_FREQUENCY_SECS)) * 1000,
containerLauncher,
host,
diff --git a/storm-server/src/main/java/org/apache/storm/daemon/supervisor/Supervisor.java b/storm-server/src/main/java/org/apache/storm/daemon/supervisor/Supervisor.java
index de46676..ed79c35 100644
--- a/storm-server/src/main/java/org/apache/storm/daemon/supervisor/Supervisor.java
+++ b/storm-server/src/main/java/org/apache/storm/daemon/supervisor/Supervisor.java
@@ -508,7 +508,7 @@ public class Supervisor implements DaemonCommon, AutoCloseable {
LOG.error("Error trying to kill {}", workerId, e);
}
}
- int shutdownSleepSecs = ObjectReader.getInt(conf.get(DaemonConfig.SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS), 1);
+ int shutdownSleepSecs = ObjectReader.getInt(conf.get(Config.SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS));
if (!containers.isEmpty()) {
Time.sleepSecs(shutdownSleepSecs);
}
[storm] 04/06: Merge branch 'STORM-3353' of
https://github.com/srdo/storm into asfgit-master
Posted by sr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
commit 18e24a31745d9f6116e4c0dfca6e2d9d78a1625f
Merge: ca3a17a 7191246
Author: Stig Rohde Døssing <sr...@apache.org>
AuthorDate: Fri Mar 15 21:58:40 2019 +0100
Merge branch 'STORM-3353' of https://github.com/srdo/storm into asfgit-master
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
[storm] 05/06: Merge branch 'STORM-3312' of
https://github.com/srdo/storm into asfgit-master
Posted by sr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
commit 65ce33cc6dfdad5261dd1609dc95965262d6c299
Merge: 18e24a3 b25c242
Author: Stig Rohde Døssing <sr...@apache.org>
AuthorDate: Fri Mar 15 22:00:14 2019 +0100
Merge branch 'STORM-3312' of https://github.com/srdo/storm into asfgit-master
examples/storm-hdfs-examples/pom.xml | 10 ++++++++++
examples/storm-hive-examples/pom.xml | 10 ++++++++++
examples/storm-mqtt-examples/pom.xml | 10 ++++++++++
examples/storm-pmml-examples/pom.xml | 10 ++++++++++
examples/storm-solr-examples/pom.xml | 10 ++++++++++
external/storm-autocreds/pom.xml | 1 -
external/storm-blobstore-migration/pom.xml | 11 +++++++++++
external/storm-elasticsearch/pom.xml | 1 -
external/storm-hdfs-blobstore/pom.xml | 11 ++++++++++-
external/storm-hdfs/pom.xml | 10 ++++++++++
external/storm-hive/pom.xml | 12 +++++++++++-
external/storm-jdbc/pom.xml | 1 -
external/storm-mqtt/pom.xml | 10 ++++++++++
external/storm-opentsdb/pom.xml | 1 -
external/storm-pmml/pom.xml | 10 ++++++++++
external/storm-redis/pom.xml | 1 -
external/storm-solr/pom.xml | 4 +++-
pom.xml | 4 ++--
shaded-deps/pom.xml | 10 +++-------
sql/storm-sql-core/pom.xml | 10 ++++++++++
sql/storm-sql-external/storm-sql-hdfs/pom.xml | 10 ++++++++++
sql/storm-sql-external/storm-sql-kafka/pom.xml | 10 ++++++++++
sql/storm-sql-runtime/pom.xml | 10 ++++++++++
23 files changed, 160 insertions(+), 17 deletions(-)