You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by sr...@apache.org on 2019/03/07 06:13:32 UTC
[storm] branch master updated (b074136 -> e0feb6c)
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git.
from b074136 Merge branch 'STORM-3348-master' of https://github.com/OjhaVivek/storm into asfgit-master
add da12d89 STORM-3321: Fix race in LocalCluster regarding Nimbus leadership, reduce poll timers for Nimbus and supervisor to speed up tests and avoid timeouts
new 57befa1 Merge branch 'STORM-3321' of https://github.com/srdo/storm into asfgit-master
new da8c453 STORM-3347: Don't use maven-exec-plugin in storm-starter
new 002bc97 Merge branch 'STORM-3347' of https://github.com/srdo/storm into asfgit-master
new 41fe57a STORM-3349: Upgrade Hadoop, Hive, HDFS, HBase to latest compatible versions
new 0a4b433 Merge branch 'STORM-3349' of https://github.com/srdo/storm into asfgit-master
new 2329331 STORM-3350: Upgrade Jackson, ActiveMQ, commons-collections, commons-compress, Kafka and Maven plugins
new e0feb6c Merge branch 'STORM-3350' of https://github.com/srdo/storm into asfgit-master
The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../storm/hbase/topology/WordCountClient.java | 6 +-
examples/storm-jms-examples/pom.xml | 2 +-
examples/storm-mqtt-examples/pom.xml | 2 +-
examples/storm-starter/pom.xml | 34 +++--------
external/storm-autocreds/pom.xml | 16 +++++
.../org/apache/storm/hbase/common/HBaseClient.java | 14 ++---
.../java/org/apache/storm/hbase/common/Utils.java | 12 ++--
.../storm/hbase/trident/state/HBaseMapState.java | 6 +-
.../hbase/trident/windowing/HBaseWindowsStore.java | 21 ++++---
external/storm-hive/pom.xml | 1 +
.../storm/flux/examples/WordCountClient.java | 6 +-
pom.xml | 71 +++++++++++++++++++---
.../maven-shade-clojure-transformer/pom.xml | 2 +-
storm-buildtools/storm-maven-plugins/pom.xml | 2 +-
.../storm/metric/cgroup/CGroupMetricsBase.java | 2 +-
.../org/apache/storm/nimbus/ILeaderElector.java | 11 ++++
.../org/apache/storm/testing/FixedTupleSpout.java | 3 -
.../jvm/org/apache/storm/utils/JCQueueTest.java | 39 ++++++------
.../apache/storm/testing/MockLeaderElector.java | 6 ++
.../main/java/org/apache/storm/LocalCluster.java | 14 +++++
.../org/apache/storm/daemon/nimbus/Nimbus.java | 6 +-
.../storm/daemon/supervisor/ReadClusterState.java | 9 +--
.../apache/storm/daemon/supervisor/Supervisor.java | 3 +-
.../apache/storm/zookeeper/LeaderElectorImp.java | 8 +++
.../test/java/org/apache/storm/TestingTest.java | 5 +-
storm-submit-tools/pom.xml | 2 +-
26 files changed, 201 insertions(+), 102 deletions(-)
[storm] 03/07: STORM-3350: Upgrade Jackson, ActiveMQ,
commons-collections, commons-compress, Kafka and Maven plugins
Posted by sr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
commit 232933175b2a1d04eb7262881d2872daec75f5d8
Author: Stig Rohde Døssing <sr...@apache.org>
AuthorDate: Tue Mar 5 12:24:48 2019 +0100
STORM-3350: Upgrade Jackson, ActiveMQ, commons-collections, commons-compress, Kafka and Maven plugins
---
examples/storm-mqtt-examples/pom.xml | 2 +-
examples/storm-starter/pom.xml | 1 -
pom.xml | 8 ++++----
storm-buildtools/maven-shade-clojure-transformer/pom.xml | 2 +-
storm-buildtools/storm-maven-plugins/pom.xml | 2 +-
storm-submit-tools/pom.xml | 2 +-
6 files changed, 8 insertions(+), 9 deletions(-)
diff --git a/examples/storm-mqtt-examples/pom.xml b/examples/storm-mqtt-examples/pom.xml
index 97a3da3..3bdb88b 100644
--- a/examples/storm-mqtt-examples/pom.xml
+++ b/examples/storm-mqtt-examples/pom.xml
@@ -50,7 +50,7 @@
<dependency>
<groupId>org.fusesource.mqtt-client</groupId>
<artifactId>mqtt-client</artifactId>
- <version>1.10</version>
+ <version>1.14</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
diff --git a/examples/storm-starter/pom.xml b/examples/storm-starter/pom.xml
index 2841e93..5777d98 100644
--- a/examples/storm-starter/pom.xml
+++ b/examples/storm-starter/pom.xml
@@ -111,7 +111,6 @@
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
- <version>3.2.1</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
diff --git a/pom.xml b/pom.xml
index b8a8740..1def5d4 100644
--- a/pom.xml
+++ b/pom.xml
@@ -258,7 +258,7 @@
<!-- dependency versions -->
<clojure.version>1.7.0</clojure.version>
- <commons-compress.version>1.16.1</commons-compress.version>
+ <commons-compress.version>1.18</commons-compress.version>
<commons-io.version>2.6</commons-io.version>
<commons-lang.version>2.6</commons-lang.version>
<commons-exec.version>1.3</commons-exec.version>
@@ -308,12 +308,12 @@
<solr.version>5.2.1</solr.version>
<jpmml.version>1.0.22</jpmml.version>
<jedis.version>2.9.0</jedis.version>
- <activemq.version>5.15.3</activemq.version>
+ <activemq.version>5.15.8</activemq.version>
<rocketmq.version>4.2.0</rocketmq.version>
- <jackson.version>2.9.4</jackson.version>
+ <jackson.version>2.9.8</jackson.version>
- <storm.kafka.client.version>0.11.0.0</storm.kafka.client.version>
+ <storm.kafka.client.version>0.11.0.3</storm.kafka.client.version>
<!-- Java and clojure build lifecycle test properties are defined here to avoid having to create a default profile -->
<java.unit.test.exclude.groups>PerformanceTest</java.unit.test.exclude.groups>
diff --git a/storm-buildtools/maven-shade-clojure-transformer/pom.xml b/storm-buildtools/maven-shade-clojure-transformer/pom.xml
index e9cdd7c..b3fc7da 100644
--- a/storm-buildtools/maven-shade-clojure-transformer/pom.xml
+++ b/storm-buildtools/maven-shade-clojure-transformer/pom.xml
@@ -31,7 +31,7 @@
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
- <version>2.2</version>
+ <version>3.1.1</version>
<scope>provided</scope>
</dependency>
</dependencies>
diff --git a/storm-buildtools/storm-maven-plugins/pom.xml b/storm-buildtools/storm-maven-plugins/pom.xml
index d29866b..3619c35 100644
--- a/storm-buildtools/storm-maven-plugins/pom.xml
+++ b/storm-buildtools/storm-maven-plugins/pom.xml
@@ -31,7 +31,7 @@
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- <maven.dependency.version>3.0</maven.dependency.version>
+ <maven.dependency.version>3.6.0</maven.dependency.version>
</properties>
<dependencies>
diff --git a/storm-submit-tools/pom.xml b/storm-submit-tools/pom.xml
index dd944a4..6652515 100644
--- a/storm-submit-tools/pom.xml
+++ b/storm-submit-tools/pom.xml
@@ -97,7 +97,7 @@
<dependency>
<groupId>org.codehaus.plexus</groupId>
<artifactId>plexus-utils</artifactId>
- <version>2.1</version>
+ <version>3.1.1</version>
</dependency>
<dependency>
[storm] 01/07: STORM-3347: Don't use maven-exec-plugin in
storm-starter
Posted by sr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
commit da8c4539e7c697b2ca706a26a0a6092930dc7ae0
Author: Stig Rohde Døssing <sd...@it-minds.dk>
AuthorDate: Wed Feb 27 20:38:52 2019 +0100
STORM-3347: Don't use maven-exec-plugin in storm-starter
---
examples/storm-starter/pom.xml | 33 +++++++--------------------------
1 file changed, 7 insertions(+), 26 deletions(-)
diff --git a/examples/storm-starter/pom.xml b/examples/storm-starter/pom.xml
index 2841e93..e147497 100644
--- a/examples/storm-starter/pom.xml
+++ b/examples/storm-starter/pom.xml
@@ -225,33 +225,14 @@
<excludedGroups>none</excludedGroups>
</configuration>
</plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>exec-maven-plugin</artifactId>
- <executions>
- <execution>
- <goals>
- <goal>exec</goal>
- </goals>
- </execution>
- </executions>
- <configuration>
- <executable>java</executable>
- <includeProjectDependencies>true</includeProjectDependencies>
- <includePluginDependencies>false</includePluginDependencies>
- <classpathScope>compile</classpathScope>
- <!-- allows you to specify which topology class to run by specifying -Dstorm.topology= on the command line -->
- <mainClass>${storm.topology}</mainClass>
- </configuration>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-checkstyle-plugin</artifactId>
+ <!--Note - the version would be inherited-->
+ <configuration>
+ <maxAllowedViolations>263</maxAllowedViolations>
+ </configuration>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- <!--Note - the version would be inherited-->
- <configuration>
- <maxAllowedViolations>263</maxAllowedViolations>
- </configuration>
- </plugin>
</plugins>
</build>
</project>
[storm] 04/07: Merge branch 'STORM-3321' of
https://github.com/srdo/storm into asfgit-master
Posted by sr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
commit 57befa1a3dbfbbca93299223007559f00c1c6f55
Merge: b074136 da12d89
Author: Stig Rohde Døssing <sr...@apache.org>
AuthorDate: Thu Mar 7 07:10:12 2019 +0100
Merge branch 'STORM-3321' of https://github.com/srdo/storm into asfgit-master
.../org/apache/storm/metric/cgroup/CGroupMetricsBase.java | 2 +-
.../src/jvm/org/apache/storm/nimbus/ILeaderElector.java | 11 +++++++++++
.../src/jvm/org/apache/storm/testing/FixedTupleSpout.java | 3 ---
.../jvm/org/apache/storm/testing/MockLeaderElector.java | 6 ++++++
.../src/main/java/org/apache/storm/LocalCluster.java | 14 ++++++++++++++
.../main/java/org/apache/storm/daemon/nimbus/Nimbus.java | 6 +++++-
.../apache/storm/daemon/supervisor/ReadClusterState.java | 9 +++++----
.../org/apache/storm/daemon/supervisor/Supervisor.java | 3 ++-
.../java/org/apache/storm/zookeeper/LeaderElectorImp.java | 8 ++++++++
.../src/test/java/org/apache/storm/TestingTest.java | 5 +----
10 files changed, 53 insertions(+), 14 deletions(-)
[storm] 07/07: Merge branch 'STORM-3350' of
https://github.com/srdo/storm into asfgit-master
Posted by sr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
commit e0feb6cf04107029d4b24420f265495cdfbdb098
Merge: 0a4b433 2329331
Author: Stig Rohde Døssing <sr...@apache.org>
AuthorDate: Thu Mar 7 07:12:04 2019 +0100
Merge branch 'STORM-3350' of https://github.com/srdo/storm into asfgit-master
examples/storm-mqtt-examples/pom.xml | 2 +-
examples/storm-starter/pom.xml | 1 -
pom.xml | 8 ++++----
storm-buildtools/maven-shade-clojure-transformer/pom.xml | 2 +-
storm-buildtools/storm-maven-plugins/pom.xml | 2 +-
storm-submit-tools/pom.xml | 2 +-
6 files changed, 8 insertions(+), 9 deletions(-)
[storm] 05/07: Merge branch 'STORM-3347' of
https://github.com/srdo/storm into asfgit-master
Posted by sr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
commit 002bc97d4acd497f9bfb0e64edd131f6b34e7ea8
Merge: 57befa1 da8c453
Author: Stig Rohde Døssing <sr...@apache.org>
AuthorDate: Thu Mar 7 07:11:40 2019 +0100
Merge branch 'STORM-3347' of https://github.com/srdo/storm into asfgit-master
examples/storm-starter/pom.xml | 33 +++++++--------------------------
1 file changed, 7 insertions(+), 26 deletions(-)
[storm] 02/07: STORM-3349: Upgrade Hadoop, Hive, HDFS,
HBase to latest compatible versions
Posted by sr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
commit 41fe57a9a65f8b0007d327f01442ecb8d0982521
Author: Stig Rohde Døssing <sd...@it-minds.dk>
AuthorDate: Wed Feb 27 13:26:06 2019 +0100
STORM-3349: Upgrade Hadoop, Hive, HDFS, HBase to latest compatible versions
---
.../storm/hbase/topology/WordCountClient.java | 6 ++-
examples/storm-jms-examples/pom.xml | 2 +-
external/storm-autocreds/pom.xml | 16 ++++++
.../org/apache/storm/hbase/common/HBaseClient.java | 14 ++---
.../java/org/apache/storm/hbase/common/Utils.java | 12 +++--
.../storm/hbase/trident/state/HBaseMapState.java | 6 +--
.../hbase/trident/windowing/HBaseWindowsStore.java | 21 ++++----
external/storm-hive/pom.xml | 1 +
.../storm/flux/examples/WordCountClient.java | 6 ++-
pom.xml | 63 ++++++++++++++++++++--
.../jvm/org/apache/storm/utils/JCQueueTest.java | 39 +++++++-------
11 files changed, 133 insertions(+), 53 deletions(-)
diff --git a/examples/storm-hbase-examples/src/main/java/org/apache/storm/hbase/topology/WordCountClient.java b/examples/storm-hbase-examples/src/main/java/org/apache/storm/hbase/topology/WordCountClient.java
index b7874bd..33acfab 100644
--- a/examples/storm-hbase-examples/src/main/java/org/apache/storm/hbase/topology/WordCountClient.java
+++ b/examples/storm-hbase-examples/src/main/java/org/apache/storm/hbase/topology/WordCountClient.java
@@ -14,9 +14,11 @@ package org.apache.storm.hbase.topology;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Get;
-import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;
/**
@@ -32,7 +34,7 @@ public class WordCountClient {
config.set("hbase.rootdir", args[0]);
}
- HTable table = new HTable(config, "WordCount");
+ Table table = ConnectionFactory.createConnection(config).getTable(TableName.valueOf("WordCount"));
for (String word : WordSpout.words) {
diff --git a/examples/storm-jms-examples/pom.xml b/examples/storm-jms-examples/pom.xml
index 10266c1..886066c 100644
--- a/examples/storm-jms-examples/pom.xml
+++ b/examples/storm-jms-examples/pom.xml
@@ -29,7 +29,7 @@
<artifactId>storm-jms-examples</artifactId>
<properties>
- <spring.version>5.0.4.RELEASE</spring.version>
+ <spring.version>5.1.5.RELEASE</spring.version>
</properties>
<dependencies>
<dependency>
diff --git a/external/storm-autocreds/pom.xml b/external/storm-autocreds/pom.xml
index 6c69740..d819145 100644
--- a/external/storm-autocreds/pom.xml
+++ b/external/storm-autocreds/pom.xml
@@ -108,6 +108,22 @@
</exclusions>
</dependency>
<dependency>
+ <!-- Needed for TokenUtil, which moved here from hbase-client -->
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-server</artifactId>
+ <version>${hbase.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.zookeeper</groupId>
+ <artifactId>zookeeper</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
<groupId>org.apache.hive.hcatalog</groupId>
<artifactId>hive-hcatalog-streaming</artifactId>
<version>${hive.version}</version>
diff --git a/external/storm-hbase/src/main/java/org/apache/storm/hbase/common/HBaseClient.java b/external/storm-hbase/src/main/java/org/apache/storm/hbase/common/HBaseClient.java
index 208e622..2d0c324 100644
--- a/external/storm-hbase/src/main/java/org/apache/storm/hbase/common/HBaseClient.java
+++ b/external/storm-hbase/src/main/java/org/apache/storm/hbase/common/HBaseClient.java
@@ -22,13 +22,13 @@ import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Durability;
import org.apache.hadoop.hbase.client.Get;
-import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Mutation;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.security.UserProvider;
import org.apache.storm.hbase.bolt.mapper.HBaseProjectionCriteria;
import org.apache.storm.hbase.security.HBaseSecurityUtil;
@@ -41,7 +41,7 @@ import org.slf4j.LoggerFactory;
public class HBaseClient implements Closeable {
private static final Logger LOG = LoggerFactory.getLogger(HBaseClient.class);
- private HTable table;
+ private Table table;
public HBaseClient(Map<String, Object> map, final Configuration configuration, final String tableName) {
try {
@@ -60,14 +60,14 @@ public class HBaseClient implements Closeable {
put.setDurability(durability);
for (ColumnList.Column col : cols.getColumns()) {
if (col.getTs() > 0) {
- put.add(
+ put.addColumn(
col.getFamily(),
col.getQualifier(),
col.getTs(),
col.getValue()
);
} else {
- put.add(
+ put.addColumn(
col.getFamily(),
col.getQualifier(),
col.getValue()
@@ -82,9 +82,9 @@ public class HBaseClient implements Closeable {
inc.setDurability(durability);
for (ColumnList.Counter cnt : cols.getCounters()) {
inc.addColumn(
- cnt.getFamily(),
- cnt.getQualifier(),
- cnt.getIncrement()
+ cnt.getFamily(),
+ cnt.getQualifier(),
+ cnt.getIncrement()
);
}
mutations.add(inc);
diff --git a/external/storm-hbase/src/main/java/org/apache/storm/hbase/common/Utils.java b/external/storm-hbase/src/main/java/org/apache/storm/hbase/common/Utils.java
index f9e6e34..defa092 100644
--- a/external/storm-hbase/src/main/java/org/apache/storm/hbase/common/Utils.java
+++ b/external/storm-hbase/src/main/java/org/apache/storm/hbase/common/Utils.java
@@ -16,7 +16,9 @@ import java.io.IOException;
import java.math.BigDecimal;
import java.security.PrivilegedExceptionAction;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.client.HTable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.security.UserProvider;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.security.UserGroupInformation;
@@ -31,7 +33,7 @@ public class Utils {
private Utils() {}
- public static HTable getTable(UserProvider provider, Configuration config, String tableName)
+ public static Table getTable(UserProvider provider, Configuration config, String tableName)
throws IOException, InterruptedException {
UserGroupInformation ugi;
if (provider != null) {
@@ -74,10 +76,10 @@ public class Utils {
}
}
- return ugi.doAs(new PrivilegedExceptionAction<HTable>() {
+ return ugi.doAs(new PrivilegedExceptionAction<Table>() {
@Override
- public HTable run() throws IOException {
- return new HTable(config, tableName);
+ public Table run() throws IOException {
+ return ConnectionFactory.createConnection(config).getTable(TableName.valueOf(tableName));
}
});
}
diff --git a/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/state/HBaseMapState.java b/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/state/HBaseMapState.java
index d2a73d4..63169fb 100644
--- a/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/state/HBaseMapState.java
+++ b/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/state/HBaseMapState.java
@@ -28,10 +28,10 @@ import java.util.Map;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Get;
-import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException;
+import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.security.UserProvider;
import org.apache.storm.hbase.common.Utils;
import org.apache.storm.hbase.security.HBaseSecurityUtil;
@@ -72,7 +72,7 @@ public class HBaseMapState<T> implements IBackingMap<T> {
private int partitionNum;
private Options<T> options;
private Serializer<T> serializer;
- private HTable table;
+ private Table table;
/**
* Constructor.
@@ -183,7 +183,7 @@ public class HBaseMapState<T> implements IBackingMap<T> {
new Object[]{ this.partitionNum, new String(hbaseKey), new String(this.serializer.serialize(values.get(i))) });
Put put = new Put(hbaseKey);
T val = values.get(i);
- put.add(this.options.columnFamily.getBytes(),
+ put.addColumn(this.options.columnFamily.getBytes(),
qualifier.getBytes(),
this.serializer.serialize(val));
diff --git a/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/windowing/HBaseWindowsStore.java b/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/windowing/HBaseWindowsStore.java
index 086b477..3fe4bc4 100644
--- a/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/windowing/HBaseWindowsStore.java
+++ b/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/windowing/HBaseWindowsStore.java
@@ -23,12 +23,15 @@ import java.util.Map;
import java.util.Queue;
import java.util.concurrent.ConcurrentLinkedQueue;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter;
import org.apache.storm.trident.windowing.WindowKryoSerializer;
import org.apache.storm.trident.windowing.WindowsStore;
@@ -41,9 +44,9 @@ import org.slf4j.LoggerFactory;
public class HBaseWindowsStore implements WindowsStore {
public static final String UTF_8 = "utf-8";
private static final Logger LOG = LoggerFactory.getLogger(HBaseWindowsStore.class);
- private final ThreadLocal<HTable> threadLocalHtable;
+ private final ThreadLocal<Table> threadLocalHtable;
private final ThreadLocal<WindowKryoSerializer> threadLocalWindowKryoSerializer;
- private final Queue<HTable> htables = new ConcurrentLinkedQueue<>();
+ private final Queue<Table> htables = new ConcurrentLinkedQueue<>();
private final byte[] family;
private final byte[] qualifier;
@@ -52,13 +55,13 @@ public class HBaseWindowsStore implements WindowsStore {
this.family = family;
this.qualifier = qualifier;
- threadLocalHtable = new ThreadLocal<HTable>() {
+ threadLocalHtable = new ThreadLocal<Table>() {
@Override
- protected HTable initialValue() {
+ protected Table initialValue() {
try {
- HTable hTable = new HTable(config, tableName);
- htables.add(hTable);
- return hTable;
+ Table table = ConnectionFactory.createConnection(config).getTable(TableName.valueOf(tableName));
+ htables.add(table);
+ return table;
} catch (IOException e) {
throw new RuntimeException(e);
}
@@ -74,7 +77,7 @@ public class HBaseWindowsStore implements WindowsStore {
}
- private HTable htable() {
+ private Table htable() {
return threadLocalHtable.get();
}
@@ -252,7 +255,7 @@ public class HBaseWindowsStore implements WindowsStore {
@Override
public void shutdown() {
// close all the created hTable instances
- for (HTable htable : htables) {
+ for (Table htable : htables) {
try {
htable.close();
} catch (IOException e) {
diff --git a/external/storm-hive/pom.xml b/external/storm-hive/pom.xml
index 7723916..2fd8ae5 100644
--- a/external/storm-hive/pom.xml
+++ b/external/storm-hive/pom.xml
@@ -154,6 +154,7 @@
<artifactId>java-hamcrest</artifactId>
</dependency>
<dependency>
+ <!-- Needed in this version by hive -->
<groupId>org.apache.thrift</groupId>
<artifactId>libthrift</artifactId>
<version>0.9.3</version>
diff --git a/flux/flux-examples/src/main/java/org/apache/storm/flux/examples/WordCountClient.java b/flux/flux-examples/src/main/java/org/apache/storm/flux/examples/WordCountClient.java
index 6e732ae..484d3c1 100644
--- a/flux/flux-examples/src/main/java/org/apache/storm/flux/examples/WordCountClient.java
+++ b/flux/flux-examples/src/main/java/org/apache/storm/flux/examples/WordCountClient.java
@@ -23,9 +23,11 @@ import java.util.Properties;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Get;
-import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;
/**
@@ -61,7 +63,7 @@ public class WordCountClient {
System.exit(1);
}
- HTable table = new HTable(config, "WordCount");
+ Table table = ConnectionFactory.createConnection(config).getTable(TableName.valueOf("WordCount"));
String[] words = new String[] {"nathan", "mike", "jackson", "golda", "bertels"};
for (String word : words) {
diff --git a/pom.xml b/pom.xml
index b8a8740..f359fc5 100644
--- a/pom.xml
+++ b/pom.xml
@@ -287,10 +287,10 @@
<mockito.version>2.19.0</mockito.version>
<zookeeper.version>3.4.6</zookeeper.version>
<jline.version>0.9.94</jline.version>
- <hive.version>2.3.3</hive.version>
- <hadoop.version>2.6.1</hadoop.version>
+ <hive.version>2.3.4</hive.version>
+ <hadoop.version>2.8.5</hadoop.version>
<hdfs.version>${hadoop.version}</hdfs.version>
- <hbase.version>1.4.4</hbase.version>
+ <hbase.version>2.1.3</hbase.version>
<kryo.version>3.0.3</kryo.version>
<servlet.version>3.1.0</servlet.version>
<joda-time.version>2.3</joda-time.version>
@@ -301,7 +301,6 @@
<hdrhistogram.version>2.1.10</hdrhistogram.version>
<hamcrest.version>2.0.0.0</hamcrest.version>
<cassandra.version>2.1.7</cassandra.version>
- <druid.version>0.8.2</druid.version>
<elasticsearch.version>5.2.2</elasticsearch.version>
<calcite.version>1.14.0</calcite.version>
<mongodb.version>3.2.0</mongodb.version>
@@ -1084,6 +1083,62 @@
</exclusion>
</exclusions>
</dependency>
+ <!-- Hadoop dependencies. Specified here so HDFS/HBase don't import older versions of these jars in their projects-->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-auth</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-annotations</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-mapreduce-client-core</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-resourcemanager</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-applicationhistoryservice</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-web-proxy</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-registry</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-archives</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
</dependencies>
</dependencyManagement>
diff --git a/storm-client/test/jvm/org/apache/storm/utils/JCQueueTest.java b/storm-client/test/jvm/org/apache/storm/utils/JCQueueTest.java
index aa7714d..896f0c2 100644
--- a/storm-client/test/jvm/org/apache/storm/utils/JCQueueTest.java
+++ b/storm-client/test/jvm/org/apache/storm/utils/JCQueueTest.java
@@ -20,6 +20,7 @@ import org.apache.storm.policy.IWaitStrategy;
import org.apache.storm.policy.WaitStrategyPark;
import org.junit.Assert;
import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.RepeatedTest;
import org.junit.jupiter.api.Test;
public class JCQueueTest {
@@ -31,34 +32,32 @@ public class JCQueueTest {
@Test
public void testFirstMessageFirst() throws InterruptedException {
Assertions.assertTimeoutPreemptively(Duration.ofSeconds(10), () -> {
- for (int i = 0; i < 100; i++) {
- JCQueue queue = createQueue("firstMessageOrder", 16);
+ JCQueue queue = createQueue("firstMessageOrder", 16);
- queue.publish("FIRST");
+ queue.publish("FIRST");
- Runnable producer = new IncProducer(queue, i + 100, 1);
+ Runnable producer = new IncProducer(queue, 100, 1);
- final AtomicReference<Object> result = new AtomicReference<>();
- Runnable consumer = new ConsumerThd(queue, new JCQueue.Consumer() {
- private boolean head = true;
+ final AtomicReference<Object> result = new AtomicReference<>();
+ Runnable consumer = new ConsumerThd(queue, new JCQueue.Consumer() {
+ private boolean head = true;
- @Override
- public void accept(Object event) {
- if (head) {
- head = false;
- result.set(event);
- }
+ @Override
+ public void accept(Object event) {
+ if (head) {
+ head = false;
+ result.set(event);
}
+ }
- @Override
- public void flush() {
- }
- });
+ @Override
+ public void flush() {
+ }
+ });
- run(producer, consumer, queue);
- Assert.assertEquals("We expect to receive first published message first, but received " + result.get(),
+ run(producer, consumer, queue);
+ Assert.assertEquals("We expect to receive first published message first, but received " + result.get(),
"FIRST", result.get());
- }
});
}
[storm] 06/07: Merge branch 'STORM-3349' of
https://github.com/srdo/storm into asfgit-master
Posted by sr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
srdo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
commit 0a4b4330e585e5a40845b1752b14f9beacb92268
Merge: 002bc97 41fe57a
Author: Stig Rohde Døssing <sr...@apache.org>
AuthorDate: Thu Mar 7 07:11:57 2019 +0100
Merge branch 'STORM-3349' of https://github.com/srdo/storm into asfgit-master
.../storm/hbase/topology/WordCountClient.java | 6 ++-
examples/storm-jms-examples/pom.xml | 2 +-
external/storm-autocreds/pom.xml | 16 ++++++
.../org/apache/storm/hbase/common/HBaseClient.java | 14 ++---
.../java/org/apache/storm/hbase/common/Utils.java | 12 +++--
.../storm/hbase/trident/state/HBaseMapState.java | 6 +--
.../hbase/trident/windowing/HBaseWindowsStore.java | 21 ++++----
external/storm-hive/pom.xml | 1 +
.../storm/flux/examples/WordCountClient.java | 6 ++-
pom.xml | 63 ++++++++++++++++++++--
.../jvm/org/apache/storm/utils/JCQueueTest.java | 39 +++++++-------
11 files changed, 133 insertions(+), 53 deletions(-)