You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by bo...@apache.org on 2015/10/02 22:18:58 UTC
[1/8] storm git commit: Shaded everything that was not already shaded,
and cleaned up build so the dependency-reduced pom.xml is used correctly.
Repository: storm
Updated Branches:
refs/heads/master 768a85926 -> c90f6fd4d
Shaded everything that was not already shaded, and cleaned up build so the dependency-reduced pom.xml is used correctly.
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/12fdaa01
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/12fdaa01
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/12fdaa01
Branch: refs/heads/master
Commit: 12fdaa01b20eb144bf18a231a05c92873c90aa09
Parents: 768a859
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Fri Sep 11 11:56:34 2015 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Fri Oct 2 14:18:11 2015 -0500
----------------------------------------------------------------------
DEVELOPER.md | 34 +--
dev-tools/travis/travis-install.sh | 15 +-
dev-tools/travis/travis-script.sh | 2 +-
external/storm-elasticsearch/pom.xml | 5 +
external/storm-kafka/pom.xml | 5 +
external/storm-solr/pom.xml | 4 +-
pom.xml | 135 ++++++++---
storm-core/pom.xml | 253 +++++++++++---------
storm-core/src/clj/backtype/storm/log.clj | 2 +-
storm-dist/binary/src/main/assembly/binary.xml | 32 ---
10 files changed, 289 insertions(+), 198 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/12fdaa01/DEVELOPER.md
----------------------------------------------------------------------
diff --git a/DEVELOPER.md b/DEVELOPER.md
index 02fe135..7376949 100644
--- a/DEVELOPER.md
+++ b/DEVELOPER.md
@@ -241,17 +241,16 @@ in order to get started as fast as possible. Users can still install a specific
The following commands must be run from the top-level directory.
- # Build the code and run the tests (requires nodejs, python and ruby installed)
- # `mvn clean package` will fail because storm-core requires storm-maven-plugin.
- # This plugin should be installed before compiling storm-core.
- $ mvn clean install
+The first step is to build/install the plugins and storm-core
- # Build the code and run the tests, with specifying default test timeout (in millisecond)
- $ export STORM_TEST_TIMEOUT_MS=10000
- $ mvn clean install
+`mvn clean install -Pstorm-core`
- # Build the code but skip the tests
- $ mvn clean install -DskipTests=true
+If you wish to skip the unit tests you can do this by adding `-DskipTests` to the command line. Once this
+completes successfully you may run
+
+`mvn clean install -Pstorm-more`
+
+to build and test all of the external libraries and examples. Again if you want to include `-DskipTests` you can.
In case you modified `storm.thrift`, you have to regenerate thrift code as java and python code before compiling whole project.
@@ -266,19 +265,7 @@ You can also run tests selectively via the Clojure REPL. The following example
[auth_test.clj](storm-core/test/clj/backtype/storm/security/auth/auth_test.clj), which has the namespace
`backtype.storm.security.auth.auth-test`.
-First, start the REPL from within the relevant sub-project (here: `storm-core`):
-
- $ cd storm-core/
- $ mvn clojure:repl
-
-Now we run the tests in `auth_test.clj` in the REPL:
-
-```clojure
-;; You can use both absolute as well as relative paths to the .clj file.
-(load-file "test/clj/backtype/storm/security/auth/auth_test.clj")
-(ns backtype.storm.security.auth.auth-test)
-(run-tests)
-```
+You can also run tests selectively with `-Dtest=<test_name>`. This works for both clojure and junit tests.
> Tip: IDEs such as IntelliJ IDEA support a built-in Clojure REPL, which you can also use to run tests selectively.
> Sometimes you may find that tests pass/fail depending on which REPL you use, which -- although frustrating --
@@ -295,7 +282,8 @@ You can create a _distribution_ (like what you can download from Apache) as foll
do not use the Maven release plugin because creating an official release is the task of our release manager.
# First, build the code.
- $ mvn clean install # you may skip tests with `-DskipTests=true` to save time
+ $ mvn clean install -Pstorm-core # you may skip tests with `-DskipTests=true` to save time
+ $ mvn clean install -Pstorm-more # you may skip tests with `-DskipTests=true` to save time
# Create the binary distribution.
$ cd storm-dist/binary && mvn package
http://git-wip-us.apache.org/repos/asf/storm/blob/12fdaa01/dev-tools/travis/travis-install.sh
----------------------------------------------------------------------
diff --git a/dev-tools/travis/travis-install.sh b/dev-tools/travis/travis-install.sh
index 6af2f90..4857a1e 100755
--- a/dev-tools/travis/travis-install.sh
+++ b/dev-tools/travis/travis-install.sh
@@ -26,12 +26,23 @@ TRAVIS_SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
cd ${STORM_SRC_ROOT_DIR}
-python ${TRAVIS_SCRIPT_DIR}/save-logs.py "install.txt" mvn clean install --batch-mode -DskipTests -Pnative
+python ${TRAVIS_SCRIPT_DIR}/save-logs.py "install-storm-core.txt" mvn clean install -DskipTests -Pnative -Pstorm-core --batch-mode
BUILD_RET_VAL=$?
if [[ "$BUILD_RET_VAL" != "0" ]];
then
- cat "install.txt"
+ cat "install-storm-core.txt"
+ exit ${BUILD_RET_VAL}
+fi
+
+exit ${BUILD_RET_VAL}
+
+python ${TRAVIS_SCRIPT_DIR}/save-logs.py "install-storm-more.txt" mvn clean install -DskipTests -Pstorm-more
+BUILD_RET_VAL=$?
+
+if [[ "$BUILD_RET_VAL" != "0" ]];
+then
+ cat "install-storm-more.txt"
fi
exit ${BUILD_RET_VAL}
http://git-wip-us.apache.org/repos/asf/storm/blob/12fdaa01/dev-tools/travis/travis-script.sh
----------------------------------------------------------------------
diff --git a/dev-tools/travis/travis-script.sh b/dev-tools/travis/travis-script.sh
index 4caff17..c59cb1a 100755
--- a/dev-tools/travis/travis-script.sh
+++ b/dev-tools/travis/travis-script.sh
@@ -31,7 +31,7 @@ cd ${STORM_SRC_ROOT_DIR}
export STORM_TEST_TIMEOUT_MS=100000
# We now lean on Travis CI's implicit behavior, ```mvn clean install -DskipTests``` before running script
-mvn --batch-mode test -fae -Pnative
+mvn --batch-mode test -fae -Pnative -Pstorm-core && mvn test -fae -Pnative -Pstorm-more
BUILD_RET_VAL=$?
for dir in `find . -type d -and -wholename \*/target/\*-reports`;
http://git-wip-us.apache.org/repos/asf/storm/blob/12fdaa01/external/storm-elasticsearch/pom.xml
----------------------------------------------------------------------
diff --git a/external/storm-elasticsearch/pom.xml b/external/storm-elasticsearch/pom.xml
index 7777549..fd263e6 100644
--- a/external/storm-elasticsearch/pom.xml
+++ b/external/storm-elasticsearch/pom.xml
@@ -41,6 +41,11 @@
<dependencies>
<dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>${project.version}</version>
http://git-wip-us.apache.org/repos/asf/storm/blob/12fdaa01/external/storm-kafka/pom.xml
----------------------------------------------------------------------
diff --git a/external/storm-kafka/pom.xml b/external/storm-kafka/pom.xml
index c731a0f..8f2867b 100644
--- a/external/storm-kafka/pom.xml
+++ b/external/storm-kafka/pom.xml
@@ -57,6 +57,11 @@
</build>
<dependencies>
<dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.9.0</version>
http://git-wip-us.apache.org/repos/asf/storm/blob/12fdaa01/external/storm-solr/pom.xml
----------------------------------------------------------------------
diff --git a/external/storm-solr/pom.xml b/external/storm-solr/pom.xml
index 2b5c742..5e7caa4 100644
--- a/external/storm-solr/pom.xml
+++ b/external/storm-solr/pom.xml
@@ -21,7 +21,7 @@
</developers>
<dependencies>
- <dependency>
+ <dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>${project.version}</version>
@@ -95,4 +95,4 @@
</plugin>
</plugins>
</build>
-</project>
\ No newline at end of file
+</project>
http://git-wip-us.apache.org/repos/asf/storm/blob/12fdaa01/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 86ce5fd..edb7f2b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -154,26 +154,6 @@
<maven>3.0.0</maven>
</prerequisites>
- <modules>
- <module>storm-buildtools/maven-shade-clojure-transformer</module>
- <module>storm-buildtools/storm-maven-plugins</module>
- <module>storm-multilang/javascript</module>
- <module>storm-multilang/python</module>
- <module>storm-multilang/ruby</module>
- <module>storm-core</module>
- <module>external/storm-kafka</module>
- <module>external/storm-hdfs</module>
- <module>external/storm-hbase</module>
- <module>external/storm-hive</module>
- <module>external/storm-jdbc</module>
- <module>external/storm-redis</module>
- <module>external/storm-eventhubs</module>
- <module>external/flux</module>
- <module>external/storm-elasticsearch</module>
- <module>external/storm-solr</module>
- <module>examples/storm-starter</module>
- </modules>
-
<scm>
<connection>scm:git:https://git-wip-us.apache.org/repos/asf/storm.git</connection>
<developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/storm.git</developerConnection>
@@ -197,10 +177,13 @@
<commons-io.version>2.4</commons-io.version>
<commons-lang.version>2.5</commons-lang.version>
<commons-exec.version>1.1</commons-exec.version>
+ <commons-fileupload.version>1.2.1</commons-fileupload.version>
+ <commons-codec.version>1.6</commons-codec.version>
<clj-time.version>0.8.0</clj-time.version>
<curator.version>2.5.0</curator.version>
<json-simple.version>1.1</json-simple.version>
<ring.version>1.3.0</ring.version>
+ <ring-json.version>0.3.1</ring-json.version>
<jetty.version>7.6.13.v20130916</jetty.version>
<clojure.tools.logging.version>0.2.3</clojure.tools.logging.version>
<clojure.math.numeric-tower.version>0.0.1</clojure.math.numeric-tower.version>
@@ -214,6 +197,7 @@
<netty.version>3.9.0.Final</netty.version>
<log4j-over-slf4j.version>1.6.6</log4j-over-slf4j.version>
<log4j.version>2.1</log4j.version>
+ <slf4j.version>1.7.7</slf4j.version>
<clojure.tools.nrepl.version>0.2.3</clojure.tools.nrepl.version>
<clojure-complete.version>0.2.3</clojure-complete.version>
<mockito.version>1.9.5</mockito.version>
@@ -224,10 +208,48 @@
<hive.version>0.14.0</hive.version>
<hadoop.version>2.6.0</hadoop.version>
<kryo.version>2.21</kryo.version>
+ <servlet.version>2.5</servlet.version>
+ <joda-time.version>2.3</joda-time.version>
+ <jackson.version>2.3.1</jackson.version>
+ <thrift.version>0.9.2</thrift.version>
+ <junit.version>4.11</junit.version>
</properties>
<profiles>
<profile>
+ <id>storm-core</id>
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <modules>
+ <module>storm-multilang/javascript</module>
+ <module>storm-multilang/python</module>
+ <module>storm-multilang/ruby</module>
+ <module>storm-buildtools/maven-shade-clojure-transformer</module>
+ <module>storm-buildtools/storm-maven-plugins</module>
+ <module>storm-core</module>
+ </modules>
+ </profile>
+ <profile>
+ <id>storm-more</id>
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <modules>
+ <module>external/storm-kafka</module>
+ <module>external/storm-hdfs</module>
+ <module>external/storm-hbase</module>
+ <module>external/storm-hive</module>
+ <module>external/storm-jdbc</module>
+ <module>external/storm-redis</module>
+ <module>external/storm-eventhubs</module>
+ <module>external/flux</module>
+ <module>external/storm-elasticsearch</module>
+ <module>external/storm-solr</module>
+ <module>examples/storm-starter</module>
+ </modules>
+ </profile>
+ <profile>
<id>sign</id>
<build>
<plugins>
@@ -380,6 +402,41 @@
</exclusions>
</dependency>
<dependency>
+ <groupId>javax.servlet</groupId>
+ <artifactId>servlet-api</artifactId>
+ <version>${servlet.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ <version>${slf4j.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>joda-time</groupId>
+ <artifactId>joda-time</artifactId>
+ <version>${joda-time.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-core</artifactId>
+ <version>${jackson.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.dataformat</groupId>
+ <artifactId>jackson-dataformat-smile</artifactId>
+ <version>${jackson.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>commons-fileupload</groupId>
+ <artifactId>commons-fileupload</artifactId>
+ <version>${commons-fileupload.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>commons-codec</groupId>
+ <artifactId>commons-codec</artifactId>
+ <version>${commons-codec.version}</version>
+ </dependency>
+ <dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-framework</artifactId>
<version>${curator.version}</version>
@@ -409,6 +466,11 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.curator</groupId>
+ <artifactId>curator-client</artifactId>
+ <version>${curator.version}</version>
+ </dependency>
<dependency>
<groupId>com.googlecode.json-simple</groupId>
@@ -436,6 +498,11 @@
<version>${ring.version}</version>
</dependency>
<dependency>
+ <groupId>ring</groupId>
+ <artifactId>ring-json</artifactId>
+ <version>${ring-json.version}</version>
+ </dependency>
+ <dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlet</artifactId>
<version>${jetty.version}</version>
@@ -446,6 +513,17 @@
<version>${jetty.version}</version>
</dependency>
<dependency>
+ <groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-server</artifactId>
+ <version>${jetty.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-util</artifactId>
+ <version>${jetty.version}</version>
+ </dependency>
+
+ <dependency>
<groupId>org.clojure</groupId>
<artifactId>tools.logging</artifactId>
<version>${clojure.tools.logging.version}</version>
@@ -570,16 +648,15 @@
<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libthrift</artifactId>
- <version>0.9.2</version>
+ <version>${thrift.version}</version>
<scope>compile</scope>
</dependency>
- <!-- used by examples/storm-starter -->
- <dependency>
- <groupId>junit</groupId>
- <artifactId>junit</artifactId>
- <version>4.11</version>
- <scope>test</scope>
- </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>${junit.version}</version>
+ <scope>test</scope>
+ </dependency>
</dependencies>
</dependencyManagement>
@@ -673,7 +750,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
- <version>2.2</version>
+ <version>2.4.1</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
http://git-wip-us.apache.org/repos/asf/storm/blob/12fdaa01/storm-core/pom.xml
----------------------------------------------------------------------
diff --git a/storm-core/pom.xml b/storm-core/pom.xml
index 3eb4112..ca40cb8 100644
--- a/storm-core/pom.xml
+++ b/storm-core/pom.xml
@@ -39,6 +39,12 @@
<dependency>
<groupId>com.esotericsoftware.kryo</groupId>
<artifactId>kryo</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>org.objenesis</groupId>
+ <artifactId>objenesis</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
<!--clojure-->
<dependency>
@@ -74,7 +80,6 @@
<dependency>
<groupId>ring</groupId>
<artifactId>ring-json</artifactId>
- <version>0.3.1</version>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
@@ -141,7 +146,6 @@
<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libthrift</artifactId>
- <version>0.9.2</version>
<scope>compile</scope>
<exclusions>
<exclusion>
@@ -154,8 +158,6 @@
</exclusion>
</exclusions>
</dependency>
-
-
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-framework</artifactId>
@@ -192,8 +194,6 @@
<artifactId>json-simple</artifactId>
<scope>compile</scope>
</dependency>
-
-
<dependency>
<groupId>com.twitter</groupId>
<artifactId>carbonite</artifactId>
@@ -244,10 +244,53 @@
<scope>test</scope>
</dependency>
<dependency>
- <groupId>junit</groupId>
- <artifactId>junit</artifactId>
- <version>4.11</version>
- <scope>test</scope>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.zookeeper</groupId>
+ <artifactId>zookeeper</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>javax.servlet</groupId>
+ <artifactId>servlet-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>joda-time</groupId>
+ <artifactId>joda-time</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-server</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-util</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-core</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.dataformat</groupId>
+ <artifactId>jackson-dataformat-smile</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.curator</groupId>
+ <artifactId>curator-client</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>commons-fileupload</groupId>
+ <artifactId>commons-fileupload</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>commons-codec</groupId>
+ <artifactId>commons-codec</artifactId>
</dependency>
</dependencies>
<build>
@@ -360,23 +403,22 @@
<minimizeJar>false</minimizeJar>
<artifactSet>
<includes>
+ <include>ns-tracker:ns-tracker</include>
+ <include>hiccup:hiccup</include>
+ <include>ring:*</include>
+ <include>compojure:compojure</include>
+ <include>clj-time:clj-time</include>
<include>org.apache.thrift:*</include>
<include>io.netty:netty</include>
<include>com.google.guava:guava</include>
<include>org.apache.httpcomponents:http*</include>
<include>org.apache.zookeeper:zookeeper</include>
<include>org.apache.curator:*</include>
- <include>org.apache.httpcomponents:http*</include>
- <include>org.apache.zookeeper:zookeeper</include>
- <include>org.apache.curator:*</include>
<include>com.twitter:carbonite</include>
<include>com.twitter:chill-java</include>
- <include>org.objenesis:objenesis</include>
<include>org.tukaani:xz</include>
<include>org.yaml:snakeyaml</include>
<include>org.jgrapht:jgrapht-core</include>
- <include>commons-httpclient:commons-httpclient</include>
- <include>org.apache.commons:commons-compress</include>
<include>org.apache.commons:commons-exec</include>
<include>commons-io:commons-io</include>
<include>commons-codec:commons-codec</include>
@@ -385,168 +427,199 @@
<include>com.googlecode.json-simple:json-simple</include>
<include>org.clojure:math.numeric-tower</include>
<include>org.clojure:tools.cli</include>
+ <include>org.clojure:tools.logging</include>
<include>org.clojure:tools.macro</include>
+ <include>org.clojure:java.jmx</include>
<include>joda-time:joda-time</include>
<include>org.eclipse.jetty:*</include>
<include>com.fasterxml.jackson.core:*</include>
<include>com.fasterxml.jackson.dataformat:*</include>
+ <include>clout:clout</include>
+ <include>org.clojure:tools.namespace</include>
+ <include>cheshire:cheshire</include>
+ <include>org.clojure:core.incubator</include>
</includes>
</artifactSet>
-
<relocations>
<relocation>
- <pattern>com.fasterxml.jackson.core</pattern>
- <shadedPattern>com.fasterxml.jackson.storm.core</shadedPattern>
+ <pattern>cheshire</pattern>
+ <shadedPattern>org.apache.storm.shade.cheshire</shadedPattern>
</relocation>
<relocation>
- <pattern>com.fasterxml.jackson.dataformat</pattern>
- <shadedPattern>com.fasterxml.jackson.storm.dataformat</shadedPattern>
+ <pattern>clojure.tools.logging</pattern>
+ <shadedPattern>org.apache.storm.shade.clojure.tools.logging</shadedPattern>
</relocation>
<relocation>
- <pattern>org.apache.thrift</pattern>
- <shadedPattern>org.apache.thrift7</shadedPattern>
+ <pattern>clojure.core.incubator</pattern>
+ <shadedPattern>org.apache.storm.shade.clojure.core.incubator</shadedPattern>
</relocation>
<relocation>
- <pattern>org.jboss.netty</pattern>
- <shadedPattern>org.apache.storm.netty</shadedPattern>
+ <pattern>clojure.tools.namespace</pattern>
+ <shadedPattern>org.apache.storm.shade.clojure.tools.namespace</shadedPattern>
</relocation>
<relocation>
- <pattern>com.google.common</pattern>
- <shadedPattern>org.apache.storm.guava</shadedPattern>
+ <pattern>clout</pattern>
+ <shadedPattern>org.apache.storm.shade.clout</shadedPattern>
</relocation>
<relocation>
- <pattern>com.google.thirdparty</pattern>
- <shadedPattern>org.apache.storm.guava.thirdparty</shadedPattern>
+ <pattern>compojure</pattern>
+ <shadedPattern>org.apache.storm.shade.compojure</shadedPattern>
</relocation>
<relocation>
- <pattern>org.apache.http</pattern>
- <shadedPattern>org.apache.storm.http</shadedPattern>
+ <pattern>ns_tracker</pattern>
+ <shadedPattern>org.apache.storm.shade.ns_tracker</shadedPattern>
</relocation>
<relocation>
- <pattern>org.apache.zookeeper</pattern>
- <shadedPattern>org.apache.storm.zookeeper</shadedPattern>
+ <pattern>ns-tracker</pattern>
+ <shadedPattern>org.apache.storm.shade.ns-tracker</shadedPattern>
</relocation>
<relocation>
- <pattern>org.apache.curator</pattern>
- <shadedPattern>org.apache.storm.curator</shadedPattern>
+ <pattern>hiccup</pattern>
+ <shadedPattern>org.apache.storm.shade.hiccup</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>ring</pattern>
+ <shadedPattern>org.apache.storm.shade.ring</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>clj_time</pattern>
+ <shadedPattern>org.apache.storm.shade.clj_time</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>clj-time</pattern>
+ <shadedPattern>org.apache.storm.shade.clj-time</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>com.fasterxml</pattern>
+ <shadedPattern>org.apache.storm.shade.com.fasterxml</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.thrift</pattern>
+ <!-- This pattern is inconsistent for backwards compatability purposes. -->
+ <shadedPattern>org.apache.thrift7</shadedPattern>
</relocation>
<relocation>
<pattern>org.jboss.netty</pattern>
- <shadedPattern>org.apache.storm.netty</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.org.jboss.netty</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.common</pattern>
- <shadedPattern>org.apache.storm.guava</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.com.google.common</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.thirdparty</pattern>
- <shadedPattern>org.apache.storm.guava.thirdparty</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.com.google.thirdparty</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.http</pattern>
- <shadedPattern>org.apache.storm.http</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.org.apache.http</shadedPattern>
</relocation>
<relocation>
- <pattern>org.apache.zookeeper</pattern>
- <shadedPattern>org.apache.storm.zookeeper</shadedPattern>
+ <pattern>org.apache.curator</pattern>
+ <shadedPattern>org.apache.storm.shade.org.apache.curator</shadedPattern>
</relocation>
<relocation>
- <pattern>org.apache.jute</pattern>
- <shadedPattern>org.apache.storm.jute</shadedPattern>
+ <pattern>org.apache.zookeeper</pattern>
+ <shadedPattern>org.apache.storm.shade.org.apache.zookeeper</shadedPattern>
</relocation>
<relocation>
- <pattern>org.apache.curator</pattern>
- <shadedPattern>org.apache.storm.curator</shadedPattern>
+ <pattern>org.apache.jute</pattern>
+ <shadedPattern>org.apache.storm.shade.org.apache.jute</shadedPattern>
</relocation>
<relocation>
<pattern>carbonite</pattern>
- <shadedPattern>org.apache.storm.carbonite</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.carbonite</shadedPattern>
</relocation>
<relocation>
<pattern>com.twitter.chill</pattern>
- <shadedPattern>org.apache.storm.chill</shadedPattern>
- </relocation>
- <relocation>
- <pattern>org.objenesis</pattern>
- <shadedPattern>org.apache.storm.objenesis</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.com.twitter.chill</shadedPattern>
</relocation>
<relocation>
<pattern>org.tukaani.xz</pattern>
- <shadedPattern>org.apache.storm.xz</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.org.tukaani.xz</shadedPattern>
</relocation>
<relocation>
<pattern>org.yaml.snakeyaml</pattern>
- <shadedPattern>org.apache.storm.snakeyaml</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.org.yaml.snakeyaml</shadedPattern>
</relocation>
<relocation>
<pattern>org.jgrapht</pattern>
- <shadedPattern>org.apache.storm.jgrapht</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.org.jgrapht</shadedPattern>
</relocation>
<relocation>
<pattern>org.fusesource</pattern>
- <shadedPattern>org.apache.storm.fusesource</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.org.fusesource</shadedPattern>
</relocation>
<relocation>
<pattern>com.metamx.http.client</pattern>
- <shadedPattern>org.apache.storm.metamx.http.client</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.com.metamx.http.client</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.io</pattern>
- <shadedPattern>org.apache.storm.commons.io</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.org.apache.commons.io</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.codec</pattern>
- <shadedPattern>org.apache.storm.commons.codec</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.org.apache.commons.codec</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.fileupload</pattern>
- <shadedPattern>org.apache.storm.commons.fileupload</shadedPattern>
- </relocation>
- <relocation>
- <pattern>org.apache.commons.compress</pattern>
- <shadedPattern>org.apache.storm.commons.compress</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.org.apache.commons.fileupload</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.exec</pattern>
- <shadedPattern>org.apache.storm.commons.exec</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.org.apache.commons.exec</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.lang</pattern>
- <shadedPattern>org.apache.storm.commons.lang</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.org.apache.commons.lang</shadedPattern>
</relocation>
<relocation>
<pattern>org.json.simple</pattern>
- <shadedPattern>org.apache.storm.json.simple</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.org.json.simple</shadedPattern>
</relocation>
<relocation>
<pattern>clojure.math</pattern>
- <shadedPattern>org.apache.storm.clojure.math</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.clojure.math</shadedPattern>
</relocation>
<relocation>
<pattern>clojure.tools.cli</pattern>
- <shadedPattern>org.apache.storm.clojure.tools.cli</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.clojure.tools.cli</shadedPattern>
</relocation>
<relocation>
<pattern>cljs.tools.cli</pattern>
- <shadedPattern>org.apache.storm.cljs.tools.cli</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.cljs.tools.cli</shadedPattern>
</relocation>
<relocation>
<pattern>clojure.tools.macro</pattern>
- <shadedPattern>org.apache.storm.clojure.tools.macro</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.clojure.tools.macro</shadedPattern>
</relocation>
<relocation>
<pattern>org.joda.time</pattern>
- <shadedPattern>org.apache.storm.joda.time</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.org.joda.time</shadedPattern>
</relocation>
<relocation>
<pattern>org.eclipse.jetty</pattern>
- <shadedPattern>org.apache.storm.jetty</shadedPattern>
+ <shadedPattern>org.apache.storm.shade.org.eclipse.jetty</shadedPattern>
</relocation>
</relocations>
<transformers>
<transformer implementation="org.apache.storm.maven.shade.clojure.ClojureTransformer" />
</transformers>
<filters>
+ <filter><artifact>org.clojure:core.incubator</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
+ <filter><artifact>cheshire:cheshire</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
+ <filter><artifact>org.clojure:tools.logging</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
+ <filter><artifact>org.clojure:tools.namespace</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
+ <filter><artifact>org.clojure:math.numeric-tower</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
+ <filter><artifact>org.clojure:tools.macro</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
+ <filter><artifact>org.clojure:tools.cli</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
+ <filter><artifact>ns-tracker:ns-tracker</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
+ <filter><artifact>clout:clout</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
+ <filter><artifact>hiccup:hiccup</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
+ <filter><artifact>clj-time:clj-time</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
+ <filter><artifact>ring:*</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
+ <filter><artifact>compojure:compojure</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
<filter>
<artifact>org.apache.thrift:*</artifact>
<excludes>
@@ -562,48 +635,12 @@
</excludes>
</filter>
<filter>
- <artifact>commons-httpclient:commons-httpclient</artifact>
- <excludes>
- <exclude>META-INF/LICENSE.txt</exclude>
- <exclude>META-INF/NOTICE.txt</exclude>
- <exclude>META-INF/README.txt</exclude>
- </excludes>
- </filter>
- <filter>
<artifact>org.apache.zookeeper:zookeeper</artifact>
<excludes>
<exclude>LICENSE.txt</exclude>
</excludes>
</filter>
<filter>
- <artifact>commons-httpclient:commons-httpclient</artifact>
- <excludes>
- <exclude>META-INF/LICENSE.txt</exclude>
- <exclude>META-INF/NOTICE.txt</exclude>
- <exclude>META-INF/README.txt</exclude>
- </excludes>
- </filter>
- <filter>
- <artifact>org.apache.zookeeper:zookeeper</artifact>
- <excludes>
- <exclude>LICENSE.txt</exclude>
- </excludes>
- </filter>
- <filter>
- <artifact>org.objenesis:objenesis</artifact>
- <excludes>
- <exclude>META-INF/LICENSE.txt</exclude>
- <exclude>META-INF/NOTICE.txt</exclude>
- </excludes>
- </filter>
- <filter>
- <artifact>org.apache.commons:commons-compress</artifact>
- <excludes>
- <exclude>META-INF/LICENSE.txt</exclude>
- <exclude>META-INF/NOTICE.txt</exclude>
- </excludes>
- </filter>
- <filter>
<artifact>org.apache.commons:commons-exec</artifact>
<excludes>
<exclude>META-INF/LICENSE.txt</exclude>
http://git-wip-us.apache.org/repos/asf/storm/blob/12fdaa01/storm-core/src/clj/backtype/storm/log.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/log.clj b/storm-core/src/clj/backtype/storm/log.clj
index 0fcf822..55b44fe 100644
--- a/storm-core/src/clj/backtype/storm/log.clj
+++ b/storm-core/src/clj/backtype/storm/log.clj
@@ -15,7 +15,7 @@
;; limitations under the License.
(ns backtype.storm.log
- (:require [clojure.tools [logging :as log]]))
+ (:require [clojure.tools.logging :as log]))
(defmacro log-message
[& args]
http://git-wip-us.apache.org/repos/asf/storm/blob/12fdaa01/storm-dist/binary/src/main/assembly/binary.xml
----------------------------------------------------------------------
diff --git a/storm-dist/binary/src/main/assembly/binary.xml b/storm-dist/binary/src/main/assembly/binary.xml
index 9f6c8f7..4cd6911 100644
--- a/storm-dist/binary/src/main/assembly/binary.xml
+++ b/storm-dist/binary/src/main/assembly/binary.xml
@@ -30,38 +30,6 @@
<useProjectArtifact>false</useProjectArtifact>
<outputDirectory>lib</outputDirectory>
<unpack>false</unpack>
- <excludes>
- <exclude>org.apache.thrift:*</exclude>
- <exclude>io.netty:netty</exclude>
- <exclude>com.google.guava:guava</exclude>
- <exclude>org.apache.httpcomponents:http*</exclude>
- <exclude>org.apache.zookeeper:zookeeper</exclude>
- <exclude>org.apache.curator:*</exclude>
- <exclude>org.apache.httpcomponents:http*</exclude>
- <exclude>org.apache.zookeeper:zookeeper</exclude>
- <exclude>org.apache.curator:*</exclude>
- <exclude>com.twitter:carbonite</exclude>
- <exclude>com.twitter:chill-java</exclude>
- <exclude>org.objenesis:objenesis</exclude>
- <exclude>org.tukaani:xz</exclude>
- <exclude>org.yaml:snakeyaml</exclude>
- <exclude>org.jgrapht:jgrapht-core</exclude>
- <exclude>commons-httpclient:commons-httpclient</exclude>
- <exclude>org.apache.commons:commons-compress</exclude>
- <exclude>org.apache.commons:commons-exec</exclude>
- <exclude>commons-io:commons-io</exclude>
- <exclude>commons-codec:commons-codec</exclude>
- <exclude>commons-fileupload:commons-fileupload</exclude>
- <exclude>commons-lang:commons-lang</exclude>
- <exclude>com.googlecode.json-simple:json-simple</exclude>
- <exclude>org.clojure:math.numeric-tower</exclude>
- <exclude>org.clojure:tools.cli</exclude>
- <exclude>org.clojure:tools.macro</exclude>
- <exclude>joda-time:joda-time</exclude>
- <exclude>org.eclipse.jetty:*</exclude>
- <exclude>com.fasterxml.jackson.core:*</exclude>
- <exclude>com.fasterxml.jackson.dataformat:*</exclude>
- </excludes>
</dependencySet>
</dependencySets>
[5/8] storm git commit: Fixed issues with split build
Posted by bo...@apache.org.
Fixed issues with split build
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/25b6b074
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/25b6b074
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/25b6b074
Branch: refs/heads/master
Commit: 25b6b074c58f9d23081eabb033d7320be58cd3ab
Parents: f04325b
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Fri Oct 2 13:53:31 2015 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Fri Oct 2 14:19:59 2015 -0500
----------------------------------------------------------------------
DEVELOPER.md | 48 +++++-----------------------
dev-tools/travis/travis-install.sh | 13 ++------
dev-tools/travis/travis-script.sh | 5 +--
examples/storm-starter/pom.xml | 1 -
external/flux/flux-core/pom.xml | 1 -
external/flux/flux-examples/pom.xml | 1 -
external/storm-eventhubs/pom.xml | 1 -
pom.xml | 54 +++++++++++++-------------------
8 files changed, 33 insertions(+), 91 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/25b6b074/DEVELOPER.md
----------------------------------------------------------------------
diff --git a/DEVELOPER.md b/DEVELOPER.md
index 541afb2..4034a45 100644
--- a/DEVELOPER.md
+++ b/DEVELOPER.md
@@ -192,6 +192,7 @@ To pull in a merge request you should generally follow the command line instruct
3. Merge the pull request into your local test branch.
$ git pull <remote_repo_url> <remote_branch>
+ You can use `./dev-tools/storm-merge.py <pull-number>` to produce the above command most of the time.
4. Assuming that the pull request merges without any conflicts:
Update the top-level `CHANGELOG.md`, and add in the JIRA ticket number (example: `STORM-1234`) and ticket
@@ -222,36 +223,10 @@ To pull in a merge request you should generally follow the command line instruct
# Build the code and run the tests
-Maven has some issues with maven pugins, shading and multi-module projects which force Storm's build process to be somewhat non-standard.
-
-Essentially what happens is that if you build a multi-module project where one module depends on another module that is a part of the same build, maven tries to be smart and will use the pom.xml and .jar or .class directory from the module that is a dependency.
-
-So if we have a project set up like the following where we have flux depend on storm-core, and storm-core is shaded. flux will see different versions of the code and pom.xml from storm-core depending on what maven command is run.
-
-- storm.git/
- - pom.xml
- - storm-core/
- - pom.xml
- - dependency-reduce-pom.xml
- - target/
- - classes/
- - storm-core.jar (shaded)
- - original.storm-core.jar (no-shaded)
- - flux/
- - pom.xml (depends on storm-core)
- - target/
- - classes/
- - flux.uber.jar
-
-If I run `mvn clean compile` the classpath for compiling flux will include `storm.git/storm-core/target/classes` which are not shaded, and if flux is written to use storm-core shaded class it will fail to compile. flux will also see the full set of non-shaded dependencies from storm-core, so if flux is creating an uber jar, it will assume that all of those dependencies will be provided by storm-core, when in reality they will not be.
-
-If I run `mvn clean package` or `mvn clean install` flux will see the shaded storm-core.jar on its classpath, but it will still see the full non-shaded set of storm-core's dependencies.
-
-The only way I found to make flux use the dependency-reduced-pom.xml from storm-core, is to first build and install storm-core by itself, and then build and install flux without storm-core. That way storm-core will install the shaded jar and dependency-reduced-pom.xml into the local repository. Then flux will get its dependencies from the local repo instead of trying to rebuild storm-core.
-
-Because of this we have split the build into two profiles storm-core and storm-more. By default both build together which works OK for simple testing, but when doing a release it is important to build them in two phases or flux, storm-starter, and possibly others will not be packaged correctly.
-
## Prerequisites
+Firt of all you need to make sure you are using maven 3.2.5 or below. There is a bug in later versions of maven as linked to from https://issues.apache.org/jira/browse/MSHADE-206 that
+cause shaded dependencies to not be packaged correctly. Also please be aware that because we are shading dependencies mvn dependency:tree will not always show the dependencies correctly.
+
In order to build `storm` you need `python`, `ruby` and `nodejs`. In order to avoid an overful page we don't provide platform/OS specific installation instructions for those here. Please refer to you platform's/OS' documentation for support.
The `ruby` package manager `rvm` and `nodejs` package manager `nvm` are for convenience and are used in the tests which run on [travis](https://travis-ci.org/apache/storm). They can be installed using `curl -L https://get.rvm.io | bash -s stable --autolibs=enabled && source ~/.profile` (see the [rvm installation instructions](https://github.com/rvm/rvm) for details) and `wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.26.1/install.sh | bash && source ~/.bashrc` (see the [nvm installation instructions](https://github.com/creationix/nvm) for details).
@@ -270,16 +245,9 @@ in order to get started as fast as possible. Users can still install a specific
The following commands must be run from the top-level directory.
-The first step is to build/install the plugins and storm-core
-
-`mvn clean install -Pstorm-core`
+`mvn clean install`
-If you wish to skip the unit tests you can do this by adding `-DskipTests` to the command line. Once this
-completes successfully you may run
-
-`mvn clean install -Pstorm-more`
-
-to build and test all of the external libraries and examples. Again if you want to include `-DskipTests` you can.
+If you wish to skip the unit tests you can do this by adding `-DskipTests` to the command line.
In case you modified `storm.thrift`, you have to regenerate thrift code as java and python code before compiling whole project.
@@ -302,7 +270,6 @@ You can also run tests selectively with `-Dtest=<test_name>`. This works for bo
Unfortunately you might experience failures in clojure tests which are wrapped in the `maven-clojure-plugin` and thus doesn't provide too much useful output at first sight - you might end up with a maven test failure with an error message as unhelpful as `Clojure failed.`. In this case it's recommended to look into `target/test-reports` of the failed project to see what actual tests have failed or scroll through the maven output looking for obvious issues like missing binaries.
-
<a name="packaging"></a>
## Create a Storm distribution (packaging)
@@ -311,8 +278,7 @@ You can create a _distribution_ (like what you can download from Apache) as foll
do not use the Maven release plugin because creating an official release is the task of our release manager.
# First, build the code.
- $ mvn clean install -Pstorm-core # you may skip tests with `-DskipTests=true` to save time
- $ mvn clean install -Pstorm-more # you may skip tests with `-DskipTests=true` to save time
+ $ mvn clean install # you may skip tests with `-DskipTests=true` to save time
# Create the binary distribution.
$ cd storm-dist/binary && mvn package
http://git-wip-us.apache.org/repos/asf/storm/blob/25b6b074/dev-tools/travis/travis-install.sh
----------------------------------------------------------------------
diff --git a/dev-tools/travis/travis-install.sh b/dev-tools/travis/travis-install.sh
index 8005866..27f503d 100755
--- a/dev-tools/travis/travis-install.sh
+++ b/dev-tools/travis/travis-install.sh
@@ -26,21 +26,12 @@ TRAVIS_SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
cd ${STORM_SRC_ROOT_DIR}
-python ${TRAVIS_SCRIPT_DIR}/save-logs.py "install-storm-core.txt" mvn clean install -DskipTests -Pnative -Pstorm-core --batch-mode
+python ${TRAVIS_SCRIPT_DIR}/save-logs.py "install.txt" mvn clean install -DskipTests -Pnative --batch-mode
BUILD_RET_VAL=$?
if [[ "$BUILD_RET_VAL" != "0" ]];
then
- cat "install-storm-core.txt"
- exit ${BUILD_RET_VAL}
-fi
-
-python ${TRAVIS_SCRIPT_DIR}/save-logs.py "install-storm-more.txt" mvn clean install -DskipTests -Pstorm-more
-BUILD_RET_VAL=$?
-
-if [[ "$BUILD_RET_VAL" != "0" ]];
-then
- cat "install-storm-more.txt"
+ cat "install.txt"
fi
exit ${BUILD_RET_VAL}
http://git-wip-us.apache.org/repos/asf/storm/blob/25b6b074/dev-tools/travis/travis-script.sh
----------------------------------------------------------------------
diff --git a/dev-tools/travis/travis-script.sh b/dev-tools/travis/travis-script.sh
index 2230c68..dd1a337 100755
--- a/dev-tools/travis/travis-script.sh
+++ b/dev-tools/travis/travis-script.sh
@@ -15,11 +15,12 @@ PYTHON_VERSION_TO_FILE=`python -V > /tmp/python_version 2>&1`
PYTHON_VERSION=`cat /tmp/python_version`
RUBY_VERSION=`ruby -v`
NODEJS_VERSION=`node -v`
+MVN_VERSION=`mvn -v`
echo "Python version : $PYTHON_VERSION"
echo "Ruby version : $RUBY_VERSION"
echo "NodeJs version : $NODEJS_VERSION"
-
+echo "mvn version : $MVN_VERSION"
STORM_SRC_ROOT_DIR=$1
@@ -31,7 +32,7 @@ cd ${STORM_SRC_ROOT_DIR}
export STORM_TEST_TIMEOUT_MS=100000
# We now lean on Travis CI's implicit behavior, ```mvn clean install -DskipTests``` before running script
-mvn --batch-mode test -fae -Pnative -Pstorm-core && mvn --batch-mode test -fae -Pstorm-more
+mvn --batch-mode install -fae -Pnative
BUILD_RET_VAL=$?
for dir in `find . -type d -and -wholename \*/target/\*-reports`;
http://git-wip-us.apache.org/repos/asf/storm/blob/25b6b074/examples/storm-starter/pom.xml
----------------------------------------------------------------------
diff --git a/examples/storm-starter/pom.xml b/examples/storm-starter/pom.xml
index b24c436..c889694 100644
--- a/examples/storm-starter/pom.xml
+++ b/examples/storm-starter/pom.xml
@@ -142,7 +142,6 @@
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<createDependencyReducedPom>true</createDependencyReducedPom>
- <minimizeJar>true</minimizeJar>
</configuration>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/storm/blob/25b6b074/external/flux/flux-core/pom.xml
----------------------------------------------------------------------
diff --git a/external/flux/flux-core/pom.xml b/external/flux/flux-core/pom.xml
index 8d0c1f2..bd5a18f 100644
--- a/external/flux/flux-core/pom.xml
+++ b/external/flux/flux-core/pom.xml
@@ -72,7 +72,6 @@
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<createDependencyReducedPom>true</createDependencyReducedPom>
- <minimizeJar>true</minimizeJar>
</configuration>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/storm/blob/25b6b074/external/flux/flux-examples/pom.xml
----------------------------------------------------------------------
diff --git a/external/flux/flux-examples/pom.xml b/external/flux/flux-examples/pom.xml
index fe43663..537b4e0 100644
--- a/external/flux/flux-examples/pom.xml
+++ b/external/flux/flux-examples/pom.xml
@@ -117,7 +117,6 @@
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<createDependencyReducedPom>true</createDependencyReducedPom>
- <minimizeJar>true</minimizeJar>
</configuration>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/storm/blob/25b6b074/external/storm-eventhubs/pom.xml
----------------------------------------------------------------------
diff --git a/external/storm-eventhubs/pom.xml b/external/storm-eventhubs/pom.xml
index 0338e1e..b227f24 100755
--- a/external/storm-eventhubs/pom.xml
+++ b/external/storm-eventhubs/pom.xml
@@ -49,7 +49,6 @@
</execution>
</executions>
<configuration>
- <minimizeJar>true</minimizeJar>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer">
</transformer>
http://git-wip-us.apache.org/repos/asf/storm/blob/25b6b074/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index edb7f2b..7f525bb 100644
--- a/pom.xml
+++ b/pom.xml
@@ -215,41 +215,29 @@
<junit.version>4.11</junit.version>
</properties>
+ <modules>
+ <module>storm-multilang/javascript</module>
+ <module>storm-multilang/python</module>
+ <module>storm-multilang/ruby</module>
+ <module>storm-buildtools/maven-shade-clojure-transformer</module>
+ <module>storm-buildtools/storm-maven-plugins</module>
+ <module>storm-core</module>
+ <module>external/storm-kafka</module>
+ <module>external/storm-hdfs</module>
+ <module>external/storm-hbase</module>
+ <module>external/storm-hive</module>
+ <module>external/storm-jdbc</module>
+ <module>external/storm-redis</module>
+ <module>external/storm-eventhubs</module>
+ <module>external/flux</module>
+ <module>external/storm-elasticsearch</module>
+ <module>external/storm-solr</module>
+ <module>examples/storm-starter</module>
+ </modules>
+
+
<profiles>
<profile>
- <id>storm-core</id>
- <activation>
- <activeByDefault>true</activeByDefault>
- </activation>
- <modules>
- <module>storm-multilang/javascript</module>
- <module>storm-multilang/python</module>
- <module>storm-multilang/ruby</module>
- <module>storm-buildtools/maven-shade-clojure-transformer</module>
- <module>storm-buildtools/storm-maven-plugins</module>
- <module>storm-core</module>
- </modules>
- </profile>
- <profile>
- <id>storm-more</id>
- <activation>
- <activeByDefault>true</activeByDefault>
- </activation>
- <modules>
- <module>external/storm-kafka</module>
- <module>external/storm-hdfs</module>
- <module>external/storm-hbase</module>
- <module>external/storm-hive</module>
- <module>external/storm-jdbc</module>
- <module>external/storm-redis</module>
- <module>external/storm-eventhubs</module>
- <module>external/flux</module>
- <module>external/storm-elasticsearch</module>
- <module>external/storm-solr</module>
- <module>examples/storm-starter</module>
- </modules>
- </profile>
- <profile>
<id>sign</id>
<build>
<plugins>
[3/8] storm git commit: Adressed review comments
Posted by bo...@apache.org.
Adressed review comments
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/feba08a0
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/feba08a0
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/feba08a0
Branch: refs/heads/master
Commit: feba08a0488a48cfe030ebb2eb8e915f5501bc8f
Parents: 12fdaa0
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Wed Sep 16 10:58:29 2015 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Fri Oct 2 14:18:58 2015 -0500
----------------------------------------------------------------------
dev-tools/travis/travis-install.sh | 2 --
dev-tools/travis/travis-script.sh | 2 +-
storm-core/pom.xml | 8 +++++++-
3 files changed, 8 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/feba08a0/dev-tools/travis/travis-install.sh
----------------------------------------------------------------------
diff --git a/dev-tools/travis/travis-install.sh b/dev-tools/travis/travis-install.sh
index 4857a1e..8005866 100755
--- a/dev-tools/travis/travis-install.sh
+++ b/dev-tools/travis/travis-install.sh
@@ -35,8 +35,6 @@ then
exit ${BUILD_RET_VAL}
fi
-exit ${BUILD_RET_VAL}
-
python ${TRAVIS_SCRIPT_DIR}/save-logs.py "install-storm-more.txt" mvn clean install -DskipTests -Pstorm-more
BUILD_RET_VAL=$?
http://git-wip-us.apache.org/repos/asf/storm/blob/feba08a0/dev-tools/travis/travis-script.sh
----------------------------------------------------------------------
diff --git a/dev-tools/travis/travis-script.sh b/dev-tools/travis/travis-script.sh
index c59cb1a..2230c68 100755
--- a/dev-tools/travis/travis-script.sh
+++ b/dev-tools/travis/travis-script.sh
@@ -31,7 +31,7 @@ cd ${STORM_SRC_ROOT_DIR}
export STORM_TEST_TIMEOUT_MS=100000
# We now lean on Travis CI's implicit behavior, ```mvn clean install -DskipTests``` before running script
-mvn --batch-mode test -fae -Pnative -Pstorm-core && mvn test -fae -Pnative -Pstorm-more
+mvn --batch-mode test -fae -Pnative -Pstorm-core && mvn --batch-mode test -fae -Pstorm-more
BUILD_RET_VAL=$?
for dir in `find . -type d -and -wholename \*/target/\*-reports`;
http://git-wip-us.apache.org/repos/asf/storm/blob/feba08a0/storm-core/pom.xml
----------------------------------------------------------------------
diff --git a/storm-core/pom.xml b/storm-core/pom.xml
index ca40cb8..8d0608e 100644
--- a/storm-core/pom.xml
+++ b/storm-core/pom.xml
@@ -495,7 +495,7 @@
</relocation>
<relocation>
<pattern>org.apache.thrift</pattern>
- <!-- This pattern is inconsistent for backwards compatability purposes. -->
+ <!-- This pattern is inconsistent for backwards compatibility purposes. -->
<shadedPattern>org.apache.thrift7</shadedPattern>
</relocation>
<relocation>
@@ -607,6 +607,12 @@
<transformer implementation="org.apache.storm.maven.shade.clojure.ClojureTransformer" />
</transformers>
<filters>
+ <!-- Several of these filters remove the .clj files from the shaded dependencies, even though only .clj files are in these jars.
+ The reason for this is a bit complex, but intentional. Durring the build process all of the dependency .clj files are
+ compiled down into .class files, and included in storm-core.jar. The regular shade thransformer handles these in
+ the majority of cases correctly. However, the ClojureTransformer does not shade everything correctly all the
+ time. Instead of spending a lot of time to get the ClojureTransformer to parse clojrue correctly we opted to remove
+ the .clj files from the uber jar. -->
<filter><artifact>org.clojure:core.incubator</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
<filter><artifact>cheshire:cheshire</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
<filter><artifact>org.clojure:tools.logging</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
[2/8] storm git commit: Fixed some spelling issues
Posted by bo...@apache.org.
Fixed some spelling issues
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/045b77dc
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/045b77dc
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/045b77dc
Branch: refs/heads/master
Commit: 045b77dcae6d40e12fbdfaeeb2b04df38f73154e
Parents: feba08a
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Thu Sep 17 14:03:20 2015 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Fri Oct 2 14:18:58 2015 -0500
----------------------------------------------------------------------
storm-core/pom.xml | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/045b77dc/storm-core/pom.xml
----------------------------------------------------------------------
diff --git a/storm-core/pom.xml b/storm-core/pom.xml
index 8d0608e..7dc5c03 100644
--- a/storm-core/pom.xml
+++ b/storm-core/pom.xml
@@ -608,10 +608,10 @@
</transformers>
<filters>
<!-- Several of these filters remove the .clj files from the shaded dependencies, even though only .clj files are in these jars.
- The reason for this is a bit complex, but intentional. Durring the build process all of the dependency .clj files are
- compiled down into .class files, and included in storm-core.jar. The regular shade thransformer handles these in
- the majority of cases correctly. However, the ClojureTransformer does not shade everything correctly all the
- time. Instead of spending a lot of time to get the ClojureTransformer to parse clojrue correctly we opted to remove
+ The reason for this is a bit complex, but intentional. During the build process all of the dependency .clj files are
+ compiled down into .class files, and included in storm-core.jar. The regular shade transformer handles these in
+ the majority of cases correctly. However, the Clojure-Transformer does not shade everything correctly all the
+ time. Instead of spending a lot of time to get the Clojure-Transformer to parse Clojure correctly we opted to remove
the .clj files from the uber jar. -->
<filter><artifact>org.clojure:core.incubator</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
<filter><artifact>cheshire:cheshire</artifact><excludes><exclude>**/*.clj</exclude></excludes></filter>
[6/8] storm git commit: Updated build a bit
Posted by bo...@apache.org.
Updated build a bit
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/f6e10a87
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/f6e10a87
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/f6e10a87
Branch: refs/heads/master
Commit: f6e10a879c615121468228761a065ce320ee3cd8
Parents: 25b6b07
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Fri Oct 2 14:16:37 2015 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Fri Oct 2 14:19:59 2015 -0500
----------------------------------------------------------------------
dev-tools/travis/travis-install.sh | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/f6e10a87/dev-tools/travis/travis-install.sh
----------------------------------------------------------------------
diff --git a/dev-tools/travis/travis-install.sh b/dev-tools/travis/travis-install.sh
index 27f503d..8f6ce21 100755
--- a/dev-tools/travis/travis-install.sh
+++ b/dev-tools/travis/travis-install.sh
@@ -15,10 +15,12 @@ PYTHON_VERSION_TO_FILE=`python -V > /tmp/python_version 2>&1`
PYTHON_VERSION=`cat /tmp/python_version`
RUBY_VERSION=`ruby -v`
NODEJS_VERSION=`node -v`
+MVN_VERSION=`mvn -v`
echo "Python version : $PYTHON_VERSION"
echo "Ruby version : $RUBY_VERSION"
echo "NodeJs version : $NODEJS_VERSION"
+echo "mvn version : $MVN_VERSION"
STORM_SRC_ROOT_DIR=$1
[8/8] storm git commit: Updated Changelog
Posted by bo...@apache.org.
Updated Changelog
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/c90f6fd4
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/c90f6fd4
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/c90f6fd4
Branch: refs/heads/master
Commit: c90f6fd4d8c9fc823ff22b4355a8f21f155e17fb
Parents: 0cdd7e8
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Fri Oct 2 15:18:16 2015 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Fri Oct 2 15:18:16 2015 -0500
----------------------------------------------------------------------
CHANGELOG.md | 4 ++++
1 file changed, 4 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/c90f6fd4/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2071434..4a8b1f5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -67,6 +67,10 @@
* STORM-949: On the topology summary UI page, last shown error should have the time and date
## 0.10.0-beta2
+ * STORM-1012: Shaded everything that was not already shaded
+ * STORM-967: Shaded everything that was not already shaded
+ * STORM-922: Shaded everything that was not already shaded
+ * STORM-1042: Shaded everything that was not already shaded
* STORM-1026: Adding external classpath elements does not work
* STORM-1055: storm-jdbc README needs fixes and context
* STORM-1044: Setting dop to zero does not raise an error
[4/8] storm git commit: Updated the docs to make the clearer.
Posted by bo...@apache.org.
Updated the docs to make the clearer.
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/f04325b0
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/f04325b0
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/f04325b0
Branch: refs/heads/master
Commit: f04325b0095f2112c9fc320f03b4806733ce987a
Parents: 045b77d
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Wed Sep 30 12:41:26 2015 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Fri Oct 2 14:18:58 2015 -0500
----------------------------------------------------------------------
DEVELOPER.md | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/f04325b0/DEVELOPER.md
----------------------------------------------------------------------
diff --git a/DEVELOPER.md b/DEVELOPER.md
index 7376949..541afb2 100644
--- a/DEVELOPER.md
+++ b/DEVELOPER.md
@@ -222,6 +222,35 @@ To pull in a merge request you should generally follow the command line instruct
# Build the code and run the tests
+Maven has some issues with maven pugins, shading and multi-module projects which force Storm's build process to be somewhat non-standard.
+
+Essentially what happens is that if you build a multi-module project where one module depends on another module that is a part of the same build, maven tries to be smart and will use the pom.xml and .jar or .class directory from the module that is a dependency.
+
+So if we have a project set up like the following where we have flux depend on storm-core, and storm-core is shaded. flux will see different versions of the code and pom.xml from storm-core depending on what maven command is run.
+
+- storm.git/
+ - pom.xml
+ - storm-core/
+ - pom.xml
+ - dependency-reduce-pom.xml
+ - target/
+ - classes/
+ - storm-core.jar (shaded)
+ - original.storm-core.jar (no-shaded)
+ - flux/
+ - pom.xml (depends on storm-core)
+ - target/
+ - classes/
+ - flux.uber.jar
+
+If I run `mvn clean compile` the classpath for compiling flux will include `storm.git/storm-core/target/classes` which are not shaded, and if flux is written to use storm-core shaded class it will fail to compile. flux will also see the full set of non-shaded dependencies from storm-core, so if flux is creating an uber jar, it will assume that all of those dependencies will be provided by storm-core, when in reality they will not be.
+
+If I run `mvn clean package` or `mvn clean install` flux will see the shaded storm-core.jar on its classpath, but it will still see the full non-shaded set of storm-core's dependencies.
+
+The only way I found to make flux use the dependency-reduced-pom.xml from storm-core, is to first build and install storm-core by itself, and then build and install flux without storm-core. That way storm-core will install the shaded jar and dependency-reduced-pom.xml into the local repository. Then flux will get its dependencies from the local repo instead of trying to rebuild storm-core.
+
+Because of this we have split the build into two profiles storm-core and storm-more. By default both build together which works OK for simple testing, but when doing a release it is important to build them in two phases or flux, storm-starter, and possibly others will not be packaged correctly.
+
## Prerequisites
In order to build `storm` you need `python`, `ruby` and `nodejs`. In order to avoid an overful page we don't provide platform/OS specific installation instructions for those here. Please refer to you platform's/OS' documentation for support.
[7/8] storm git commit: Merge branch 'STORM-1042' of
https://github.com/revans2/incubator-storm into STORM-1042
Posted by bo...@apache.org.
Merge branch 'STORM-1042' of https://github.com/revans2/incubator-storm into STORM-1042
STORM-1012:
STORM-967:
STORM-922:
STORM-1042: Shaded everything that was not already shaded
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/0cdd7e83
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/0cdd7e83
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/0cdd7e83
Branch: refs/heads/master
Commit: 0cdd7e83c893e2e550256e6daeea36c5a7379002
Parents: ce93d5f f6e10a8
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Fri Oct 2 15:16:53 2015 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Fri Oct 2 15:16:53 2015 -0500
----------------------------------------------------------------------
DEVELOPER.md | 33 +--
dev-tools/travis/travis-install.sh | 4 +-
dev-tools/travis/travis-script.sh | 5 +-
examples/storm-starter/pom.xml | 1 -
external/flux/flux-core/pom.xml | 1 -
external/flux/flux-examples/pom.xml | 1 -
external/storm-elasticsearch/pom.xml | 5 +
external/storm-eventhubs/pom.xml | 1 -
external/storm-kafka/pom.xml | 5 +
external/storm-solr/pom.xml | 4 +-
pom.xml | 123 +++++++---
storm-core/pom.xml | 259 ++++++++++++--------
storm-core/src/clj/backtype/storm/log.clj | 2 +-
storm-dist/binary/src/main/assembly/binary.xml | 32 ---
14 files changed, 272 insertions(+), 204 deletions(-)
----------------------------------------------------------------------