You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "P. Taylor Goetz" <pt...@apache.org> on 2017/02/24 20:40:05 UTC
[VOTE] Release Apache Storm 1.1.0 (RC2)
This is a call to vote on releasing Apache Storm 1.1.0 (rc2)
Full list of changes in this release:
https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=17a2017fb644e353fb2a0f5bf50d400ee28036ba
The tag/commit to be voted upon is v1.1.0:
https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob;f=CHANGELOG.md;h=0352f6d27dad1c88c24bb99fd894a211ea6c1e71;hb=7c8ea09e09160d36bba7e363ed53781816b46597
The source archive being voted upon can be found here:
https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc2/apache-storm-1.1.0-src.tar.gz
Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc2/
The release artifacts are signed with the following key:
https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
The Nexus staging repository for this release is:
https://repository.apache.org/content/repositories/orgapachestorm-1046
Please vote on releasing this package as Apache Storm 1.1.0.
When voting, please list the actions taken to verify the release.
This vote will be open for at least 72 hours.
[ ] +1 Release this package as Apache Storm 1.1.0
[ ] 0 No opinion
[ ] -1 Do not release this package because...
Thanks to everyone who contributed to this release.
-Taylor
Re: [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Jungtaek Lim <ka...@gmail.com>.
+1
> source
- verify file (signature, MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK
- extract file
-- source, tar.gz : OK
-- source, zip : OK
- diff-ing extracted files between tar.gz and zip : OK
- build source with JDK 7
-- source, tar.gz : OK
> binary
- verify file (signature, MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK
- extract file
-- binary, tar.gz : OK
-- binary, zip : OK
- diff-ing extracted files between tar.gz and zip : OK
- launch daemons : OK
- run RollingTopWords (local) : OK
- run RollingTopWords (remote) : OK
I've done other manual tests from RC1 so looks OK to me.
Thanks,
Jungtaek Lim (HeartSaVioR)
2017년 2월 25일 (토) 오전 5:40, P. Taylor Goetz <pt...@apache.org>님이 작성:
> This is a call to vote on releasing Apache Storm 1.1.0 (rc2)
>
> Full list of changes in this release:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=17a2017fb644e353fb2a0f5bf50d400ee28036ba
>
> The tag/commit to be voted upon is v1.1.0:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob;f=CHANGELOG.md;h=0352f6d27dad1c88c24bb99fd894a211ea6c1e71;hb=7c8ea09e09160d36bba7e363ed53781816b46597
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc2/apache-storm-1.1.0-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc2/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1046
>
> Please vote on releasing this package as Apache Storm 1.1.0.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.1.0
> [ ] 0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Jungtaek Lim <ka...@gmail.com>.
+1 from me, too.
On Sat, 11 Mar 2017 at 5:05 AM P. Taylor Goetz <pt...@gmail.com> wrote:
> Yes, that's the plan. All examples would be source only with poms that
> allow the user to compile.
>
> -Taylor
>
> > On Mar 10, 2017, at 10:28 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>
> wrote:
> >
> > +1 for that, although I think it is the examples that ends up being the
> heavy part. The examples are all uber jars and pull in lots of
> dependencies. Some of them multiple times over. Do we want the examples
> to only be a source release so before running an example they have to build
> it?
> >
> >
> > - Bobby
> >
> > On Thursday, March 9, 2017, 2:22:42 PM CST, P. Taylor Goetz <
> ptgoetz@gmail.com> wrote:I’m fine with dropping connectors from the
> binary distribution in favor having users use Maven, gradle, etc. handle
> user dependencies. Building topology jars manually (vs. letting Maven do
> the heavy lifting) is painful and something to be avoided, specifically for
> new users.
> >
> > So the “binary release” for connectors, etc. would essentially become
> just a README that documents how to use them, and what the Maven
> coordinates are for each component.
> >
> > So with the connector jars just hosted in the Maven repo, and all
> examples source-only (to be built by users) the size of the binary
> distribution should be reduced dramatically.
> >
> > Any other opinions?
> >
> > -Taylor
> >
> >
> >> On Mar 8, 2017, at 4:26 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> >>
> >> Regarding 3, if we don't make uber jar for connectors, adding jar file
> in
> >> binary dist. to the classpath will not work, which means that including
> >> connector jar libraries in binary dist. is no effect.
> >> (Someone proposed shading storm-redis because of dependencies... We can
> >> cover it by various way like adding to pom, or using '--artifact' so no
> >> issue at all for now.)
> >>
> >> So if we want to go with 3, I'd also like to propose excluding connector
> >> libraries in binary dist. at all.
> >>
> >> Others look great to me.
> >>
> >> - Jungtaek Lim (HeartSaVioR)
> >>
> >> 2017년 3월 9일 (목) 오전 6:12, P. Taylor Goetz <pt...@gmail.com>님이 작성:
> >>
> >>> I’d like to propose the following to address the binary distribution
> size:
> >>>
> >>> 1. Finish moving connector examples to the “examples” directory.
> >>> 2. Make all examples source-only in the binary distribution. This will
> >>> mean users will have to build the examples themselves, but that’s
> typically
> >>> what one does when looking at example code.
> >>> 3. Stop unnecessarily shading connector libraries (i.e. let the user’s
> pom
> >>> pull in the transitive dependencies when building a topology jar).
> >>>
> >>> If we agree on this approach, I’ll take up the necessary changes for
> the
> >>> 1.1.0 release.
> >>>
> >>> -Taylor
> >>>
> >>>
> >>>
> >>>> On Mar 8, 2017, at 9:16 AM, Xin Wang <da...@gmail.com> wrote:
> >>>>
> >>>> Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release.
> >>>>
> >>>> - Xin
> >>>>
> >>>> 2017-03-07 9:16 GMT+08:00 Jungtaek Lim <ka...@gmail.com>:
> >>>>
> >>>>> Sorry Roshan but I don't get it.
> >>>>> We're already having separate modules for each of the connectors, and
> >>> even
> >>>>> examples. We also don't have any common modules for external
> >>> connectors, so
> >>>>> those are completely individual.
> >>>>>
> >>>>> As I reported earlier, binary dist. will go up more than 500 MB when
> we
> >>>>> address STORM-2390(STORM-2249 actually), which the community seemed
> not
> >>>>> want to.
> >>>>>
> >>>>> Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd
> rather
> >>>>> unlink from epic to get out of release, and discuss more.
> >>>>> Any objections?
> >>>>>
> >>>>> - Jungtaek Lim (HeartSaVioR)
> >>>>>
> >>>>> 2017년 3월 4일 (토) 오전 10:02, Roshan Naik <ro...@hortonworks.com>님이 작성:
> >>>>>
> >>>>>> As having a separate module for each of the connectors will cause a
> lot
> >>>>> of
> >>>>>> bloat … may be good to club them into one.
> >>>>>>
> >>>>>> -roshan
> >>>>>>
> >>>>>>
> >>>>>> On 3/2/17, 10:39 AM, "Sree V" <sr...@yahoo.com.INVALID>
> wrote:
> >>>>>>
> >>>>>> +1separate main binaries, connectors and samples/examples.
> >>>>>>
> >>>>>> Thanking you.
> >>>>>> With Regards
> >>>>>> Sree
> >>>>>>
> >>>>>> On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
> >>>>>> satish.duggana@gmail.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>> Agree that such huge binaries may not be acceptable. We should
> >>>>> really
> >>>>>> think
> >>>>>> about the options proposed earlier like excluding some of the
> >>>>> external
> >>>>>> connectors from binary and update the documentation respectively.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Satish.
> >>>>>>
> >>>>>> On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim <ka...@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Adding my observation to my last mail:
> >>>>>>> I just give STORM-2249 a try (against master branch) and compare
> >>>>>> before vs
> >>>>>>> after.
> >>>>>>> It adds more than 300 MB, and the final archive is more than 550
> >>>>> MB.
> >>>>>> I
> >>>>>>> guess it would be similar for 1.x branch.
> >>>>>>>
> >>>>>>> Before ----------
> >>>>>>> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:41
> >>>>>> apache-storm-2.0.0-SNAPSHOT.
> >>>>>>> pom
> >>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
> >>>>>> apache-storm-2.0.0-SNAPSHOT.
> >>>>>>> pom.asc
> >>>>>>> -rw-r--r-- 1 jlim staff 264M 3 2 19:41
> >>>>>> apache-storm-2.0.0-SNAPSHOT.
> >>>>>>> tar.gz
> >>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
> >>>>>> apache-storm-2.0.0-SNAPSHOT.
> >>>>>>> tar.gz.asc
> >>>>>>> -rw-r--r-- 1 jlim staff 264M 3 2 19:41
> >>>>>> apache-storm-2.0.0-SNAPSHOT.
> >>>>>>> zip
> >>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
> >>>>>> apache-storm-2.0.0-SNAPSHOT.
> >>>>>>> zip.asc
> >>>>>>>
> >>>>>>> After -----------
> >>>>>>> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:46
> >>>>>> apache-storm-2.0.0-SNAPSHOT.
> >>>>>>> pom
> >>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:46
> >>>>>> apache-storm-2.0.0-SNAPSHOT.
> >>>>>>> pom.asc
> >>>>>>> -rw-r--r-- 1 jlim staff 564M 3 2 19:46
> >>>>>> apache-storm-2.0.0-SNAPSHOT.
> >>>>>>> tar.gz
> >>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:46
> >>>>>> apache-storm-2.0.0-SNAPSHOT.
> >>>>>>> tar.gz.asc
> >>>>>>> -rw-r--r-- 1 jlim staff 565M 3 2 19:46
> >>>>>> apache-storm-2.0.0-SNAPSHOT.
> >>>>>>> zip
> >>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:47
> >>>>>> apache-storm-2.0.0-SNAPSHOT.
> >>>>>>> zip.asc
> >>>>>>>
> >>>>>>> While 264M is already a bit huge for me, 564M is not that I can
> >>>>>> accept.
> >>>>>>> (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka
> >>>>>> 0.10.2 is
> >>>>>>> 37M.)
> >>>>>>>
> >>>>>>> Btw, we're including source code of examples, and "mvn clean
> >>>>>> package" will
> >>>>>>> work for every example modules.
> >>>>>>>
> >>>>>>> 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >>>>>>>
> >>>>>>>> I guess it might be good time to think why we add all connectors
> >>>>>> to the
> >>>>>>>> binary distribution.
> >>>>>>>>
> >>>>>>>> Spark and Flink don't include them to binary dist. They even
> >>>>> moved
> >>>>>> some
> >>>>>>> or
> >>>>>>>> most of connectors out of repo, have been maintaining them in
> >>>>>> Apache
> >>>>>>> Bahir.
> >>>>>>>> (Personally this is something I'm in favor of. We have lots of
> >>>>>> connectors
> >>>>>>>> and many of them are outdated - clear example is
> >>>>>> storm-elasticsearch.)
> >>>>>>>>
> >>>>>>>> If we are assuming online then we don't even need to think about
> >>>>>> users
> >>>>>>>> touching binary dist. version of connectors. Users have been
> >>>>>> including
> >>>>>>> them
> >>>>>>>> via build tools' dependency management, or even starting 1.1.0,
> >>>>>> users can
> >>>>>>>> include them via '--artifact' option.
> >>>>>>>>
> >>>>>>>> I also was just one of users for Storm, and I haven't use them
> >>>>>> directly.
> >>>>>>>> How much UX gets worse when we remove connectors to binary dist?
> >>>>>> It only
> >>>>>>>> helps some users who are not connected to the internet, and IMHO
> >>>>>> it's a
> >>>>>>>> rare case.
> >>>>>>>>
> >>>>>>>> I would like to see the opposite approach, removing all
> >>>>> connectors
> >>>>>> (or
> >>>>>>>> just keeping storm-kafka/storm-kafka-client and some more
> >>>>> preferred
> >>>>>>> things)
> >>>>>>>> and its relevant examples from binary dist.
> >>>>>>>>
> >>>>>>>> What do you think about it?
> >>>>>>>>
> >>>>>>>> - Jungtaek Lim (HeartSaVioR)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> 2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ro...@hortonworks.com>님이
> >>>>>> 작성:
> >>>>>>>>
> >>>>>>>> Once all of the shaded examples are included the size will go up
> >>>>>> further.
> >>>>>>>>
> >>>>>>>> But currently as they are not part of the tar.gz … something else
> >>>>>> is the
> >>>>>>>> culprit for the bloat.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases
> >>>>>> .. of
> >>>>>>>> files that are larger than 4MB.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> @Jungtaek Lim<ma...@hortonworks.com> : I am thinking,
> >>>>>> since the
> >>>>>>>> code for the examples can be easily viewed online … it would be
> >>>>>> valuable
> >>>>>>> to
> >>>>>>>> have the executable topologies made available to the user as part
> >>>>>> of the
> >>>>>>>> binary release … rather have them figure out how to build it
> >>>>>> correctly
> >>>>>>>> before trying them out.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -roshan
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh
> >>>>>> {} \;
> >>>>>>>>
> >>>>>>>> -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33
> >>>>>>> ./examples/storm-starter/
> >>>>>>>> storm-starter-topologies-1.0.3.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30
> >>>>>>>> ./external/flux/flux-examples-1.0.3.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32
> >>>>>>>> ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32
> >>>>>>>> ./external/sql/storm-sql-core/guava-16.0.1.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30
> >>>>>>>> ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-
> >>>>>>> with-dependencies.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33
> >>>>>>>>
> >>>>>>
> ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33
> >>>>>>>> ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
> >>>>>>>>
> >>>>>>>> -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02
> >>>>>> ./lib/clojure-1.7.0.jar
> >>>>>>>>
> >>>>>>>> -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26
> >>>>>>> ./lib/storm-core-1.0.3.jar
> >>>>>>>>
> >>>>>>>> -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26
> >>>>>>>> ./lib/storm-rename-hack-1.0.3.jar
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh
> >>>>>> {} \;
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23
> >>>>>> ./examples/storm-pmml-
> >>>>>>>> examples/storm-pmml-examples-1.1.0.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20
> >>>>>>> ./examples/storm-starter/
> >>>>>>>> storm-starter-topologies-1.1.0.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11
> >>>>>>>> ./external/flux/flux-examples-1.1.0.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16
> >>>>>>>> ./external/sql/storm-sql-core/calcite-core-1.11.0.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16
> >>>>>>>> ./external/sql/storm-sql-core/guava-16.0.1.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12
> >>>>>>> ./external/sql/storm-sql-
> >>>>>>>> runtime/calcite-core-1.11.0.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12
> >>>>>>> ./external/sql/storm-sql-
> >>>>>>>> runtime/guava-16.0.1.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18
> >>>>>>>> ./external/storm-druid/storm-druid-1.1.0.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11
> >>>>>>>> ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-
> >>>>>>> with-dependencies.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20
> >>>>>>>>
> >>>>>>
> ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18
> >>>>>>>> ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02
> >>>>>> ./lib/clojure-1.7.0.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07
> >>>>>>> ./lib/storm-core-1.1.0.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07
> >>>>>>>> ./lib/storm-rename-hack-1.1.0.jar
> >>>>>>>>
> >>>>>>>> -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19
> >>>>>>>> ./toollib/storm-kafka-monitor-1.1.0.jar
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> About STORM-2249, since examples are shading their
> >>>>> dependencies,
> >>>>>>> binary
> >>>>>>>>
> >>>>>>>> dist will grow much bigger. I've left some comments regarding
> >>>>>> that.
> >>>>>>>>
> >>>>>>>> Btw, I have another view of this. Showing example codes is
> >>>>> more
> >>>>>>>> important
> >>>>>>>>
> >>>>>>>> than just let users execute some topologies. That's what
> >>>>> example
> >>>>>>>> modules
> >>>>>>>>
> >>>>>>>> are for. We need to include source as well. If we need to pick
> >>>>>> one,
> >>>>>>>> source
> >>>>>>>>
> >>>>>>>> code would be better.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> STORM-2343 seems better to add to 1.1.0. I just am not enough
> >>>>>>> familiar
> >>>>>>>> with
> >>>>>>>>
> >>>>>>>> storm-kafka-client so not sure I can review that, but I'll try
> >>>>>> to. I
> >>>>>>>> feel
> >>>>>>>>
> >>>>>>>> it's not that make release dragged. Let's add to 1.1.0 epic.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Let's make minimum merge before another RC vote. Addressing
> >>>>>>> STORM-2389
> >>>>>>>> (and
> >>>>>>>>
> >>>>>>>> maybe STORM-2343) is enough for me. Others are not that
> >>>>>> critical.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Jungtaek Lim (HeartSaVioR)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <
> >>>>>>>> hlouro@hortonworks.com>
> >>>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Roshan, does this PR<
> >>>>>> https://github.com/apache/storm/pull/1831>
> >>>>>>> and
> >>>>>>>> JIRA<
> >>>>>>>>
> >>>>>>>>> https://issues.apache.org/jira/browse/STORM-2249> address
> >>>>> the
> >>>>>>>> missing
> >>>>>>>>
> >>>>>>>>> jars problem that you mentioned. I had created it in
> >>>>> December
> >>>>>> 2016,
> >>>>>>>> but
> >>>>>>>>
> >>>>>>>>> there is an ongoing discussion if we should indeed put the
> >>>>>> jars in
> >>>>>>>> the
> >>>>>>>>
> >>>>>>>>> examples location or not.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> On a different note, this storm-kafka-client/KafkaSpout PR<
> >>>>>>>>
> >>>>>>>>> https://github.com/apache/storm/pull/1924> fixes a bug with
> >>>>>> the
> >>>>>>>> number of
> >>>>>>>>
> >>>>>>>>> uncommitted offsets that is quite important. It is not a
> >>>>>> blocker,
> >>>>>>>> but it is
> >>>>>>>>
> >>>>>>>>> quite critical. I am going to do one last pass reviewing
> >>>>>> today. It
> >>>>>>>> would be
> >>>>>>>>
> >>>>>>>>> good if we could have this PR included with the release. Can
> >>>>>> anyone
> >>>>>>>> else
> >>>>>>>>
> >>>>>>>>> review it as well ?
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>>> Hugo
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <
> >>>>>> ptgoetz@gmail.com
> >>>>>>>> <mailto:ptgo
> >>>>>>>>
> >>>>>>>>> etz@gmail.com>> wrote:
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> Yeah, I don’t think the file size is a killer/blocker. It’s
> >>>>>> largely
> >>>>>>>> due to
> >>>>>>>>
> >>>>>>>>> shaded examples, etc. But it’s something to keep an eye on.
> >>>>>> Our
> >>>>>>>> binary
> >>>>>>>>
> >>>>>>>>> releases shouldn’t have to be that big.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> -Taylor
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Mar 1, 2017, at 12:09 PM, Roshan Naik <
> >>>>>> roshan@hortonworks.com<
> >>>>>>>> mailto:
> >>>>>>>>
> >>>>>>>>> roshan@hortonworks.com>> wrote:
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> Have filed Jiras so for the 3 issues mentioned. Not sure if
> >>>>>> we need
> >>>>>>>> a JIRA
> >>>>>>>>
> >>>>>>>>> for the file size getting bloated by that much.
> >>>>>>>>
> >>>>>>>>> Somebody better familiar with the matter may want to take
> >>>>>> about
> >>>>>>> that?
> >>>>>>>>
> >>>>>>>>> -roshan
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com
> >>>>>> <mailto:
> >>>>>>> ptgo
> >>>>>>>>
> >>>>>>>>> etz@gmail.com>> wrote:
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> Thanks for bringing these up Roshan. Feel free to file JIRA
> >>>>>>>> tickets for
> >>>>>>>>
> >>>>>>>>> these issues and assign the “Release Apache Storm 1.1.0”
> >>>>> epic
> >>>>>> so
> >>>>>>>> they can
> >>>>>>>>
> >>>>>>>>> be tracked for this release.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> -Taylor
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Mar 1, 2017, at 9:27 AM, Roshan Naik <
> >>>>>> roshan@hortonworks.com<
> >>>>>>>> mailto:
> >>>>>>>>
> >>>>>>>>> roshan@hortonworks.com>> wrote:
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> Found these additional issues:
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> 1- BUG: Even if topology.eventlogger.executors=0, the
> >>>>>>> event_logger
> >>>>>>>> bolt
> >>>>>>>>
> >>>>>>>>> is instantiated … previously observed to cause ~10%
> >>>>>> degradation in
> >>>>>>>> perf
> >>>>>>>>
> >>>>>>>>> even with logging disabled.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> 2- Missing Jars: The storm-*-examples jars are missing in
> >>>>> the
> >>>>>>> binary
> >>>>>>>>
> >>>>>>>>> distro (other than a storm-pmml-examples.jar,
> >>>>>>> storm-jms-examples.jar
> >>>>>>>> &
> >>>>>>>>
> >>>>>>>>> flux-examples.jar).
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> 3- Minor: HdfsSpoutTopology example has not been moved into
> >>>>>>>>
> >>>>>>>>> storm-hdfs-examples from storm-starter
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> Another side observation … v1.0.3 tar.gz downloadable was
> >>>>>> 190MB.
> >>>>>>>> This
> >>>>>>>>
> >>>>>>>>> v1.1.0 tar.gz downloadable is 297MB !! …. even though some
> >>>>>> of the
> >>>>>>>> example
> >>>>>>>>
> >>>>>>>>> topologies didn’t make it.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>> -roshan
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>> Name : Jungtaek Lim
> >>>>>>>>
> >>>>>>>> Blog : http://medium.com/@heartsavior
> >>>>>>>>
> >>>>>>>> Twitter : http://twitter.com/heartsavior
> >>>>>>>>
> >>>>>>>> LinkedIn : http://www.linkedin.com/in/heartsavior
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
>
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Yes, that's the plan. All examples would be source only with poms that allow the user to compile.
-Taylor
> On Mar 10, 2017, at 10:28 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
>
> +1 for that, although I think it is the examples that ends up being the heavy part. The examples are all uber jars and pull in lots of dependencies. Some of them multiple times over. Do we want the examples to only be a source release so before running an example they have to build it?
>
>
> - Bobby
>
> On Thursday, March 9, 2017, 2:22:42 PM CST, P. Taylor Goetz <pt...@gmail.com> wrote:I’m fine with dropping connectors from the binary distribution in favor having users use Maven, gradle, etc. handle user dependencies. Building topology jars manually (vs. letting Maven do the heavy lifting) is painful and something to be avoided, specifically for new users.
>
> So the “binary release” for connectors, etc. would essentially become just a README that documents how to use them, and what the Maven coordinates are for each component.
>
> So with the connector jars just hosted in the Maven repo, and all examples source-only (to be built by users) the size of the binary distribution should be reduced dramatically.
>
> Any other opinions?
>
> -Taylor
>
>
>> On Mar 8, 2017, at 4:26 PM, Jungtaek Lim <ka...@gmail.com> wrote:
>>
>> Regarding 3, if we don't make uber jar for connectors, adding jar file in
>> binary dist. to the classpath will not work, which means that including
>> connector jar libraries in binary dist. is no effect.
>> (Someone proposed shading storm-redis because of dependencies... We can
>> cover it by various way like adding to pom, or using '--artifact' so no
>> issue at all for now.)
>>
>> So if we want to go with 3, I'd also like to propose excluding connector
>> libraries in binary dist. at all.
>>
>> Others look great to me.
>>
>> - Jungtaek Lim (HeartSaVioR)
>>
>> 2017년 3월 9일 (목) 오전 6:12, P. Taylor Goetz <pt...@gmail.com>님이 작성:
>>
>>> I’d like to propose the following to address the binary distribution size:
>>>
>>> 1. Finish moving connector examples to the “examples” directory.
>>> 2. Make all examples source-only in the binary distribution. This will
>>> mean users will have to build the examples themselves, but that’s typically
>>> what one does when looking at example code.
>>> 3. Stop unnecessarily shading connector libraries (i.e. let the user’s pom
>>> pull in the transitive dependencies when building a topology jar).
>>>
>>> If we agree on this approach, I’ll take up the necessary changes for the
>>> 1.1.0 release.
>>>
>>> -Taylor
>>>
>>>
>>>
>>>> On Mar 8, 2017, at 9:16 AM, Xin Wang <da...@gmail.com> wrote:
>>>>
>>>> Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release.
>>>>
>>>> - Xin
>>>>
>>>> 2017-03-07 9:16 GMT+08:00 Jungtaek Lim <ka...@gmail.com>:
>>>>
>>>>> Sorry Roshan but I don't get it.
>>>>> We're already having separate modules for each of the connectors, and
>>> even
>>>>> examples. We also don't have any common modules for external
>>> connectors, so
>>>>> those are completely individual.
>>>>>
>>>>> As I reported earlier, binary dist. will go up more than 500 MB when we
>>>>> address STORM-2390(STORM-2249 actually), which the community seemed not
>>>>> want to.
>>>>>
>>>>> Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather
>>>>> unlink from epic to get out of release, and discuss more.
>>>>> Any objections?
>>>>>
>>>>> - Jungtaek Lim (HeartSaVioR)
>>>>>
>>>>> 2017년 3월 4일 (토) 오전 10:02, Roshan Naik <ro...@hortonworks.com>님이 작성:
>>>>>
>>>>>> As having a separate module for each of the connectors will cause a lot
>>>>> of
>>>>>> bloat … may be good to club them into one.
>>>>>>
>>>>>> -roshan
>>>>>>
>>>>>>
>>>>>> On 3/2/17, 10:39 AM, "Sree V" <sr...@yahoo.com.INVALID> wrote:
>>>>>>
>>>>>> +1separate main binaries, connectors and samples/examples.
>>>>>>
>>>>>> Thanking you.
>>>>>> With Regards
>>>>>> Sree
>>>>>>
>>>>>> On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
>>>>>> satish.duggana@gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>> Agree that such huge binaries may not be acceptable. We should
>>>>> really
>>>>>> think
>>>>>> about the options proposed earlier like excluding some of the
>>>>> external
>>>>>> connectors from binary and update the documentation respectively.
>>>>>>
>>>>>> Thanks,
>>>>>> Satish.
>>>>>>
>>>>>> On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim <ka...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Adding my observation to my last mail:
>>>>>>> I just give STORM-2249 a try (against master branch) and compare
>>>>>> before vs
>>>>>>> after.
>>>>>>> It adds more than 300 MB, and the final archive is more than 550
>>>>> MB.
>>>>>> I
>>>>>>> guess it would be similar for 1.x branch.
>>>>>>>
>>>>>>> Before ----------
>>>>>>> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:41
>>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>>> pom
>>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
>>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>>> pom.asc
>>>>>>> -rw-r--r-- 1 jlim staff 264M 3 2 19:41
>>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>>> tar.gz
>>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
>>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>>> tar.gz.asc
>>>>>>> -rw-r--r-- 1 jlim staff 264M 3 2 19:41
>>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>>> zip
>>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
>>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>>> zip.asc
>>>>>>>
>>>>>>> After -----------
>>>>>>> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:46
>>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>>> pom
>>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:46
>>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>>> pom.asc
>>>>>>> -rw-r--r-- 1 jlim staff 564M 3 2 19:46
>>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>>> tar.gz
>>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:46
>>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>>> tar.gz.asc
>>>>>>> -rw-r--r-- 1 jlim staff 565M 3 2 19:46
>>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>>> zip
>>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:47
>>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>>> zip.asc
>>>>>>>
>>>>>>> While 264M is already a bit huge for me, 564M is not that I can
>>>>>> accept.
>>>>>>> (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka
>>>>>> 0.10.2 is
>>>>>>> 37M.)
>>>>>>>
>>>>>>> Btw, we're including source code of examples, and "mvn clean
>>>>>> package" will
>>>>>>> work for every example modules.
>>>>>>>
>>>>>>> 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>>>>>>
>>>>>>>> I guess it might be good time to think why we add all connectors
>>>>>> to the
>>>>>>>> binary distribution.
>>>>>>>>
>>>>>>>> Spark and Flink don't include them to binary dist. They even
>>>>> moved
>>>>>> some
>>>>>>> or
>>>>>>>> most of connectors out of repo, have been maintaining them in
>>>>>> Apache
>>>>>>> Bahir.
>>>>>>>> (Personally this is something I'm in favor of. We have lots of
>>>>>> connectors
>>>>>>>> and many of them are outdated - clear example is
>>>>>> storm-elasticsearch.)
>>>>>>>>
>>>>>>>> If we are assuming online then we don't even need to think about
>>>>>> users
>>>>>>>> touching binary dist. version of connectors. Users have been
>>>>>> including
>>>>>>> them
>>>>>>>> via build tools' dependency management, or even starting 1.1.0,
>>>>>> users can
>>>>>>>> include them via '--artifact' option.
>>>>>>>>
>>>>>>>> I also was just one of users for Storm, and I haven't use them
>>>>>> directly.
>>>>>>>> How much UX gets worse when we remove connectors to binary dist?
>>>>>> It only
>>>>>>>> helps some users who are not connected to the internet, and IMHO
>>>>>> it's a
>>>>>>>> rare case.
>>>>>>>>
>>>>>>>> I would like to see the opposite approach, removing all
>>>>> connectors
>>>>>> (or
>>>>>>>> just keeping storm-kafka/storm-kafka-client and some more
>>>>> preferred
>>>>>>> things)
>>>>>>>> and its relevant examples from binary dist.
>>>>>>>>
>>>>>>>> What do you think about it?
>>>>>>>>
>>>>>>>> - Jungtaek Lim (HeartSaVioR)
>>>>>>>>
>>>>>>>>
>>>>>>>> 2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ro...@hortonworks.com>님이
>>>>>> 작성:
>>>>>>>>
>>>>>>>> Once all of the shaded examples are included the size will go up
>>>>>> further.
>>>>>>>>
>>>>>>>> But currently as they are not part of the tar.gz … something else
>>>>>> is the
>>>>>>>> culprit for the bloat.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases
>>>>>> .. of
>>>>>>>> files that are larger than 4MB.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> @Jungtaek Lim<ma...@hortonworks.com> : I am thinking,
>>>>>> since the
>>>>>>>> code for the examples can be easily viewed online … it would be
>>>>>> valuable
>>>>>>> to
>>>>>>>> have the executable topologies made available to the user as part
>>>>>> of the
>>>>>>>> binary release … rather have them figure out how to build it
>>>>>> correctly
>>>>>>>> before trying them out.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -roshan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh
>>>>>> {} \;
>>>>>>>>
>>>>>>>> -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33
>>>>>>> ./examples/storm-starter/
>>>>>>>> storm-starter-topologies-1.0.3.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30
>>>>>>>> ./external/flux/flux-examples-1.0.3.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32
>>>>>>>> ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32
>>>>>>>> ./external/sql/storm-sql-core/guava-16.0.1.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30
>>>>>>>> ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-
>>>>>>> with-dependencies.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33
>>>>>>>>
>>>>>> ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33
>>>>>>>> ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
>>>>>>>>
>>>>>>>> -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02
>>>>>> ./lib/clojure-1.7.0.jar
>>>>>>>>
>>>>>>>> -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26
>>>>>>> ./lib/storm-core-1.0.3.jar
>>>>>>>>
>>>>>>>> -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26
>>>>>>>> ./lib/storm-rename-hack-1.0.3.jar
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh
>>>>>> {} \;
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23
>>>>>> ./examples/storm-pmml-
>>>>>>>> examples/storm-pmml-examples-1.1.0.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20
>>>>>>> ./examples/storm-starter/
>>>>>>>> storm-starter-topologies-1.1.0.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11
>>>>>>>> ./external/flux/flux-examples-1.1.0.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16
>>>>>>>> ./external/sql/storm-sql-core/calcite-core-1.11.0.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16
>>>>>>>> ./external/sql/storm-sql-core/guava-16.0.1.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12
>>>>>>> ./external/sql/storm-sql-
>>>>>>>> runtime/calcite-core-1.11.0.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12
>>>>>>> ./external/sql/storm-sql-
>>>>>>>> runtime/guava-16.0.1.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18
>>>>>>>> ./external/storm-druid/storm-druid-1.1.0.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11
>>>>>>>> ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-
>>>>>>> with-dependencies.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20
>>>>>>>>
>>>>>> ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18
>>>>>>>> ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02
>>>>>> ./lib/clojure-1.7.0.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07
>>>>>>> ./lib/storm-core-1.1.0.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07
>>>>>>>> ./lib/storm-rename-hack-1.1.0.jar
>>>>>>>>
>>>>>>>> -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19
>>>>>>>> ./toollib/storm-kafka-monitor-1.1.0.jar
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> About STORM-2249, since examples are shading their
>>>>> dependencies,
>>>>>>> binary
>>>>>>>>
>>>>>>>> dist will grow much bigger. I've left some comments regarding
>>>>>> that.
>>>>>>>>
>>>>>>>> Btw, I have another view of this. Showing example codes is
>>>>> more
>>>>>>>> important
>>>>>>>>
>>>>>>>> than just let users execute some topologies. That's what
>>>>> example
>>>>>>>> modules
>>>>>>>>
>>>>>>>> are for. We need to include source as well. If we need to pick
>>>>>> one,
>>>>>>>> source
>>>>>>>>
>>>>>>>> code would be better.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> STORM-2343 seems better to add to 1.1.0. I just am not enough
>>>>>>> familiar
>>>>>>>> with
>>>>>>>>
>>>>>>>> storm-kafka-client so not sure I can review that, but I'll try
>>>>>> to. I
>>>>>>>> feel
>>>>>>>>
>>>>>>>> it's not that make release dragged. Let's add to 1.1.0 epic.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Let's make minimum merge before another RC vote. Addressing
>>>>>>> STORM-2389
>>>>>>>> (and
>>>>>>>>
>>>>>>>> maybe STORM-2343) is enough for me. Others are not that
>>>>>> critical.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <
>>>>>>>> hlouro@hortonworks.com>
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Roshan, does this PR<
>>>>>> https://github.com/apache/storm/pull/1831>
>>>>>>> and
>>>>>>>> JIRA<
>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/STORM-2249> address
>>>>> the
>>>>>>>> missing
>>>>>>>>
>>>>>>>>> jars problem that you mentioned. I had created it in
>>>>> December
>>>>>> 2016,
>>>>>>>> but
>>>>>>>>
>>>>>>>>> there is an ongoing discussion if we should indeed put the
>>>>>> jars in
>>>>>>>> the
>>>>>>>>
>>>>>>>>> examples location or not.
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> On a different note, this storm-kafka-client/KafkaSpout PR<
>>>>>>>>
>>>>>>>>> https://github.com/apache/storm/pull/1924> fixes a bug with
>>>>>> the
>>>>>>>> number of
>>>>>>>>
>>>>>>>>> uncommitted offsets that is quite important. It is not a
>>>>>> blocker,
>>>>>>>> but it is
>>>>>>>>
>>>>>>>>> quite critical. I am going to do one last pass reviewing
>>>>>> today. It
>>>>>>>> would be
>>>>>>>>
>>>>>>>>> good if we could have this PR included with the release. Can
>>>>>> anyone
>>>>>>>> else
>>>>>>>>
>>>>>>>>> review it as well ?
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>>> Hugo
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <
>>>>>> ptgoetz@gmail.com
>>>>>>>> <mailto:ptgo
>>>>>>>>
>>>>>>>>> etz@gmail.com>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> Yeah, I don’t think the file size is a killer/blocker. It’s
>>>>>> largely
>>>>>>>> due to
>>>>>>>>
>>>>>>>>> shaded examples, etc. But it’s something to keep an eye on.
>>>>>> Our
>>>>>>>> binary
>>>>>>>>
>>>>>>>>> releases shouldn’t have to be that big.
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> -Taylor
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> On Mar 1, 2017, at 12:09 PM, Roshan Naik <
>>>>>> roshan@hortonworks.com<
>>>>>>>> mailto:
>>>>>>>>
>>>>>>>>> roshan@hortonworks.com>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> Have filed Jiras so for the 3 issues mentioned. Not sure if
>>>>>> we need
>>>>>>>> a JIRA
>>>>>>>>
>>>>>>>>> for the file size getting bloated by that much.
>>>>>>>>
>>>>>>>>> Somebody better familiar with the matter may want to take
>>>>>> about
>>>>>>> that?
>>>>>>>>
>>>>>>>>> -roshan
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com
>>>>>> <mailto:
>>>>>>> ptgo
>>>>>>>>
>>>>>>>>> etz@gmail.com>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks for bringing these up Roshan. Feel free to file JIRA
>>>>>>>> tickets for
>>>>>>>>
>>>>>>>>> these issues and assign the “Release Apache Storm 1.1.0”
>>>>> epic
>>>>>> so
>>>>>>>> they can
>>>>>>>>
>>>>>>>>> be tracked for this release.
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> -Taylor
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> On Mar 1, 2017, at 9:27 AM, Roshan Naik <
>>>>>> roshan@hortonworks.com<
>>>>>>>> mailto:
>>>>>>>>
>>>>>>>>> roshan@hortonworks.com>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> Found these additional issues:
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> 1- BUG: Even if topology.eventlogger.executors=0, the
>>>>>>> event_logger
>>>>>>>> bolt
>>>>>>>>
>>>>>>>>> is instantiated … previously observed to cause ~10%
>>>>>> degradation in
>>>>>>>> perf
>>>>>>>>
>>>>>>>>> even with logging disabled.
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> 2- Missing Jars: The storm-*-examples jars are missing in
>>>>> the
>>>>>>> binary
>>>>>>>>
>>>>>>>>> distro (other than a storm-pmml-examples.jar,
>>>>>>> storm-jms-examples.jar
>>>>>>>> &
>>>>>>>>
>>>>>>>>> flux-examples.jar).
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> 3- Minor: HdfsSpoutTopology example has not been moved into
>>>>>>>>
>>>>>>>>> storm-hdfs-examples from storm-starter
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> Another side observation … v1.0.3 tar.gz downloadable was
>>>>>> 190MB.
>>>>>>>> This
>>>>>>>>
>>>>>>>>> v1.1.0 tar.gz downloadable is 297MB !! …. even though some
>>>>>> of the
>>>>>>>> example
>>>>>>>>
>>>>>>>>> topologies didn’t make it.
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> -roshan
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Name : Jungtaek Lim
>>>>>>>>
>>>>>>>> Blog : http://medium.com/@heartsavior
>>>>>>>>
>>>>>>>> Twitter : http://twitter.com/heartsavior
>>>>>>>>
>>>>>>>> LinkedIn : http://www.linkedin.com/in/heartsavior
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>>
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
+1 for that, although I think it is the examples that ends up being the heavy part. The examples are all uber jars and pull in lots of dependencies. Some of them multiple times over. Do we want the examples to only be a source release so before running an example they have to build it?
- Bobby
On Thursday, March 9, 2017, 2:22:42 PM CST, P. Taylor Goetz <pt...@gmail.com> wrote:I’m fine with dropping connectors from the binary distribution in favor having users use Maven, gradle, etc. handle user dependencies. Building topology jars manually (vs. letting Maven do the heavy lifting) is painful and something to be avoided, specifically for new users.
So the “binary release” for connectors, etc. would essentially become just a README that documents how to use them, and what the Maven coordinates are for each component.
So with the connector jars just hosted in the Maven repo, and all examples source-only (to be built by users) the size of the binary distribution should be reduced dramatically.
Any other opinions?
-Taylor
> On Mar 8, 2017, at 4:26 PM, Jungtaek Lim <ka...@gmail.com> wrote:
>
> Regarding 3, if we don't make uber jar for connectors, adding jar file in
> binary dist. to the classpath will not work, which means that including
> connector jar libraries in binary dist. is no effect.
> (Someone proposed shading storm-redis because of dependencies... We can
> cover it by various way like adding to pom, or using '--artifact' so no
> issue at all for now.)
>
> So if we want to go with 3, I'd also like to propose excluding connector
> libraries in binary dist. at all.
>
> Others look great to me.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 3월 9일 (목) 오전 6:12, P. Taylor Goetz <pt...@gmail.com>님이 작성:
>
>> I’d like to propose the following to address the binary distribution size:
>>
>> 1. Finish moving connector examples to the “examples” directory.
>> 2. Make all examples source-only in the binary distribution. This will
>> mean users will have to build the examples themselves, but that’s typically
>> what one does when looking at example code.
>> 3. Stop unnecessarily shading connector libraries (i.e. let the user’s pom
>> pull in the transitive dependencies when building a topology jar).
>>
>> If we agree on this approach, I’ll take up the necessary changes for the
>> 1.1.0 release.
>>
>> -Taylor
>>
>>
>>
>>> On Mar 8, 2017, at 9:16 AM, Xin Wang <da...@gmail.com> wrote:
>>>
>>> Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release.
>>>
>>> - Xin
>>>
>>> 2017-03-07 9:16 GMT+08:00 Jungtaek Lim <ka...@gmail.com>:
>>>
>>>> Sorry Roshan but I don't get it.
>>>> We're already having separate modules for each of the connectors, and
>> even
>>>> examples. We also don't have any common modules for external
>> connectors, so
>>>> those are completely individual.
>>>>
>>>> As I reported earlier, binary dist. will go up more than 500 MB when we
>>>> address STORM-2390(STORM-2249 actually), which the community seemed not
>>>> want to.
>>>>
>>>> Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather
>>>> unlink from epic to get out of release, and discuss more.
>>>> Any objections?
>>>>
>>>> - Jungtaek Lim (HeartSaVioR)
>>>>
>>>> 2017년 3월 4일 (토) 오전 10:02, Roshan Naik <ro...@hortonworks.com>님이 작성:
>>>>
>>>>> As having a separate module for each of the connectors will cause a lot
>>>> of
>>>>> bloat … may be good to club them into one.
>>>>>
>>>>> -roshan
>>>>>
>>>>>
>>>>> On 3/2/17, 10:39 AM, "Sree V" <sr...@yahoo.com.INVALID> wrote:
>>>>>
>>>>> +1separate main binaries, connectors and samples/examples.
>>>>>
>>>>> Thanking you.
>>>>> With Regards
>>>>> Sree
>>>>>
>>>>> On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
>>>>> satish.duggana@gmail.com> wrote:
>>>>>
>>>>>
>>>>> Agree that such huge binaries may not be acceptable. We should
>>>> really
>>>>> think
>>>>> about the options proposed earlier like excluding some of the
>>>> external
>>>>> connectors from binary and update the documentation respectively.
>>>>>
>>>>> Thanks,
>>>>> Satish.
>>>>>
>>>>> On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim <ka...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Adding my observation to my last mail:
>>>>>> I just give STORM-2249 a try (against master branch) and compare
>>>>> before vs
>>>>>> after.
>>>>>> It adds more than 300 MB, and the final archive is more than 550
>>>> MB.
>>>>> I
>>>>>> guess it would be similar for 1.x branch.
>>>>>>
>>>>>> Before ----------
>>>>>> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:41
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> pom
>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> pom.asc
>>>>>> -rw-r--r-- 1 jlim staff 264M 3 2 19:41
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> tar.gz
>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> tar.gz.asc
>>>>>> -rw-r--r-- 1 jlim staff 264M 3 2 19:41
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> zip
>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> zip.asc
>>>>>>
>>>>>> After -----------
>>>>>> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:46
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> pom
>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:46
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> pom.asc
>>>>>> -rw-r--r-- 1 jlim staff 564M 3 2 19:46
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> tar.gz
>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:46
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> tar.gz.asc
>>>>>> -rw-r--r-- 1 jlim staff 565M 3 2 19:46
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> zip
>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:47
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> zip.asc
>>>>>>
>>>>>> While 264M is already a bit huge for me, 564M is not that I can
>>>>> accept.
>>>>>> (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka
>>>>> 0.10.2 is
>>>>>> 37M.)
>>>>>>
>>>>>> Btw, we're including source code of examples, and "mvn clean
>>>>> package" will
>>>>>> work for every example modules.
>>>>>>
>>>>>> 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>>>>>
>>>>>>> I guess it might be good time to think why we add all connectors
>>>>> to the
>>>>>>> binary distribution.
>>>>>>>
>>>>>>> Spark and Flink don't include them to binary dist. They even
>>>> moved
>>>>> some
>>>>>> or
>>>>>>> most of connectors out of repo, have been maintaining them in
>>>>> Apache
>>>>>> Bahir.
>>>>>>> (Personally this is something I'm in favor of. We have lots of
>>>>> connectors
>>>>>>> and many of them are outdated - clear example is
>>>>> storm-elasticsearch.)
>>>>>>>
>>>>>>> If we are assuming online then we don't even need to think about
>>>>> users
>>>>>>> touching binary dist. version of connectors. Users have been
>>>>> including
>>>>>> them
>>>>>>> via build tools' dependency management, or even starting 1.1.0,
>>>>> users can
>>>>>>> include them via '--artifact' option.
>>>>>>>
>>>>>>> I also was just one of users for Storm, and I haven't use them
>>>>> directly.
>>>>>>> How much UX gets worse when we remove connectors to binary dist?
>>>>> It only
>>>>>>> helps some users who are not connected to the internet, and IMHO
>>>>> it's a
>>>>>>> rare case.
>>>>>>>
>>>>>>> I would like to see the opposite approach, removing all
>>>> connectors
>>>>> (or
>>>>>>> just keeping storm-kafka/storm-kafka-client and some more
>>>> preferred
>>>>>> things)
>>>>>>> and its relevant examples from binary dist.
>>>>>>>
>>>>>>> What do you think about it?
>>>>>>>
>>>>>>> - Jungtaek Lim (HeartSaVioR)
>>>>>>>
>>>>>>>
>>>>>>> 2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ro...@hortonworks.com>님이
>>>>> 작성:
>>>>>>>
>>>>>>> Once all of the shaded examples are included the size will go up
>>>>> further.
>>>>>>>
>>>>>>> But currently as they are not part of the tar.gz … something else
>>>>> is the
>>>>>>> culprit for the bloat.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases
>>>>> .. of
>>>>>>> files that are larger than 4MB.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> @Jungtaek Lim<ma...@hortonworks.com> : I am thinking,
>>>>> since the
>>>>>>> code for the examples can be easily viewed online … it would be
>>>>> valuable
>>>>>> to
>>>>>>> have the executable topologies made available to the user as part
>>>>> of the
>>>>>>> binary release … rather have them figure out how to build it
>>>>> correctly
>>>>>>> before trying them out.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -roshan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh
>>>>> {} \;
>>>>>>>
>>>>>>> -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33
>>>>>> ./examples/storm-starter/
>>>>>>> storm-starter-topologies-1.0.3.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30
>>>>>>> ./external/flux/flux-examples-1.0.3.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32
>>>>>>> ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32
>>>>>>> ./external/sql/storm-sql-core/guava-16.0.1.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30
>>>>>>> ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-
>>>>>> with-dependencies.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33
>>>>>>>
>>>>> ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33
>>>>>>> ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
>>>>>>>
>>>>>>> -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02
>>>>> ./lib/clojure-1.7.0.jar
>>>>>>>
>>>>>>> -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26
>>>>>> ./lib/storm-core-1.0.3.jar
>>>>>>>
>>>>>>> -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26
>>>>>>> ./lib/storm-rename-hack-1.0.3.jar
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh
>>>>> {} \;
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23
>>>>> ./examples/storm-pmml-
>>>>>>> examples/storm-pmml-examples-1.1.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20
>>>>>> ./examples/storm-starter/
>>>>>>> storm-starter-topologies-1.1.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11
>>>>>>> ./external/flux/flux-examples-1.1.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16
>>>>>>> ./external/sql/storm-sql-core/calcite-core-1.11.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16
>>>>>>> ./external/sql/storm-sql-core/guava-16.0.1.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12
>>>>>> ./external/sql/storm-sql-
>>>>>>> runtime/calcite-core-1.11.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12
>>>>>> ./external/sql/storm-sql-
>>>>>>> runtime/guava-16.0.1.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18
>>>>>>> ./external/storm-druid/storm-druid-1.1.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11
>>>>>>> ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-
>>>>>> with-dependencies.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20
>>>>>>>
>>>>> ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18
>>>>>>> ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02
>>>>> ./lib/clojure-1.7.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07
>>>>>> ./lib/storm-core-1.1.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07
>>>>>>> ./lib/storm-rename-hack-1.1.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19
>>>>>>> ./toollib/storm-kafka-monitor-1.1.0.jar
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> About STORM-2249, since examples are shading their
>>>> dependencies,
>>>>>> binary
>>>>>>>
>>>>>>> dist will grow much bigger. I've left some comments regarding
>>>>> that.
>>>>>>>
>>>>>>> Btw, I have another view of this. Showing example codes is
>>>> more
>>>>>>> important
>>>>>>>
>>>>>>> than just let users execute some topologies. That's what
>>>> example
>>>>>>> modules
>>>>>>>
>>>>>>> are for. We need to include source as well. If we need to pick
>>>>> one,
>>>>>>> source
>>>>>>>
>>>>>>> code would be better.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> STORM-2343 seems better to add to 1.1.0. I just am not enough
>>>>>> familiar
>>>>>>> with
>>>>>>>
>>>>>>> storm-kafka-client so not sure I can review that, but I'll try
>>>>> to. I
>>>>>>> feel
>>>>>>>
>>>>>>> it's not that make release dragged. Let's add to 1.1.0 epic.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Let's make minimum merge before another RC vote. Addressing
>>>>>> STORM-2389
>>>>>>> (and
>>>>>>>
>>>>>>> maybe STORM-2343) is enough for me. Others are not that
>>>>> critical.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <
>>>>>>> hlouro@hortonworks.com>
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Roshan, does this PR<
>>>>> https://github.com/apache/storm/pull/1831>
>>>>>> and
>>>>>>> JIRA<
>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/STORM-2249> address
>>>> the
>>>>>>> missing
>>>>>>>
>>>>>>>> jars problem that you mentioned. I had created it in
>>>> December
>>>>> 2016,
>>>>>>> but
>>>>>>>
>>>>>>>> there is an ongoing discussion if we should indeed put the
>>>>> jars in
>>>>>>> the
>>>>>>>
>>>>>>>> examples location or not.
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> On a different note, this storm-kafka-client/KafkaSpout PR<
>>>>>>>
>>>>>>>> https://github.com/apache/storm/pull/1924> fixes a bug with
>>>>> the
>>>>>>> number of
>>>>>>>
>>>>>>>> uncommitted offsets that is quite important. It is not a
>>>>> blocker,
>>>>>>> but it is
>>>>>>>
>>>>>>>> quite critical. I am going to do one last pass reviewing
>>>>> today. It
>>>>>>> would be
>>>>>>>
>>>>>>>> good if we could have this PR included with the release. Can
>>>>> anyone
>>>>>>> else
>>>>>>>
>>>>>>>> review it as well ?
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>
>>>>>>>> Hugo
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <
>>>>> ptgoetz@gmail.com
>>>>>>> <mailto:ptgo
>>>>>>>
>>>>>>>> etz@gmail.com>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> Yeah, I don’t think the file size is a killer/blocker. It’s
>>>>> largely
>>>>>>> due to
>>>>>>>
>>>>>>>> shaded examples, etc. But it’s something to keep an eye on.
>>>>> Our
>>>>>>> binary
>>>>>>>
>>>>>>>> releases shouldn’t have to be that big.
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> -Taylor
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> On Mar 1, 2017, at 12:09 PM, Roshan Naik <
>>>>> roshan@hortonworks.com<
>>>>>>> mailto:
>>>>>>>
>>>>>>>> roshan@hortonworks.com>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> Have filed Jiras so for the 3 issues mentioned. Not sure if
>>>>> we need
>>>>>>> a JIRA
>>>>>>>
>>>>>>>> for the file size getting bloated by that much.
>>>>>>>
>>>>>>>> Somebody better familiar with the matter may want to take
>>>>> about
>>>>>> that?
>>>>>>>
>>>>>>>> -roshan
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com
>>>>> <mailto:
>>>>>> ptgo
>>>>>>>
>>>>>>>> etz@gmail.com>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> Thanks for bringing these up Roshan. Feel free to file JIRA
>>>>>>> tickets for
>>>>>>>
>>>>>>>> these issues and assign the “Release Apache Storm 1.1.0”
>>>> epic
>>>>> so
>>>>>>> they can
>>>>>>>
>>>>>>>> be tracked for this release.
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> -Taylor
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> On Mar 1, 2017, at 9:27 AM, Roshan Naik <
>>>>> roshan@hortonworks.com<
>>>>>>> mailto:
>>>>>>>
>>>>>>>> roshan@hortonworks.com>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> Found these additional issues:
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> 1- BUG: Even if topology.eventlogger.executors=0, the
>>>>>> event_logger
>>>>>>> bolt
>>>>>>>
>>>>>>>> is instantiated … previously observed to cause ~10%
>>>>> degradation in
>>>>>>> perf
>>>>>>>
>>>>>>>> even with logging disabled.
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> 2- Missing Jars: The storm-*-examples jars are missing in
>>>> the
>>>>>> binary
>>>>>>>
>>>>>>>> distro (other than a storm-pmml-examples.jar,
>>>>>> storm-jms-examples.jar
>>>>>>> &
>>>>>>>
>>>>>>>> flux-examples.jar).
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> 3- Minor: HdfsSpoutTopology example has not been moved into
>>>>>>>
>>>>>>>> storm-hdfs-examples from storm-starter
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> Another side observation … v1.0.3 tar.gz downloadable was
>>>>> 190MB.
>>>>>>> This
>>>>>>>
>>>>>>>> v1.1.0 tar.gz downloadable is 297MB !! …. even though some
>>>>> of the
>>>>>>> example
>>>>>>>
>>>>>>>> topologies didn’t make it.
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> -roshan
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Name : Jungtaek Lim
>>>>>>>
>>>>>>> Blog : http://medium.com/@heartsavior
>>>>>>>
>>>>>>> Twitter : http://twitter.com/heartsavior
>>>>>>>
>>>>>>> LinkedIn : http://www.linkedin.com/in/heartsavior
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>
>>
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
I’m fine with dropping connectors from the binary distribution in favor having users use Maven, gradle, etc. handle user dependencies. Building topology jars manually (vs. letting Maven do the heavy lifting) is painful and something to be avoided, specifically for new users.
So the “binary release” for connectors, etc. would essentially become just a README that documents how to use them, and what the Maven coordinates are for each component.
So with the connector jars just hosted in the Maven repo, and all examples source-only (to be built by users) the size of the binary distribution should be reduced dramatically.
Any other opinions?
-Taylor
> On Mar 8, 2017, at 4:26 PM, Jungtaek Lim <ka...@gmail.com> wrote:
>
> Regarding 3, if we don't make uber jar for connectors, adding jar file in
> binary dist. to the classpath will not work, which means that including
> connector jar libraries in binary dist. is no effect.
> (Someone proposed shading storm-redis because of dependencies... We can
> cover it by various way like adding to pom, or using '--artifact' so no
> issue at all for now.)
>
> So if we want to go with 3, I'd also like to propose excluding connector
> libraries in binary dist. at all.
>
> Others look great to me.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 3월 9일 (목) 오전 6:12, P. Taylor Goetz <pt...@gmail.com>님이 작성:
>
>> I’d like to propose the following to address the binary distribution size:
>>
>> 1. Finish moving connector examples to the “examples” directory.
>> 2. Make all examples source-only in the binary distribution. This will
>> mean users will have to build the examples themselves, but that’s typically
>> what one does when looking at example code.
>> 3. Stop unnecessarily shading connector libraries (i.e. let the user’s pom
>> pull in the transitive dependencies when building a topology jar).
>>
>> If we agree on this approach, I’ll take up the necessary changes for the
>> 1.1.0 release.
>>
>> -Taylor
>>
>>
>>
>>> On Mar 8, 2017, at 9:16 AM, Xin Wang <da...@gmail.com> wrote:
>>>
>>> Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release.
>>>
>>> - Xin
>>>
>>> 2017-03-07 9:16 GMT+08:00 Jungtaek Lim <ka...@gmail.com>:
>>>
>>>> Sorry Roshan but I don't get it.
>>>> We're already having separate modules for each of the connectors, and
>> even
>>>> examples. We also don't have any common modules for external
>> connectors, so
>>>> those are completely individual.
>>>>
>>>> As I reported earlier, binary dist. will go up more than 500 MB when we
>>>> address STORM-2390(STORM-2249 actually), which the community seemed not
>>>> want to.
>>>>
>>>> Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather
>>>> unlink from epic to get out of release, and discuss more.
>>>> Any objections?
>>>>
>>>> - Jungtaek Lim (HeartSaVioR)
>>>>
>>>> 2017년 3월 4일 (토) 오전 10:02, Roshan Naik <ro...@hortonworks.com>님이 작성:
>>>>
>>>>> As having a separate module for each of the connectors will cause a lot
>>>> of
>>>>> bloat … may be good to club them into one.
>>>>>
>>>>> -roshan
>>>>>
>>>>>
>>>>> On 3/2/17, 10:39 AM, "Sree V" <sr...@yahoo.com.INVALID> wrote:
>>>>>
>>>>> +1separate main binaries, connectors and samples/examples.
>>>>>
>>>>> Thanking you.
>>>>> With Regards
>>>>> Sree
>>>>>
>>>>> On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
>>>>> satish.duggana@gmail.com> wrote:
>>>>>
>>>>>
>>>>> Agree that such huge binaries may not be acceptable. We should
>>>> really
>>>>> think
>>>>> about the options proposed earlier like excluding some of the
>>>> external
>>>>> connectors from binary and update the documentation respectively.
>>>>>
>>>>> Thanks,
>>>>> Satish.
>>>>>
>>>>> On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim <ka...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Adding my observation to my last mail:
>>>>>> I just give STORM-2249 a try (against master branch) and compare
>>>>> before vs
>>>>>> after.
>>>>>> It adds more than 300 MB, and the final archive is more than 550
>>>> MB.
>>>>> I
>>>>>> guess it would be similar for 1.x branch.
>>>>>>
>>>>>> Before ----------
>>>>>> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:41
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> pom
>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> pom.asc
>>>>>> -rw-r--r-- 1 jlim staff 264M 3 2 19:41
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> tar.gz
>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> tar.gz.asc
>>>>>> -rw-r--r-- 1 jlim staff 264M 3 2 19:41
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> zip
>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> zip.asc
>>>>>>
>>>>>> After -----------
>>>>>> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:46
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> pom
>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:46
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> pom.asc
>>>>>> -rw-r--r-- 1 jlim staff 564M 3 2 19:46
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> tar.gz
>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:46
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> tar.gz.asc
>>>>>> -rw-r--r-- 1 jlim staff 565M 3 2 19:46
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> zip
>>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:47
>>>>> apache-storm-2.0.0-SNAPSHOT.
>>>>>> zip.asc
>>>>>>
>>>>>> While 264M is already a bit huge for me, 564M is not that I can
>>>>> accept.
>>>>>> (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka
>>>>> 0.10.2 is
>>>>>> 37M.)
>>>>>>
>>>>>> Btw, we're including source code of examples, and "mvn clean
>>>>> package" will
>>>>>> work for every example modules.
>>>>>>
>>>>>> 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>>>>>
>>>>>>> I guess it might be good time to think why we add all connectors
>>>>> to the
>>>>>>> binary distribution.
>>>>>>>
>>>>>>> Spark and Flink don't include them to binary dist. They even
>>>> moved
>>>>> some
>>>>>> or
>>>>>>> most of connectors out of repo, have been maintaining them in
>>>>> Apache
>>>>>> Bahir.
>>>>>>> (Personally this is something I'm in favor of. We have lots of
>>>>> connectors
>>>>>>> and many of them are outdated - clear example is
>>>>> storm-elasticsearch.)
>>>>>>>
>>>>>>> If we are assuming online then we don't even need to think about
>>>>> users
>>>>>>> touching binary dist. version of connectors. Users have been
>>>>> including
>>>>>> them
>>>>>>> via build tools' dependency management, or even starting 1.1.0,
>>>>> users can
>>>>>>> include them via '--artifact' option.
>>>>>>>
>>>>>>> I also was just one of users for Storm, and I haven't use them
>>>>> directly.
>>>>>>> How much UX gets worse when we remove connectors to binary dist?
>>>>> It only
>>>>>>> helps some users who are not connected to the internet, and IMHO
>>>>> it's a
>>>>>>> rare case.
>>>>>>>
>>>>>>> I would like to see the opposite approach, removing all
>>>> connectors
>>>>> (or
>>>>>>> just keeping storm-kafka/storm-kafka-client and some more
>>>> preferred
>>>>>> things)
>>>>>>> and its relevant examples from binary dist.
>>>>>>>
>>>>>>> What do you think about it?
>>>>>>>
>>>>>>> - Jungtaek Lim (HeartSaVioR)
>>>>>>>
>>>>>>>
>>>>>>> 2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ro...@hortonworks.com>님이
>>>>> 작성:
>>>>>>>
>>>>>>> Once all of the shaded examples are included the size will go up
>>>>> further.
>>>>>>>
>>>>>>> But currently as they are not part of the tar.gz … something else
>>>>> is the
>>>>>>> culprit for the bloat.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases
>>>>> .. of
>>>>>>> files that are larger than 4MB.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> @Jungtaek Lim<ma...@hortonworks.com> : I am thinking,
>>>>> since the
>>>>>>> code for the examples can be easily viewed online … it would be
>>>>> valuable
>>>>>> to
>>>>>>> have the executable topologies made available to the user as part
>>>>> of the
>>>>>>> binary release … rather have them figure out how to build it
>>>>> correctly
>>>>>>> before trying them out.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -roshan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh
>>>>> {} \;
>>>>>>>
>>>>>>> -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33
>>>>>> ./examples/storm-starter/
>>>>>>> storm-starter-topologies-1.0.3.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30
>>>>>>> ./external/flux/flux-examples-1.0.3.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32
>>>>>>> ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32
>>>>>>> ./external/sql/storm-sql-core/guava-16.0.1.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30
>>>>>>> ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-
>>>>>> with-dependencies.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33
>>>>>>>
>>>>> ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33
>>>>>>> ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
>>>>>>>
>>>>>>> -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02
>>>>> ./lib/clojure-1.7.0.jar
>>>>>>>
>>>>>>> -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26
>>>>>> ./lib/storm-core-1.0.3.jar
>>>>>>>
>>>>>>> -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26
>>>>>>> ./lib/storm-rename-hack-1.0.3.jar
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh
>>>>> {} \;
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23
>>>>> ./examples/storm-pmml-
>>>>>>> examples/storm-pmml-examples-1.1.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20
>>>>>> ./examples/storm-starter/
>>>>>>> storm-starter-topologies-1.1.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11
>>>>>>> ./external/flux/flux-examples-1.1.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16
>>>>>>> ./external/sql/storm-sql-core/calcite-core-1.11.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16
>>>>>>> ./external/sql/storm-sql-core/guava-16.0.1.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12
>>>>>> ./external/sql/storm-sql-
>>>>>>> runtime/calcite-core-1.11.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12
>>>>>> ./external/sql/storm-sql-
>>>>>>> runtime/guava-16.0.1.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18
>>>>>>> ./external/storm-druid/storm-druid-1.1.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11
>>>>>>> ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-
>>>>>> with-dependencies.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20
>>>>>>>
>>>>> ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18
>>>>>>> ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02
>>>>> ./lib/clojure-1.7.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07
>>>>>> ./lib/storm-core-1.1.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07
>>>>>>> ./lib/storm-rename-hack-1.1.0.jar
>>>>>>>
>>>>>>> -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19
>>>>>>> ./toollib/storm-kafka-monitor-1.1.0.jar
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> About STORM-2249, since examples are shading their
>>>> dependencies,
>>>>>> binary
>>>>>>>
>>>>>>> dist will grow much bigger. I've left some comments regarding
>>>>> that.
>>>>>>>
>>>>>>> Btw, I have another view of this. Showing example codes is
>>>> more
>>>>>>> important
>>>>>>>
>>>>>>> than just let users execute some topologies. That's what
>>>> example
>>>>>>> modules
>>>>>>>
>>>>>>> are for. We need to include source as well. If we need to pick
>>>>> one,
>>>>>>> source
>>>>>>>
>>>>>>> code would be better.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> STORM-2343 seems better to add to 1.1.0. I just am not enough
>>>>>> familiar
>>>>>>> with
>>>>>>>
>>>>>>> storm-kafka-client so not sure I can review that, but I'll try
>>>>> to. I
>>>>>>> feel
>>>>>>>
>>>>>>> it's not that make release dragged. Let's add to 1.1.0 epic.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Let's make minimum merge before another RC vote. Addressing
>>>>>> STORM-2389
>>>>>>> (and
>>>>>>>
>>>>>>> maybe STORM-2343) is enough for me. Others are not that
>>>>> critical.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <
>>>>>>> hlouro@hortonworks.com>
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Roshan, does this PR<
>>>>> https://github.com/apache/storm/pull/1831>
>>>>>> and
>>>>>>> JIRA<
>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/STORM-2249> address
>>>> the
>>>>>>> missing
>>>>>>>
>>>>>>>> jars problem that you mentioned. I had created it in
>>>> December
>>>>> 2016,
>>>>>>> but
>>>>>>>
>>>>>>>> there is an ongoing discussion if we should indeed put the
>>>>> jars in
>>>>>>> the
>>>>>>>
>>>>>>>> examples location or not.
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> On a different note, this storm-kafka-client/KafkaSpout PR<
>>>>>>>
>>>>>>>> https://github.com/apache/storm/pull/1924> fixes a bug with
>>>>> the
>>>>>>> number of
>>>>>>>
>>>>>>>> uncommitted offsets that is quite important. It is not a
>>>>> blocker,
>>>>>>> but it is
>>>>>>>
>>>>>>>> quite critical. I am going to do one last pass reviewing
>>>>> today. It
>>>>>>> would be
>>>>>>>
>>>>>>>> good if we could have this PR included with the release. Can
>>>>> anyone
>>>>>>> else
>>>>>>>
>>>>>>>> review it as well ?
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>
>>>>>>>> Hugo
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <
>>>>> ptgoetz@gmail.com
>>>>>>> <mailto:ptgo
>>>>>>>
>>>>>>>> etz@gmail.com>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> Yeah, I don’t think the file size is a killer/blocker. It’s
>>>>> largely
>>>>>>> due to
>>>>>>>
>>>>>>>> shaded examples, etc. But it’s something to keep an eye on.
>>>>> Our
>>>>>>> binary
>>>>>>>
>>>>>>>> releases shouldn’t have to be that big.
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> -Taylor
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> On Mar 1, 2017, at 12:09 PM, Roshan Naik <
>>>>> roshan@hortonworks.com<
>>>>>>> mailto:
>>>>>>>
>>>>>>>> roshan@hortonworks.com>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> Have filed Jiras so for the 3 issues mentioned. Not sure if
>>>>> we need
>>>>>>> a JIRA
>>>>>>>
>>>>>>>> for the file size getting bloated by that much.
>>>>>>>
>>>>>>>> Somebody better familiar with the matter may want to take
>>>>> about
>>>>>> that?
>>>>>>>
>>>>>>>> -roshan
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com
>>>>> <mailto:
>>>>>> ptgo
>>>>>>>
>>>>>>>> etz@gmail.com>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> Thanks for bringing these up Roshan. Feel free to file JIRA
>>>>>>> tickets for
>>>>>>>
>>>>>>>> these issues and assign the “Release Apache Storm 1.1.0”
>>>> epic
>>>>> so
>>>>>>> they can
>>>>>>>
>>>>>>>> be tracked for this release.
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> -Taylor
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> On Mar 1, 2017, at 9:27 AM, Roshan Naik <
>>>>> roshan@hortonworks.com<
>>>>>>> mailto:
>>>>>>>
>>>>>>>> roshan@hortonworks.com>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> Found these additional issues:
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> 1- BUG: Even if topology.eventlogger.executors=0, the
>>>>>> event_logger
>>>>>>> bolt
>>>>>>>
>>>>>>>> is instantiated … previously observed to cause ~10%
>>>>> degradation in
>>>>>>> perf
>>>>>>>
>>>>>>>> even with logging disabled.
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> 2- Missing Jars: The storm-*-examples jars are missing in
>>>> the
>>>>>> binary
>>>>>>>
>>>>>>>> distro (other than a storm-pmml-examples.jar,
>>>>>> storm-jms-examples.jar
>>>>>>> &
>>>>>>>
>>>>>>>> flux-examples.jar).
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> 3- Minor: HdfsSpoutTopology example has not been moved into
>>>>>>>
>>>>>>>> storm-hdfs-examples from storm-starter
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> Another side observation … v1.0.3 tar.gz downloadable was
>>>>> 190MB.
>>>>>>> This
>>>>>>>
>>>>>>>> v1.1.0 tar.gz downloadable is 297MB !! …. even though some
>>>>> of the
>>>>>>> example
>>>>>>>
>>>>>>>> topologies didn’t make it.
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> -roshan
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Name : Jungtaek Lim
>>>>>>>
>>>>>>> Blog : http://medium.com/@heartsavior
>>>>>>>
>>>>>>> Twitter : http://twitter.com/heartsavior
>>>>>>>
>>>>>>> LinkedIn : http://www.linkedin.com/in/heartsavior
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>
>>
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Jungtaek Lim <ka...@gmail.com>.
Regarding 3, if we don't make uber jar for connectors, adding jar file in
binary dist. to the classpath will not work, which means that including
connector jar libraries in binary dist. is no effect.
(Someone proposed shading storm-redis because of dependencies... We can
cover it by various way like adding to pom, or using '--artifact' so no
issue at all for now.)
So if we want to go with 3, I'd also like to propose excluding connector
libraries in binary dist. at all.
Others look great to me.
- Jungtaek Lim (HeartSaVioR)
2017년 3월 9일 (목) 오전 6:12, P. Taylor Goetz <pt...@gmail.com>님이 작성:
> I’d like to propose the following to address the binary distribution size:
>
> 1. Finish moving connector examples to the “examples” directory.
> 2. Make all examples source-only in the binary distribution. This will
> mean users will have to build the examples themselves, but that’s typically
> what one does when looking at example code.
> 3. Stop unnecessarily shading connector libraries (i.e. let the user’s pom
> pull in the transitive dependencies when building a topology jar).
>
> If we agree on this approach, I’ll take up the necessary changes for the
> 1.1.0 release.
>
> -Taylor
>
>
>
> > On Mar 8, 2017, at 9:16 AM, Xin Wang <da...@gmail.com> wrote:
> >
> > Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release.
> >
> > - Xin
> >
> > 2017-03-07 9:16 GMT+08:00 Jungtaek Lim <ka...@gmail.com>:
> >
> >> Sorry Roshan but I don't get it.
> >> We're already having separate modules for each of the connectors, and
> even
> >> examples. We also don't have any common modules for external
> connectors, so
> >> those are completely individual.
> >>
> >> As I reported earlier, binary dist. will go up more than 500 MB when we
> >> address STORM-2390(STORM-2249 actually), which the community seemed not
> >> want to.
> >>
> >> Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather
> >> unlink from epic to get out of release, and discuss more.
> >> Any objections?
> >>
> >> - Jungtaek Lim (HeartSaVioR)
> >>
> >> 2017년 3월 4일 (토) 오전 10:02, Roshan Naik <ro...@hortonworks.com>님이 작성:
> >>
> >>> As having a separate module for each of the connectors will cause a lot
> >> of
> >>> bloat … may be good to club them into one.
> >>>
> >>> -roshan
> >>>
> >>>
> >>> On 3/2/17, 10:39 AM, "Sree V" <sr...@yahoo.com.INVALID> wrote:
> >>>
> >>> +1separate main binaries, connectors and samples/examples.
> >>>
> >>> Thanking you.
> >>> With Regards
> >>> Sree
> >>>
> >>> On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
> >>> satish.duggana@gmail.com> wrote:
> >>>
> >>>
> >>> Agree that such huge binaries may not be acceptable. We should
> >> really
> >>> think
> >>> about the options proposed earlier like excluding some of the
> >> external
> >>> connectors from binary and update the documentation respectively.
> >>>
> >>> Thanks,
> >>> Satish.
> >>>
> >>> On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim <ka...@gmail.com>
> >>> wrote:
> >>>
> >>>> Adding my observation to my last mail:
> >>>> I just give STORM-2249 a try (against master branch) and compare
> >>> before vs
> >>>> after.
> >>>> It adds more than 300 MB, and the final archive is more than 550
> >> MB.
> >>> I
> >>>> guess it would be similar for 1.x branch.
> >>>>
> >>>> Before ----------
> >>>> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:41
> >>> apache-storm-2.0.0-SNAPSHOT.
> >>>> pom
> >>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
> >>> apache-storm-2.0.0-SNAPSHOT.
> >>>> pom.asc
> >>>> -rw-r--r-- 1 jlim staff 264M 3 2 19:41
> >>> apache-storm-2.0.0-SNAPSHOT.
> >>>> tar.gz
> >>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
> >>> apache-storm-2.0.0-SNAPSHOT.
> >>>> tar.gz.asc
> >>>> -rw-r--r-- 1 jlim staff 264M 3 2 19:41
> >>> apache-storm-2.0.0-SNAPSHOT.
> >>>> zip
> >>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
> >>> apache-storm-2.0.0-SNAPSHOT.
> >>>> zip.asc
> >>>>
> >>>> After -----------
> >>>> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:46
> >>> apache-storm-2.0.0-SNAPSHOT.
> >>>> pom
> >>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:46
> >>> apache-storm-2.0.0-SNAPSHOT.
> >>>> pom.asc
> >>>> -rw-r--r-- 1 jlim staff 564M 3 2 19:46
> >>> apache-storm-2.0.0-SNAPSHOT.
> >>>> tar.gz
> >>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:46
> >>> apache-storm-2.0.0-SNAPSHOT.
> >>>> tar.gz.asc
> >>>> -rw-r--r-- 1 jlim staff 565M 3 2 19:46
> >>> apache-storm-2.0.0-SNAPSHOT.
> >>>> zip
> >>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:47
> >>> apache-storm-2.0.0-SNAPSHOT.
> >>>> zip.asc
> >>>>
> >>>> While 264M is already a bit huge for me, 564M is not that I can
> >>> accept.
> >>>> (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka
> >>> 0.10.2 is
> >>>> 37M.)
> >>>>
> >>>> Btw, we're including source code of examples, and "mvn clean
> >>> package" will
> >>>> work for every example modules.
> >>>>
> >>>> 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >>>>
> >>>>> I guess it might be good time to think why we add all connectors
> >>> to the
> >>>>> binary distribution.
> >>>>>
> >>>>> Spark and Flink don't include them to binary dist. They even
> >> moved
> >>> some
> >>>> or
> >>>>> most of connectors out of repo, have been maintaining them in
> >>> Apache
> >>>> Bahir.
> >>>>> (Personally this is something I'm in favor of. We have lots of
> >>> connectors
> >>>>> and many of them are outdated - clear example is
> >>> storm-elasticsearch.)
> >>>>>
> >>>>> If we are assuming online then we don't even need to think about
> >>> users
> >>>>> touching binary dist. version of connectors. Users have been
> >>> including
> >>>> them
> >>>>> via build tools' dependency management, or even starting 1.1.0,
> >>> users can
> >>>>> include them via '--artifact' option.
> >>>>>
> >>>>> I also was just one of users for Storm, and I haven't use them
> >>> directly.
> >>>>> How much UX gets worse when we remove connectors to binary dist?
> >>> It only
> >>>>> helps some users who are not connected to the internet, and IMHO
> >>> it's a
> >>>>> rare case.
> >>>>>
> >>>>> I would like to see the opposite approach, removing all
> >> connectors
> >>> (or
> >>>>> just keeping storm-kafka/storm-kafka-client and some more
> >> preferred
> >>>> things)
> >>>>> and its relevant examples from binary dist.
> >>>>>
> >>>>> What do you think about it?
> >>>>>
> >>>>> - Jungtaek Lim (HeartSaVioR)
> >>>>>
> >>>>>
> >>>>> 2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ro...@hortonworks.com>님이
> >>> 작성:
> >>>>>
> >>>>> Once all of the shaded examples are included the size will go up
> >>> further.
> >>>>>
> >>>>> But currently as they are not part of the tar.gz … something else
> >>> is the
> >>>>> culprit for the bloat.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases
> >>> .. of
> >>>>> files that are larger than 4MB.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> @Jungtaek Lim<ma...@hortonworks.com> : I am thinking,
> >>> since the
> >>>>> code for the examples can be easily viewed online … it would be
> >>> valuable
> >>>> to
> >>>>> have the executable topologies made available to the user as part
> >>> of the
> >>>>> binary release … rather have them figure out how to build it
> >>> correctly
> >>>>> before trying them out.
> >>>>>
> >>>>>
> >>>>>
> >>>>> -roshan
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh
> >>> {} \;
> >>>>>
> >>>>> -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33
> >>>> ./examples/storm-starter/
> >>>>> storm-starter-topologies-1.0.3.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30
> >>>>> ./external/flux/flux-examples-1.0.3.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32
> >>>>> ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32
> >>>>> ./external/sql/storm-sql-core/guava-16.0.1.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30
> >>>>> ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-
> >>>> with-dependencies.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33
> >>>>>
> >>> ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33
> >>>>> ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
> >>>>>
> >>>>> -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02
> >>> ./lib/clojure-1.7.0.jar
> >>>>>
> >>>>> -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26
> >>>> ./lib/storm-core-1.0.3.jar
> >>>>>
> >>>>> -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26
> >>>>> ./lib/storm-rename-hack-1.0.3.jar
> >>>>>
> >>>>>
> >>>>>
> >>>>> ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh
> >>> {} \;
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23
> >>> ./examples/storm-pmml-
> >>>>> examples/storm-pmml-examples-1.1.0.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20
> >>>> ./examples/storm-starter/
> >>>>> storm-starter-topologies-1.1.0.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11
> >>>>> ./external/flux/flux-examples-1.1.0.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16
> >>>>> ./external/sql/storm-sql-core/calcite-core-1.11.0.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16
> >>>>> ./external/sql/storm-sql-core/guava-16.0.1.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12
> >>>> ./external/sql/storm-sql-
> >>>>> runtime/calcite-core-1.11.0.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12
> >>>> ./external/sql/storm-sql-
> >>>>> runtime/guava-16.0.1.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18
> >>>>> ./external/storm-druid/storm-druid-1.1.0.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11
> >>>>> ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-
> >>>> with-dependencies.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20
> >>>>>
> >>> ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18
> >>>>> ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02
> >>> ./lib/clojure-1.7.0.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07
> >>>> ./lib/storm-core-1.1.0.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07
> >>>>> ./lib/storm-rename-hack-1.1.0.jar
> >>>>>
> >>>>> -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19
> >>>>> ./toollib/storm-kafka-monitor-1.1.0.jar
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> About STORM-2249, since examples are shading their
> >> dependencies,
> >>>> binary
> >>>>>
> >>>>> dist will grow much bigger. I've left some comments regarding
> >>> that.
> >>>>>
> >>>>> Btw, I have another view of this. Showing example codes is
> >> more
> >>>>> important
> >>>>>
> >>>>> than just let users execute some topologies. That's what
> >> example
> >>>>> modules
> >>>>>
> >>>>> are for. We need to include source as well. If we need to pick
> >>> one,
> >>>>> source
> >>>>>
> >>>>> code would be better.
> >>>>>
> >>>>>
> >>>>>
> >>>>> STORM-2343 seems better to add to 1.1.0. I just am not enough
> >>>> familiar
> >>>>> with
> >>>>>
> >>>>> storm-kafka-client so not sure I can review that, but I'll try
> >>> to. I
> >>>>> feel
> >>>>>
> >>>>> it's not that make release dragged. Let's add to 1.1.0 epic.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Let's make minimum merge before another RC vote. Addressing
> >>>> STORM-2389
> >>>>> (and
> >>>>>
> >>>>> maybe STORM-2343) is enough for me. Others are not that
> >>> critical.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Jungtaek Lim (HeartSaVioR)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <
> >>>>> hlouro@hortonworks.com>
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Roshan, does this PR<
> >>> https://github.com/apache/storm/pull/1831>
> >>>> and
> >>>>> JIRA<
> >>>>>
> >>>>>> https://issues.apache.org/jira/browse/STORM-2249> address
> >> the
> >>>>> missing
> >>>>>
> >>>>>> jars problem that you mentioned. I had created it in
> >> December
> >>> 2016,
> >>>>> but
> >>>>>
> >>>>>> there is an ongoing discussion if we should indeed put the
> >>> jars in
> >>>>> the
> >>>>>
> >>>>>> examples location or not.
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> On a different note, this storm-kafka-client/KafkaSpout PR<
> >>>>>
> >>>>>> https://github.com/apache/storm/pull/1924> fixes a bug with
> >>> the
> >>>>> number of
> >>>>>
> >>>>>> uncommitted offsets that is quite important. It is not a
> >>> blocker,
> >>>>> but it is
> >>>>>
> >>>>>> quite critical. I am going to do one last pass reviewing
> >>> today. It
> >>>>> would be
> >>>>>
> >>>>>> good if we could have this PR included with the release. Can
> >>> anyone
> >>>>> else
> >>>>>
> >>>>>> review it as well ?
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> Thanks,
> >>>>>
> >>>>>> Hugo
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <
> >>> ptgoetz@gmail.com
> >>>>> <mailto:ptgo
> >>>>>
> >>>>>> etz@gmail.com>> wrote:
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> Yeah, I don’t think the file size is a killer/blocker. It’s
> >>> largely
> >>>>> due to
> >>>>>
> >>>>>> shaded examples, etc. But it’s something to keep an eye on.
> >>> Our
> >>>>> binary
> >>>>>
> >>>>>> releases shouldn’t have to be that big.
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> -Taylor
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> On Mar 1, 2017, at 12:09 PM, Roshan Naik <
> >>> roshan@hortonworks.com<
> >>>>> mailto:
> >>>>>
> >>>>>> roshan@hortonworks.com>> wrote:
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> Have filed Jiras so for the 3 issues mentioned. Not sure if
> >>> we need
> >>>>> a JIRA
> >>>>>
> >>>>>> for the file size getting bloated by that much.
> >>>>>
> >>>>>> Somebody better familiar with the matter may want to take
> >>> about
> >>>> that?
> >>>>>
> >>>>>> -roshan
> >>>>>
> >>>>>>
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com
> >>> <mailto:
> >>>> ptgo
> >>>>>
> >>>>>> etz@gmail.com>> wrote:
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> Thanks for bringing these up Roshan. Feel free to file JIRA
> >>>>> tickets for
> >>>>>
> >>>>>> these issues and assign the “Release Apache Storm 1.1.0”
> >> epic
> >>> so
> >>>>> they can
> >>>>>
> >>>>>> be tracked for this release.
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> -Taylor
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> On Mar 1, 2017, at 9:27 AM, Roshan Naik <
> >>> roshan@hortonworks.com<
> >>>>> mailto:
> >>>>>
> >>>>>> roshan@hortonworks.com>> wrote:
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> Found these additional issues:
> >>>>>
> >>>>>>
> >>>>>
> >>>>>>
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> 1- BUG: Even if topology.eventlogger.executors=0, the
> >>>> event_logger
> >>>>> bolt
> >>>>>
> >>>>>> is instantiated … previously observed to cause ~10%
> >>> degradation in
> >>>>> perf
> >>>>>
> >>>>>> even with logging disabled.
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> 2- Missing Jars: The storm-*-examples jars are missing in
> >> the
> >>>> binary
> >>>>>
> >>>>>> distro (other than a storm-pmml-examples.jar,
> >>>> storm-jms-examples.jar
> >>>>> &
> >>>>>
> >>>>>> flux-examples.jar).
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> 3- Minor: HdfsSpoutTopology example has not been moved into
> >>>>>
> >>>>>> storm-hdfs-examples from storm-starter
> >>>>>
> >>>>>>
> >>>>>
> >>>>>>
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> Another side observation … v1.0.3 tar.gz downloadable was
> >>> 190MB.
> >>>>> This
> >>>>>
> >>>>>> v1.1.0 tar.gz downloadable is 297MB !! …. even though some
> >>> of the
> >>>>> example
> >>>>>
> >>>>>> topologies didn’t make it.
> >>>>>
> >>>>>>
> >>>>>
> >>>>>>
> >>>>>
> >>>>>>
> >>>>>
> >>>>>> -roshan
> >>>>>
> >>>>>>
> >>>>>
> >>>>>>
> >>>>>
> >>>>>>
> >>>>>
> >>>>>>
> >>>>>
> >>>>>>
> >>>>>
> >>>>>>
> >>>>>
> >>>>>>
> >>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Name : Jungtaek Lim
> >>>>>
> >>>>> Blog : http://medium.com/@heartsavior
> >>>>>
> >>>>> Twitter : http://twitter.com/heartsavior
> >>>>>
> >>>>> LinkedIn : http://www.linkedin.com/in/heartsavior
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>
>
>
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
I’d like to propose the following to address the binary distribution size:
1. Finish moving connector examples to the “examples” directory.
2. Make all examples source-only in the binary distribution. This will mean users will have to build the examples themselves, but that’s typically what one does when looking at example code.
3. Stop unnecessarily shading connector libraries (i.e. let the user’s pom pull in the transitive dependencies when building a topology jar).
If we agree on this approach, I’ll take up the necessary changes for the 1.1.0 release.
-Taylor
> On Mar 8, 2017, at 9:16 AM, Xin Wang <da...@gmail.com> wrote:
>
> Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release.
>
> - Xin
>
> 2017-03-07 9:16 GMT+08:00 Jungtaek Lim <ka...@gmail.com>:
>
>> Sorry Roshan but I don't get it.
>> We're already having separate modules for each of the connectors, and even
>> examples. We also don't have any common modules for external connectors, so
>> those are completely individual.
>>
>> As I reported earlier, binary dist. will go up more than 500 MB when we
>> address STORM-2390(STORM-2249 actually), which the community seemed not
>> want to.
>>
>> Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather
>> unlink from epic to get out of release, and discuss more.
>> Any objections?
>>
>> - Jungtaek Lim (HeartSaVioR)
>>
>> 2017년 3월 4일 (토) 오전 10:02, Roshan Naik <ro...@hortonworks.com>님이 작성:
>>
>>> As having a separate module for each of the connectors will cause a lot
>> of
>>> bloat … may be good to club them into one.
>>>
>>> -roshan
>>>
>>>
>>> On 3/2/17, 10:39 AM, "Sree V" <sr...@yahoo.com.INVALID> wrote:
>>>
>>> +1separate main binaries, connectors and samples/examples.
>>>
>>> Thanking you.
>>> With Regards
>>> Sree
>>>
>>> On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
>>> satish.duggana@gmail.com> wrote:
>>>
>>>
>>> Agree that such huge binaries may not be acceptable. We should
>> really
>>> think
>>> about the options proposed earlier like excluding some of the
>> external
>>> connectors from binary and update the documentation respectively.
>>>
>>> Thanks,
>>> Satish.
>>>
>>> On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim <ka...@gmail.com>
>>> wrote:
>>>
>>>> Adding my observation to my last mail:
>>>> I just give STORM-2249 a try (against master branch) and compare
>>> before vs
>>>> after.
>>>> It adds more than 300 MB, and the final archive is more than 550
>> MB.
>>> I
>>>> guess it would be similar for 1.x branch.
>>>>
>>>> Before ----------
>>>> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:41
>>> apache-storm-2.0.0-SNAPSHOT.
>>>> pom
>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
>>> apache-storm-2.0.0-SNAPSHOT.
>>>> pom.asc
>>>> -rw-r--r-- 1 jlim staff 264M 3 2 19:41
>>> apache-storm-2.0.0-SNAPSHOT.
>>>> tar.gz
>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
>>> apache-storm-2.0.0-SNAPSHOT.
>>>> tar.gz.asc
>>>> -rw-r--r-- 1 jlim staff 264M 3 2 19:41
>>> apache-storm-2.0.0-SNAPSHOT.
>>>> zip
>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41
>>> apache-storm-2.0.0-SNAPSHOT.
>>>> zip.asc
>>>>
>>>> After -----------
>>>> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:46
>>> apache-storm-2.0.0-SNAPSHOT.
>>>> pom
>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:46
>>> apache-storm-2.0.0-SNAPSHOT.
>>>> pom.asc
>>>> -rw-r--r-- 1 jlim staff 564M 3 2 19:46
>>> apache-storm-2.0.0-SNAPSHOT.
>>>> tar.gz
>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:46
>>> apache-storm-2.0.0-SNAPSHOT.
>>>> tar.gz.asc
>>>> -rw-r--r-- 1 jlim staff 565M 3 2 19:46
>>> apache-storm-2.0.0-SNAPSHOT.
>>>> zip
>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:47
>>> apache-storm-2.0.0-SNAPSHOT.
>>>> zip.asc
>>>>
>>>> While 264M is already a bit huge for me, 564M is not that I can
>>> accept.
>>>> (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka
>>> 0.10.2 is
>>>> 37M.)
>>>>
>>>> Btw, we're including source code of examples, and "mvn clean
>>> package" will
>>>> work for every example modules.
>>>>
>>>> 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>>>
>>>>> I guess it might be good time to think why we add all connectors
>>> to the
>>>>> binary distribution.
>>>>>
>>>>> Spark and Flink don't include them to binary dist. They even
>> moved
>>> some
>>>> or
>>>>> most of connectors out of repo, have been maintaining them in
>>> Apache
>>>> Bahir.
>>>>> (Personally this is something I'm in favor of. We have lots of
>>> connectors
>>>>> and many of them are outdated - clear example is
>>> storm-elasticsearch.)
>>>>>
>>>>> If we are assuming online then we don't even need to think about
>>> users
>>>>> touching binary dist. version of connectors. Users have been
>>> including
>>>> them
>>>>> via build tools' dependency management, or even starting 1.1.0,
>>> users can
>>>>> include them via '--artifact' option.
>>>>>
>>>>> I also was just one of users for Storm, and I haven't use them
>>> directly.
>>>>> How much UX gets worse when we remove connectors to binary dist?
>>> It only
>>>>> helps some users who are not connected to the internet, and IMHO
>>> it's a
>>>>> rare case.
>>>>>
>>>>> I would like to see the opposite approach, removing all
>> connectors
>>> (or
>>>>> just keeping storm-kafka/storm-kafka-client and some more
>> preferred
>>>> things)
>>>>> and its relevant examples from binary dist.
>>>>>
>>>>> What do you think about it?
>>>>>
>>>>> - Jungtaek Lim (HeartSaVioR)
>>>>>
>>>>>
>>>>> 2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ro...@hortonworks.com>님이
>>> 작성:
>>>>>
>>>>> Once all of the shaded examples are included the size will go up
>>> further.
>>>>>
>>>>> But currently as they are not part of the tar.gz … something else
>>> is the
>>>>> culprit for the bloat.
>>>>>
>>>>>
>>>>>
>>>>> Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases
>>> .. of
>>>>> files that are larger than 4MB.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> @Jungtaek Lim<ma...@hortonworks.com> : I am thinking,
>>> since the
>>>>> code for the examples can be easily viewed online … it would be
>>> valuable
>>>> to
>>>>> have the executable topologies made available to the user as part
>>> of the
>>>>> binary release … rather have them figure out how to build it
>>> correctly
>>>>> before trying them out.
>>>>>
>>>>>
>>>>>
>>>>> -roshan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh
>>> {} \;
>>>>>
>>>>> -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33
>>>> ./examples/storm-starter/
>>>>> storm-starter-topologies-1.0.3.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30
>>>>> ./external/flux/flux-examples-1.0.3.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32
>>>>> ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32
>>>>> ./external/sql/storm-sql-core/guava-16.0.1.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30
>>>>> ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-
>>>> with-dependencies.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33
>>>>>
>>> ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33
>>>>> ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
>>>>>
>>>>> -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02
>>> ./lib/clojure-1.7.0.jar
>>>>>
>>>>> -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26
>>>> ./lib/storm-core-1.0.3.jar
>>>>>
>>>>> -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26
>>>>> ./lib/storm-rename-hack-1.0.3.jar
>>>>>
>>>>>
>>>>>
>>>>> ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh
>>> {} \;
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23
>>> ./examples/storm-pmml-
>>>>> examples/storm-pmml-examples-1.1.0.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20
>>>> ./examples/storm-starter/
>>>>> storm-starter-topologies-1.1.0.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11
>>>>> ./external/flux/flux-examples-1.1.0.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16
>>>>> ./external/sql/storm-sql-core/calcite-core-1.11.0.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16
>>>>> ./external/sql/storm-sql-core/guava-16.0.1.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12
>>>> ./external/sql/storm-sql-
>>>>> runtime/calcite-core-1.11.0.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12
>>>> ./external/sql/storm-sql-
>>>>> runtime/guava-16.0.1.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18
>>>>> ./external/storm-druid/storm-druid-1.1.0.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11
>>>>> ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-
>>>> with-dependencies.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20
>>>>>
>>> ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18
>>>>> ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02
>>> ./lib/clojure-1.7.0.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07
>>>> ./lib/storm-core-1.1.0.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07
>>>>> ./lib/storm-rename-hack-1.1.0.jar
>>>>>
>>>>> -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19
>>>>> ./toollib/storm-kafka-monitor-1.1.0.jar
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> About STORM-2249, since examples are shading their
>> dependencies,
>>>> binary
>>>>>
>>>>> dist will grow much bigger. I've left some comments regarding
>>> that.
>>>>>
>>>>> Btw, I have another view of this. Showing example codes is
>> more
>>>>> important
>>>>>
>>>>> than just let users execute some topologies. That's what
>> example
>>>>> modules
>>>>>
>>>>> are for. We need to include source as well. If we need to pick
>>> one,
>>>>> source
>>>>>
>>>>> code would be better.
>>>>>
>>>>>
>>>>>
>>>>> STORM-2343 seems better to add to 1.1.0. I just am not enough
>>>> familiar
>>>>> with
>>>>>
>>>>> storm-kafka-client so not sure I can review that, but I'll try
>>> to. I
>>>>> feel
>>>>>
>>>>> it's not that make release dragged. Let's add to 1.1.0 epic.
>>>>>
>>>>>
>>>>>
>>>>> Let's make minimum merge before another RC vote. Addressing
>>>> STORM-2389
>>>>> (and
>>>>>
>>>>> maybe STORM-2343) is enough for me. Others are not that
>>> critical.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <
>>>>> hlouro@hortonworks.com>
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Roshan, does this PR<
>>> https://github.com/apache/storm/pull/1831>
>>>> and
>>>>> JIRA<
>>>>>
>>>>>> https://issues.apache.org/jira/browse/STORM-2249> address
>> the
>>>>> missing
>>>>>
>>>>>> jars problem that you mentioned. I had created it in
>> December
>>> 2016,
>>>>> but
>>>>>
>>>>>> there is an ongoing discussion if we should indeed put the
>>> jars in
>>>>> the
>>>>>
>>>>>> examples location or not.
>>>>>
>>>>>>
>>>>>
>>>>>> On a different note, this storm-kafka-client/KafkaSpout PR<
>>>>>
>>>>>> https://github.com/apache/storm/pull/1924> fixes a bug with
>>> the
>>>>> number of
>>>>>
>>>>>> uncommitted offsets that is quite important. It is not a
>>> blocker,
>>>>> but it is
>>>>>
>>>>>> quite critical. I am going to do one last pass reviewing
>>> today. It
>>>>> would be
>>>>>
>>>>>> good if we could have this PR included with the release. Can
>>> anyone
>>>>> else
>>>>>
>>>>>> review it as well ?
>>>>>
>>>>>>
>>>>>
>>>>>> Thanks,
>>>>>
>>>>>> Hugo
>>>>>
>>>>>>
>>>>>
>>>>>> On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <
>>> ptgoetz@gmail.com
>>>>> <mailto:ptgo
>>>>>
>>>>>> etz@gmail.com>> wrote:
>>>>>
>>>>>>
>>>>>
>>>>>> Yeah, I don’t think the file size is a killer/blocker. It’s
>>> largely
>>>>> due to
>>>>>
>>>>>> shaded examples, etc. But it’s something to keep an eye on.
>>> Our
>>>>> binary
>>>>>
>>>>>> releases shouldn’t have to be that big.
>>>>>
>>>>>>
>>>>>
>>>>>> -Taylor
>>>>>
>>>>>>
>>>>>
>>>>>> On Mar 1, 2017, at 12:09 PM, Roshan Naik <
>>> roshan@hortonworks.com<
>>>>> mailto:
>>>>>
>>>>>> roshan@hortonworks.com>> wrote:
>>>>>
>>>>>>
>>>>>
>>>>>> Have filed Jiras so for the 3 issues mentioned. Not sure if
>>> we need
>>>>> a JIRA
>>>>>
>>>>>> for the file size getting bloated by that much.
>>>>>
>>>>>> Somebody better familiar with the matter may want to take
>>> about
>>>> that?
>>>>>
>>>>>> -roshan
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>> On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com
>>> <mailto:
>>>> ptgo
>>>>>
>>>>>> etz@gmail.com>> wrote:
>>>>>
>>>>>>
>>>>>
>>>>>> Thanks for bringing these up Roshan. Feel free to file JIRA
>>>>> tickets for
>>>>>
>>>>>> these issues and assign the “Release Apache Storm 1.1.0”
>> epic
>>> so
>>>>> they can
>>>>>
>>>>>> be tracked for this release.
>>>>>
>>>>>>
>>>>>
>>>>>> -Taylor
>>>>>
>>>>>>
>>>>>
>>>>>> On Mar 1, 2017, at 9:27 AM, Roshan Naik <
>>> roshan@hortonworks.com<
>>>>> mailto:
>>>>>
>>>>>> roshan@hortonworks.com>> wrote:
>>>>>
>>>>>>
>>>>>
>>>>>> Found these additional issues:
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>> 1- BUG: Even if topology.eventlogger.executors=0, the
>>>> event_logger
>>>>> bolt
>>>>>
>>>>>> is instantiated … previously observed to cause ~10%
>>> degradation in
>>>>> perf
>>>>>
>>>>>> even with logging disabled.
>>>>>
>>>>>>
>>>>>
>>>>>> 2- Missing Jars: The storm-*-examples jars are missing in
>> the
>>>> binary
>>>>>
>>>>>> distro (other than a storm-pmml-examples.jar,
>>>> storm-jms-examples.jar
>>>>> &
>>>>>
>>>>>> flux-examples.jar).
>>>>>
>>>>>>
>>>>>
>>>>>> 3- Minor: HdfsSpoutTopology example has not been moved into
>>>>>
>>>>>> storm-hdfs-examples from storm-starter
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>> Another side observation … v1.0.3 tar.gz downloadable was
>>> 190MB.
>>>>> This
>>>>>
>>>>>> v1.1.0 tar.gz downloadable is 297MB !! …. even though some
>>> of the
>>>>> example
>>>>>
>>>>>> topologies didn’t make it.
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>> -roshan
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Name : Jungtaek Lim
>>>>>
>>>>> Blog : http://medium.com/@heartsavior
>>>>>
>>>>> Twitter : http://twitter.com/heartsavior
>>>>>
>>>>> LinkedIn : http://www.linkedin.com/in/heartsavior
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>>
>>
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Xin Wang <da...@gmail.com>.
Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release.
- Xin
2017-03-07 9:16 GMT+08:00 Jungtaek Lim <ka...@gmail.com>:
> Sorry Roshan but I don't get it.
> We're already having separate modules for each of the connectors, and even
> examples. We also don't have any common modules for external connectors, so
> those are completely individual.
>
> As I reported earlier, binary dist. will go up more than 500 MB when we
> address STORM-2390(STORM-2249 actually), which the community seemed not
> want to.
>
> Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather
> unlink from epic to get out of release, and discuss more.
> Any objections?
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 3월 4일 (토) 오전 10:02, Roshan Naik <ro...@hortonworks.com>님이 작성:
>
> > As having a separate module for each of the connectors will cause a lot
> of
> > bloat … may be good to club them into one.
> >
> > -roshan
> >
> >
> > On 3/2/17, 10:39 AM, "Sree V" <sr...@yahoo.com.INVALID> wrote:
> >
> > +1separate main binaries, connectors and samples/examples.
> >
> > Thanking you.
> > With Regards
> > Sree
> >
> > On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
> > satish.duggana@gmail.com> wrote:
> >
> >
> > Agree that such huge binaries may not be acceptable. We should
> really
> > think
> > about the options proposed earlier like excluding some of the
> external
> > connectors from binary and update the documentation respectively.
> >
> > Thanks,
> > Satish.
> >
> > On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim <ka...@gmail.com>
> > wrote:
> >
> > > Adding my observation to my last mail:
> > > I just give STORM-2249 a try (against master branch) and compare
> > before vs
> > > after.
> > > It adds more than 300 MB, and the final archive is more than 550
> MB.
> > I
> > > guess it would be similar for 1.x branch.
> > >
> > > Before ----------
> > > -rw-r--r-- 1 jlim staff 3.3K 3 2 19:41
> > apache-storm-2.0.0-SNAPSHOT.
> > > pom
> > > -rw-r--r-- 1 jlim staff 473B 3 2 19:41
> > apache-storm-2.0.0-SNAPSHOT.
> > > pom.asc
> > > -rw-r--r-- 1 jlim staff 264M 3 2 19:41
> > apache-storm-2.0.0-SNAPSHOT.
> > > tar.gz
> > > -rw-r--r-- 1 jlim staff 473B 3 2 19:41
> > apache-storm-2.0.0-SNAPSHOT.
> > > tar.gz.asc
> > > -rw-r--r-- 1 jlim staff 264M 3 2 19:41
> > apache-storm-2.0.0-SNAPSHOT.
> > > zip
> > > -rw-r--r-- 1 jlim staff 473B 3 2 19:41
> > apache-storm-2.0.0-SNAPSHOT.
> > > zip.asc
> > >
> > > After -----------
> > > -rw-r--r-- 1 jlim staff 3.3K 3 2 19:46
> > apache-storm-2.0.0-SNAPSHOT.
> > > pom
> > > -rw-r--r-- 1 jlim staff 473B 3 2 19:46
> > apache-storm-2.0.0-SNAPSHOT.
> > > pom.asc
> > > -rw-r--r-- 1 jlim staff 564M 3 2 19:46
> > apache-storm-2.0.0-SNAPSHOT.
> > > tar.gz
> > > -rw-r--r-- 1 jlim staff 473B 3 2 19:46
> > apache-storm-2.0.0-SNAPSHOT.
> > > tar.gz.asc
> > > -rw-r--r-- 1 jlim staff 565M 3 2 19:46
> > apache-storm-2.0.0-SNAPSHOT.
> > > zip
> > > -rw-r--r-- 1 jlim staff 473B 3 2 19:47
> > apache-storm-2.0.0-SNAPSHOT.
> > > zip.asc
> > >
> > > While 264M is already a bit huge for me, 564M is not that I can
> > accept.
> > > (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka
> > 0.10.2 is
> > > 37M.)
> > >
> > > Btw, we're including source code of examples, and "mvn clean
> > package" will
> > > work for every example modules.
> > >
> > > 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim <ka...@gmail.com>님이 작성:
> > >
> > > > I guess it might be good time to think why we add all connectors
> > to the
> > > > binary distribution.
> > > >
> > > > Spark and Flink don't include them to binary dist. They even
> moved
> > some
> > > or
> > > > most of connectors out of repo, have been maintaining them in
> > Apache
> > > Bahir.
> > > > (Personally this is something I'm in favor of. We have lots of
> > connectors
> > > > and many of them are outdated - clear example is
> > storm-elasticsearch.)
> > > >
> > > > If we are assuming online then we don't even need to think about
> > users
> > > > touching binary dist. version of connectors. Users have been
> > including
> > > them
> > > > via build tools' dependency management, or even starting 1.1.0,
> > users can
> > > > include them via '--artifact' option.
> > > >
> > > > I also was just one of users for Storm, and I haven't use them
> > directly.
> > > > How much UX gets worse when we remove connectors to binary dist?
> > It only
> > > > helps some users who are not connected to the internet, and IMHO
> > it's a
> > > > rare case.
> > > >
> > > > I would like to see the opposite approach, removing all
> connectors
> > (or
> > > > just keeping storm-kafka/storm-kafka-client and some more
> preferred
> > > things)
> > > > and its relevant examples from binary dist.
> > > >
> > > > What do you think about it?
> > > >
> > > > - Jungtaek Lim (HeartSaVioR)
> > > >
> > > >
> > > > 2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ro...@hortonworks.com>님이
> > 작성:
> > > >
> > > > Once all of the shaded examples are included the size will go up
> > further.
> > > >
> > > > But currently as they are not part of the tar.gz … something else
> > is the
> > > > culprit for the bloat.
> > > >
> > > >
> > > >
> > > > Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases
> > .. of
> > > > files that are larger than 4MB.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > @Jungtaek Lim<ma...@hortonworks.com> : I am thinking,
> > since the
> > > > code for the examples can be easily viewed online … it would be
> > valuable
> > > to
> > > > have the executable topologies made available to the user as part
> > of the
> > > > binary release … rather have them figure out how to build it
> > correctly
> > > > before trying them out.
> > > >
> > > >
> > > >
> > > > -roshan
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh
> > {} \;
> > > >
> > > > -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33
> > > ./examples/storm-starter/
> > > > storm-starter-topologies-1.0.3.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30
> > > > ./external/flux/flux-examples-1.0.3.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32
> > > > ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32
> > > > ./external/sql/storm-sql-core/guava-16.0.1.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30
> > > > ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-
> > > with-dependencies.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33
> > > >
> > ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33
> > > > ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
> > > >
> > > > -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02
> > ./lib/clojure-1.7.0.jar
> > > >
> > > > -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26
> > > ./lib/storm-core-1.0.3.jar
> > > >
> > > > -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26
> > > > ./lib/storm-rename-hack-1.0.3.jar
> > > >
> > > >
> > > >
> > > > ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh
> > {} \;
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23
> > ./examples/storm-pmml-
> > > > examples/storm-pmml-examples-1.1.0.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20
> > > ./examples/storm-starter/
> > > > storm-starter-topologies-1.1.0.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11
> > > > ./external/flux/flux-examples-1.1.0.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16
> > > > ./external/sql/storm-sql-core/calcite-core-1.11.0.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16
> > > > ./external/sql/storm-sql-core/guava-16.0.1.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12
> > > ./external/sql/storm-sql-
> > > > runtime/calcite-core-1.11.0.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12
> > > ./external/sql/storm-sql-
> > > > runtime/guava-16.0.1.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18
> > > > ./external/storm-druid/storm-druid-1.1.0.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11
> > > > ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-
> > > with-dependencies.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20
> > > >
> > ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18
> > > > ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02
> > ./lib/clojure-1.7.0.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07
> > > ./lib/storm-core-1.1.0.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07
> > > > ./lib/storm-rename-hack-1.1.0.jar
> > > >
> > > > -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19
> > > > ./toollib/storm-kafka-monitor-1.1.0.jar
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> > > >
> > > >
> > > >
> > > > About STORM-2249, since examples are shading their
> dependencies,
> > > binary
> > > >
> > > > dist will grow much bigger. I've left some comments regarding
> > that.
> > > >
> > > > Btw, I have another view of this. Showing example codes is
> more
> > > > important
> > > >
> > > > than just let users execute some topologies. That's what
> example
> > > > modules
> > > >
> > > > are for. We need to include source as well. If we need to pick
> > one,
> > > > source
> > > >
> > > > code would be better.
> > > >
> > > >
> > > >
> > > > STORM-2343 seems better to add to 1.1.0. I just am not enough
> > > familiar
> > > > with
> > > >
> > > > storm-kafka-client so not sure I can review that, but I'll try
> > to. I
> > > > feel
> > > >
> > > > it's not that make release dragged. Let's add to 1.1.0 epic.
> > > >
> > > >
> > > >
> > > > Let's make minimum merge before another RC vote. Addressing
> > > STORM-2389
> > > > (and
> > > >
> > > > maybe STORM-2343) is enough for me. Others are not that
> > critical.
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Jungtaek Lim (HeartSaVioR)
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <
> > > > hlouro@hortonworks.com>
> > > >
> > > > wrote:
> > > >
> > > >
> > > >
> > > > > Roshan, does this PR<
> > https://github.com/apache/storm/pull/1831>
> > > and
> > > > JIRA<
> > > >
> > > > > https://issues.apache.org/jira/browse/STORM-2249> address
> the
> > > > missing
> > > >
> > > > > jars problem that you mentioned. I had created it in
> December
> > 2016,
> > > > but
> > > >
> > > > > there is an ongoing discussion if we should indeed put the
> > jars in
> > > > the
> > > >
> > > > > examples location or not.
> > > >
> > > > >
> > > >
> > > > > On a different note, this storm-kafka-client/KafkaSpout PR<
> > > >
> > > > > https://github.com/apache/storm/pull/1924> fixes a bug with
> > the
> > > > number of
> > > >
> > > > > uncommitted offsets that is quite important. It is not a
> > blocker,
> > > > but it is
> > > >
> > > > > quite critical. I am going to do one last pass reviewing
> > today. It
> > > > would be
> > > >
> > > > > good if we could have this PR included with the release. Can
> > anyone
> > > > else
> > > >
> > > > > review it as well ?
> > > >
> > > > >
> > > >
> > > > > Thanks,
> > > >
> > > > > Hugo
> > > >
> > > > >
> > > >
> > > > > On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <
> > ptgoetz@gmail.com
> > > > <mailto:ptgo
> > > >
> > > > > etz@gmail.com>> wrote:
> > > >
> > > > >
> > > >
> > > > > Yeah, I don’t think the file size is a killer/blocker. It’s
> > largely
> > > > due to
> > > >
> > > > > shaded examples, etc. But it’s something to keep an eye on.
> > Our
> > > > binary
> > > >
> > > > > releases shouldn’t have to be that big.
> > > >
> > > > >
> > > >
> > > > > -Taylor
> > > >
> > > > >
> > > >
> > > > > On Mar 1, 2017, at 12:09 PM, Roshan Naik <
> > roshan@hortonworks.com<
> > > > mailto:
> > > >
> > > > > roshan@hortonworks.com>> wrote:
> > > >
> > > > >
> > > >
> > > > > Have filed Jiras so for the 3 issues mentioned. Not sure if
> > we need
> > > > a JIRA
> > > >
> > > > > for the file size getting bloated by that much.
> > > >
> > > > > Somebody better familiar with the matter may want to take
> > about
> > > that?
> > > >
> > > > > -roshan
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > > On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com
> > <mailto:
> > > ptgo
> > > >
> > > > > etz@gmail.com>> wrote:
> > > >
> > > > >
> > > >
> > > > > Thanks for bringing these up Roshan. Feel free to file JIRA
> > > > tickets for
> > > >
> > > > > these issues and assign the “Release Apache Storm 1.1.0”
> epic
> > so
> > > > they can
> > > >
> > > > > be tracked for this release.
> > > >
> > > > >
> > > >
> > > > > -Taylor
> > > >
> > > > >
> > > >
> > > > > On Mar 1, 2017, at 9:27 AM, Roshan Naik <
> > roshan@hortonworks.com<
> > > > mailto:
> > > >
> > > > > roshan@hortonworks.com>> wrote:
> > > >
> > > > >
> > > >
> > > > > Found these additional issues:
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > > 1- BUG: Even if topology.eventlogger.executors=0, the
> > > event_logger
> > > > bolt
> > > >
> > > > > is instantiated … previously observed to cause ~10%
> > degradation in
> > > > perf
> > > >
> > > > > even with logging disabled.
> > > >
> > > > >
> > > >
> > > > > 2- Missing Jars: The storm-*-examples jars are missing in
> the
> > > binary
> > > >
> > > > > distro (other than a storm-pmml-examples.jar,
> > > storm-jms-examples.jar
> > > > &
> > > >
> > > > > flux-examples.jar).
> > > >
> > > > >
> > > >
> > > > > 3- Minor: HdfsSpoutTopology example has not been moved into
> > > >
> > > > > storm-hdfs-examples from storm-starter
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > > Another side observation … v1.0.3 tar.gz downloadable was
> > 190MB.
> > > > This
> > > >
> > > > > v1.1.0 tar.gz downloadable is 297MB !! …. even though some
> > of the
> > > > example
> > > >
> > > > > topologies didn’t make it.
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > > -roshan
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Name : Jungtaek Lim
> > > >
> > > > Blog : http://medium.com/@heartsavior
> > > >
> > > > Twitter : http://twitter.com/heartsavior
> > > >
> > > > LinkedIn : http://www.linkedin.com/in/heartsavior
> > > >
> > > >
> > > >
> > >
> >
> >
> >
> >
>
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Jungtaek Lim <ka...@gmail.com>.
Sorry Roshan but I don't get it.
We're already having separate modules for each of the connectors, and even
examples. We also don't have any common modules for external connectors, so
those are completely individual.
As I reported earlier, binary dist. will go up more than 500 MB when we
address STORM-2390(STORM-2249 actually), which the community seemed not
want to.
Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather
unlink from epic to get out of release, and discuss more.
Any objections?
- Jungtaek Lim (HeartSaVioR)
2017년 3월 4일 (토) 오전 10:02, Roshan Naik <ro...@hortonworks.com>님이 작성:
> As having a separate module for each of the connectors will cause a lot of
> bloat … may be good to club them into one.
>
> -roshan
>
>
> On 3/2/17, 10:39 AM, "Sree V" <sr...@yahoo.com.INVALID> wrote:
>
> +1separate main binaries, connectors and samples/examples.
>
> Thanking you.
> With Regards
> Sree
>
> On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
> satish.duggana@gmail.com> wrote:
>
>
> Agree that such huge binaries may not be acceptable. We should really
> think
> about the options proposed earlier like excluding some of the external
> connectors from binary and update the documentation respectively.
>
> Thanks,
> Satish.
>
> On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim <ka...@gmail.com>
> wrote:
>
> > Adding my observation to my last mail:
> > I just give STORM-2249 a try (against master branch) and compare
> before vs
> > after.
> > It adds more than 300 MB, and the final archive is more than 550 MB.
> I
> > guess it would be similar for 1.x branch.
> >
> > Before ----------
> > -rw-r--r-- 1 jlim staff 3.3K 3 2 19:41
> apache-storm-2.0.0-SNAPSHOT.
> > pom
> > -rw-r--r-- 1 jlim staff 473B 3 2 19:41
> apache-storm-2.0.0-SNAPSHOT.
> > pom.asc
> > -rw-r--r-- 1 jlim staff 264M 3 2 19:41
> apache-storm-2.0.0-SNAPSHOT.
> > tar.gz
> > -rw-r--r-- 1 jlim staff 473B 3 2 19:41
> apache-storm-2.0.0-SNAPSHOT.
> > tar.gz.asc
> > -rw-r--r-- 1 jlim staff 264M 3 2 19:41
> apache-storm-2.0.0-SNAPSHOT.
> > zip
> > -rw-r--r-- 1 jlim staff 473B 3 2 19:41
> apache-storm-2.0.0-SNAPSHOT.
> > zip.asc
> >
> > After -----------
> > -rw-r--r-- 1 jlim staff 3.3K 3 2 19:46
> apache-storm-2.0.0-SNAPSHOT.
> > pom
> > -rw-r--r-- 1 jlim staff 473B 3 2 19:46
> apache-storm-2.0.0-SNAPSHOT.
> > pom.asc
> > -rw-r--r-- 1 jlim staff 564M 3 2 19:46
> apache-storm-2.0.0-SNAPSHOT.
> > tar.gz
> > -rw-r--r-- 1 jlim staff 473B 3 2 19:46
> apache-storm-2.0.0-SNAPSHOT.
> > tar.gz.asc
> > -rw-r--r-- 1 jlim staff 565M 3 2 19:46
> apache-storm-2.0.0-SNAPSHOT.
> > zip
> > -rw-r--r-- 1 jlim staff 473B 3 2 19:47
> apache-storm-2.0.0-SNAPSHOT.
> > zip.asc
> >
> > While 264M is already a bit huge for me, 564M is not that I can
> accept.
> > (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka
> 0.10.2 is
> > 37M.)
> >
> > Btw, we're including source code of examples, and "mvn clean
> package" will
> > work for every example modules.
> >
> > 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >
> > > I guess it might be good time to think why we add all connectors
> to the
> > > binary distribution.
> > >
> > > Spark and Flink don't include them to binary dist. They even moved
> some
> > or
> > > most of connectors out of repo, have been maintaining them in
> Apache
> > Bahir.
> > > (Personally this is something I'm in favor of. We have lots of
> connectors
> > > and many of them are outdated - clear example is
> storm-elasticsearch.)
> > >
> > > If we are assuming online then we don't even need to think about
> users
> > > touching binary dist. version of connectors. Users have been
> including
> > them
> > > via build tools' dependency management, or even starting 1.1.0,
> users can
> > > include them via '--artifact' option.
> > >
> > > I also was just one of users for Storm, and I haven't use them
> directly.
> > > How much UX gets worse when we remove connectors to binary dist?
> It only
> > > helps some users who are not connected to the internet, and IMHO
> it's a
> > > rare case.
> > >
> > > I would like to see the opposite approach, removing all connectors
> (or
> > > just keeping storm-kafka/storm-kafka-client and some more preferred
> > things)
> > > and its relevant examples from binary dist.
> > >
> > > What do you think about it?
> > >
> > > - Jungtaek Lim (HeartSaVioR)
> > >
> > >
> > > 2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ro...@hortonworks.com>님이
> 작성:
> > >
> > > Once all of the shaded examples are included the size will go up
> further.
> > >
> > > But currently as they are not part of the tar.gz … something else
> is the
> > > culprit for the bloat.
> > >
> > >
> > >
> > > Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases
> .. of
> > > files that are larger than 4MB.
> > >
> > >
> > >
> > >
> > >
> > > @Jungtaek Lim<ma...@hortonworks.com> : I am thinking,
> since the
> > > code for the examples can be easily viewed online … it would be
> valuable
> > to
> > > have the executable topologies made available to the user as part
> of the
> > > binary release … rather have them figure out how to build it
> correctly
> > > before trying them out.
> > >
> > >
> > >
> > > -roshan
> > >
> > >
> > >
> > >
> > >
> > > ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh
> {} \;
> > >
> > > -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33
> > ./examples/storm-starter/
> > > storm-starter-topologies-1.0.3.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30
> > > ./external/flux/flux-examples-1.0.3.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32
> > > ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32
> > > ./external/sql/storm-sql-core/guava-16.0.1.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30
> > > ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-
> > with-dependencies.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33
> > >
> ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33
> > > ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
> > >
> > > -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02
> ./lib/clojure-1.7.0.jar
> > >
> > > -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26
> > ./lib/storm-core-1.0.3.jar
> > >
> > > -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26
> > > ./lib/storm-rename-hack-1.0.3.jar
> > >
> > >
> > >
> > > ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh
> {} \;
> > >
> > > -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23
> ./examples/storm-pmml-
> > > examples/storm-pmml-examples-1.1.0.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20
> > ./examples/storm-starter/
> > > storm-starter-topologies-1.1.0.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11
> > > ./external/flux/flux-examples-1.1.0.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16
> > > ./external/sql/storm-sql-core/calcite-core-1.11.0.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16
> > > ./external/sql/storm-sql-core/guava-16.0.1.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12
> > ./external/sql/storm-sql-
> > > runtime/calcite-core-1.11.0.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12
> > ./external/sql/storm-sql-
> > > runtime/guava-16.0.1.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18
> > > ./external/storm-druid/storm-druid-1.1.0.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11
> > > ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-
> > with-dependencies.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20
> > >
> ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18
> > > ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02
> ./lib/clojure-1.7.0.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07
> > ./lib/storm-core-1.1.0.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07
> > > ./lib/storm-rename-hack-1.1.0.jar
> > >
> > > -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19
> > > ./toollib/storm-kafka-monitor-1.1.0.jar
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> > >
> > >
> > >
> > > About STORM-2249, since examples are shading their dependencies,
> > binary
> > >
> > > dist will grow much bigger. I've left some comments regarding
> that.
> > >
> > > Btw, I have another view of this. Showing example codes is more
> > > important
> > >
> > > than just let users execute some topologies. That's what example
> > > modules
> > >
> > > are for. We need to include source as well. If we need to pick
> one,
> > > source
> > >
> > > code would be better.
> > >
> > >
> > >
> > > STORM-2343 seems better to add to 1.1.0. I just am not enough
> > familiar
> > > with
> > >
> > > storm-kafka-client so not sure I can review that, but I'll try
> to. I
> > > feel
> > >
> > > it's not that make release dragged. Let's add to 1.1.0 epic.
> > >
> > >
> > >
> > > Let's make minimum merge before another RC vote. Addressing
> > STORM-2389
> > > (and
> > >
> > > maybe STORM-2343) is enough for me. Others are not that
> critical.
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Jungtaek Lim (HeartSaVioR)
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <
> > > hlouro@hortonworks.com>
> > >
> > > wrote:
> > >
> > >
> > >
> > > > Roshan, does this PR<
> https://github.com/apache/storm/pull/1831>
> > and
> > > JIRA<
> > >
> > > > https://issues.apache.org/jira/browse/STORM-2249> address the
> > > missing
> > >
> > > > jars problem that you mentioned. I had created it in December
> 2016,
> > > but
> > >
> > > > there is an ongoing discussion if we should indeed put the
> jars in
> > > the
> > >
> > > > examples location or not.
> > >
> > > >
> > >
> > > > On a different note, this storm-kafka-client/KafkaSpout PR<
> > >
> > > > https://github.com/apache/storm/pull/1924> fixes a bug with
> the
> > > number of
> > >
> > > > uncommitted offsets that is quite important. It is not a
> blocker,
> > > but it is
> > >
> > > > quite critical. I am going to do one last pass reviewing
> today. It
> > > would be
> > >
> > > > good if we could have this PR included with the release. Can
> anyone
> > > else
> > >
> > > > review it as well ?
> > >
> > > >
> > >
> > > > Thanks,
> > >
> > > > Hugo
> > >
> > > >
> > >
> > > > On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <
> ptgoetz@gmail.com
> > > <mailto:ptgo
> > >
> > > > etz@gmail.com>> wrote:
> > >
> > > >
> > >
> > > > Yeah, I don’t think the file size is a killer/blocker. It’s
> largely
> > > due to
> > >
> > > > shaded examples, etc. But it’s something to keep an eye on.
> Our
> > > binary
> > >
> > > > releases shouldn’t have to be that big.
> > >
> > > >
> > >
> > > > -Taylor
> > >
> > > >
> > >
> > > > On Mar 1, 2017, at 12:09 PM, Roshan Naik <
> roshan@hortonworks.com<
> > > mailto:
> > >
> > > > roshan@hortonworks.com>> wrote:
> > >
> > > >
> > >
> > > > Have filed Jiras so for the 3 issues mentioned. Not sure if
> we need
> > > a JIRA
> > >
> > > > for the file size getting bloated by that much.
> > >
> > > > Somebody better familiar with the matter may want to take
> about
> > that?
> > >
> > > > -roshan
> > >
> > > >
> > >
> > > >
> > >
> > > > On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com
> <mailto:
> > ptgo
> > >
> > > > etz@gmail.com>> wrote:
> > >
> > > >
> > >
> > > > Thanks for bringing these up Roshan. Feel free to file JIRA
> > > tickets for
> > >
> > > > these issues and assign the “Release Apache Storm 1.1.0” epic
> so
> > > they can
> > >
> > > > be tracked for this release.
> > >
> > > >
> > >
> > > > -Taylor
> > >
> > > >
> > >
> > > > On Mar 1, 2017, at 9:27 AM, Roshan Naik <
> roshan@hortonworks.com<
> > > mailto:
> > >
> > > > roshan@hortonworks.com>> wrote:
> > >
> > > >
> > >
> > > > Found these additional issues:
> > >
> > > >
> > >
> > > >
> > >
> > > >
> > >
> > > > 1- BUG: Even if topology.eventlogger.executors=0, the
> > event_logger
> > > bolt
> > >
> > > > is instantiated … previously observed to cause ~10%
> degradation in
> > > perf
> > >
> > > > even with logging disabled.
> > >
> > > >
> > >
> > > > 2- Missing Jars: The storm-*-examples jars are missing in the
> > binary
> > >
> > > > distro (other than a storm-pmml-examples.jar,
> > storm-jms-examples.jar
> > > &
> > >
> > > > flux-examples.jar).
> > >
> > > >
> > >
> > > > 3- Minor: HdfsSpoutTopology example has not been moved into
> > >
> > > > storm-hdfs-examples from storm-starter
> > >
> > > >
> > >
> > > >
> > >
> > > >
> > >
> > > > Another side observation … v1.0.3 tar.gz downloadable was
> 190MB.
> > > This
> > >
> > > > v1.1.0 tar.gz downloadable is 297MB !! …. even though some
> of the
> > > example
> > >
> > > > topologies didn’t make it.
> > >
> > > >
> > >
> > > >
> > >
> > > >
> > >
> > > > -roshan
> > >
> > > >
> > >
> > > >
> > >
> > > >
> > >
> > > >
> > >
> > > >
> > >
> > > >
> > >
> > > >
> > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > >
> > > Name : Jungtaek Lim
> > >
> > > Blog : http://medium.com/@heartsavior
> > >
> > > Twitter : http://twitter.com/heartsavior
> > >
> > > LinkedIn : http://www.linkedin.com/in/heartsavior
> > >
> > >
> > >
> >
>
>
>
>
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Roshan Naik <ro...@hortonworks.com>.
As having a separate module for each of the connectors will cause a lot of bloat … may be good to club them into one.
-roshan
On 3/2/17, 10:39 AM, "Sree V" <sr...@yahoo.com.INVALID> wrote:
+1separate main binaries, connectors and samples/examples.
Thanking you.
With Regards
Sree
On Thursday, March 2, 2017 8:02 AM, Satish Duggana <sa...@gmail.com> wrote:
Agree that such huge binaries may not be acceptable. We should really think
about the options proposed earlier like excluding some of the external
connectors from binary and update the documentation respectively.
Thanks,
Satish.
On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> Adding my observation to my last mail:
> I just give STORM-2249 a try (against master branch) and compare before vs
> after.
> It adds more than 300 MB, and the final archive is more than 550 MB. I
> guess it would be similar for 1.x branch.
>
> Before ----------
> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> pom
> -rw-r--r-- 1 jlim staff 473B 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> pom.asc
> -rw-r--r-- 1 jlim staff 264M 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> tar.gz
> -rw-r--r-- 1 jlim staff 473B 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> tar.gz.asc
> -rw-r--r-- 1 jlim staff 264M 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> zip
> -rw-r--r-- 1 jlim staff 473B 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> zip.asc
>
> After -----------
> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> pom
> -rw-r--r-- 1 jlim staff 473B 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> pom.asc
> -rw-r--r-- 1 jlim staff 564M 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> tar.gz
> -rw-r--r-- 1 jlim staff 473B 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> tar.gz.asc
> -rw-r--r-- 1 jlim staff 565M 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> zip
> -rw-r--r-- 1 jlim staff 473B 3 2 19:47 apache-storm-2.0.0-SNAPSHOT.
> zip.asc
>
> While 264M is already a bit huge for me, 564M is not that I can accept.
> (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka 0.10.2 is
> 37M.)
>
> Btw, we're including source code of examples, and "mvn clean package" will
> work for every example modules.
>
> 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim <ka...@gmail.com>님이 작성:
>
> > I guess it might be good time to think why we add all connectors to the
> > binary distribution.
> >
> > Spark and Flink don't include them to binary dist. They even moved some
> or
> > most of connectors out of repo, have been maintaining them in Apache
> Bahir.
> > (Personally this is something I'm in favor of. We have lots of connectors
> > and many of them are outdated - clear example is storm-elasticsearch.)
> >
> > If we are assuming online then we don't even need to think about users
> > touching binary dist. version of connectors. Users have been including
> them
> > via build tools' dependency management, or even starting 1.1.0, users can
> > include them via '--artifact' option.
> >
> > I also was just one of users for Storm, and I haven't use them directly.
> > How much UX gets worse when we remove connectors to binary dist? It only
> > helps some users who are not connected to the internet, and IMHO it's a
> > rare case.
> >
> > I would like to see the opposite approach, removing all connectors (or
> > just keeping storm-kafka/storm-kafka-client and some more preferred
> things)
> > and its relevant examples from binary dist.
> >
> > What do you think about it?
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> >
> > 2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ro...@hortonworks.com>님이 작성:
> >
> > Once all of the shaded examples are included the size will go up further.
> >
> > But currently as they are not part of the tar.gz … something else is the
> > culprit for the bloat.
> >
> >
> >
> > Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases .. of
> > files that are larger than 4MB.
> >
> >
> >
> >
> >
> > @Jungtaek Lim<ma...@hortonworks.com> : I am thinking, since the
> > code for the examples can be easily viewed online … it would be valuable
> to
> > have the executable topologies made available to the user as part of the
> > binary release … rather have them figure out how to build it correctly
> > before trying them out.
> >
> >
> >
> > -roshan
> >
> >
> >
> >
> >
> > ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh {} \;
> >
> > -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33
> ./examples/storm-starter/
> > storm-starter-topologies-1.0.3.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30
> > ./external/flux/flux-examples-1.0.3.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32
> > ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32
> > ./external/sql/storm-sql-core/guava-16.0.1.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30
> > ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-
> with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33
> > ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33
> > ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
> >
> > -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar
> >
> > -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26
> ./lib/storm-core-1.0.3.jar
> >
> > -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26
> > ./lib/storm-rename-hack-1.0.3.jar
> >
> >
> >
> > ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh {} \;
> >
> > -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23 ./examples/storm-pmml-
> > examples/storm-pmml-examples-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20
> ./examples/storm-starter/
> > storm-starter-topologies-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11
> > ./external/flux/flux-examples-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16
> > ./external/sql/storm-sql-core/calcite-core-1.11.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16
> > ./external/sql/storm-sql-core/guava-16.0.1.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12
> ./external/sql/storm-sql-
> > runtime/calcite-core-1.11.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12
> ./external/sql/storm-sql-
> > runtime/guava-16.0.1.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18
> > ./external/storm-druid/storm-druid-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11
> > ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-
> with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20
> > ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18
> > ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07
> ./lib/storm-core-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07
> > ./lib/storm-rename-hack-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19
> > ./toollib/storm-kafka-monitor-1.1.0.jar
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >
> >
> >
> > About STORM-2249, since examples are shading their dependencies,
> binary
> >
> > dist will grow much bigger. I've left some comments regarding that.
> >
> > Btw, I have another view of this. Showing example codes is more
> > important
> >
> > than just let users execute some topologies. That's what example
> > modules
> >
> > are for. We need to include source as well. If we need to pick one,
> > source
> >
> > code would be better.
> >
> >
> >
> > STORM-2343 seems better to add to 1.1.0. I just am not enough
> familiar
> > with
> >
> > storm-kafka-client so not sure I can review that, but I'll try to. I
> > feel
> >
> > it's not that make release dragged. Let's add to 1.1.0 epic.
> >
> >
> >
> > Let's make minimum merge before another RC vote. Addressing
> STORM-2389
> > (and
> >
> > maybe STORM-2343) is enough for me. Others are not that critical.
> >
> >
> >
> > Thanks,
> >
> > Jungtaek Lim (HeartSaVioR)
> >
> >
> >
> >
> >
> > On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <
> > hlouro@hortonworks.com>
> >
> > wrote:
> >
> >
> >
> > > Roshan, does this PR<https://github.com/apache/storm/pull/1831>
> and
> > JIRA<
> >
> > > https://issues.apache.org/jira/browse/STORM-2249> address the
> > missing
> >
> > > jars problem that you mentioned. I had created it in December 2016,
> > but
> >
> > > there is an ongoing discussion if we should indeed put the jars in
> > the
> >
> > > examples location or not.
> >
> > >
> >
> > > On a different note, this storm-kafka-client/KafkaSpout PR<
> >
> > > https://github.com/apache/storm/pull/1924> fixes a bug with the
> > number of
> >
> > > uncommitted offsets that is quite important. It is not a blocker,
> > but it is
> >
> > > quite critical. I am going to do one last pass reviewing today. It
> > would be
> >
> > > good if we could have this PR included with the release. Can anyone
> > else
> >
> > > review it as well ?
> >
> > >
> >
> > > Thanks,
> >
> > > Hugo
> >
> > >
> >
> > > On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <ptgoetz@gmail.com
> > <mailto:ptgo
> >
> > > etz@gmail.com>> wrote:
> >
> > >
> >
> > > Yeah, I don’t think the file size is a killer/blocker. It’s largely
> > due to
> >
> > > shaded examples, etc. But it’s something to keep an eye on. Our
> > binary
> >
> > > releases shouldn’t have to be that big.
> >
> > >
> >
> > > -Taylor
> >
> > >
> >
> > > On Mar 1, 2017, at 12:09 PM, Roshan Naik <roshan@hortonworks.com<
> > mailto:
> >
> > > roshan@hortonworks.com>> wrote:
> >
> > >
> >
> > > Have filed Jiras so for the 3 issues mentioned. Not sure if we need
> > a JIRA
> >
> > > for the file size getting bloated by that much.
> >
> > > Somebody better familiar with the matter may want to take about
> that?
> >
> > > -roshan
> >
> > >
> >
> > >
> >
> > > On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com<mailto:
> ptgo
> >
> > > etz@gmail.com>> wrote:
> >
> > >
> >
> > > Thanks for bringing these up Roshan. Feel free to file JIRA
> > tickets for
> >
> > > these issues and assign the “Release Apache Storm 1.1.0” epic so
> > they can
> >
> > > be tracked for this release.
> >
> > >
> >
> > > -Taylor
> >
> > >
> >
> > > On Mar 1, 2017, at 9:27 AM, Roshan Naik <roshan@hortonworks.com<
> > mailto:
> >
> > > roshan@hortonworks.com>> wrote:
> >
> > >
> >
> > > Found these additional issues:
> >
> > >
> >
> > >
> >
> > >
> >
> > > 1- BUG: Even if topology.eventlogger.executors=0, the
> event_logger
> > bolt
> >
> > > is instantiated … previously observed to cause ~10% degradation in
> > perf
> >
> > > even with logging disabled.
> >
> > >
> >
> > > 2- Missing Jars: The storm-*-examples jars are missing in the
> binary
> >
> > > distro (other than a storm-pmml-examples.jar,
> storm-jms-examples.jar
> > &
> >
> > > flux-examples.jar).
> >
> > >
> >
> > > 3- Minor: HdfsSpoutTopology example has not been moved into
> >
> > > storm-hdfs-examples from storm-starter
> >
> > >
> >
> > >
> >
> > >
> >
> > > Another side observation … v1.0.3 tar.gz downloadable was 190MB.
> > This
> >
> > > v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the
> > example
> >
> > > topologies didn’t make it.
> >
> > >
> >
> > >
> >
> > >
> >
> > > -roshan
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> >
> >
> >
> >
> > --
> >
> > Name : Jungtaek Lim
> >
> > Blog : http://medium.com/@heartsavior
> >
> > Twitter : http://twitter.com/heartsavior
> >
> > LinkedIn : http://www.linkedin.com/in/heartsavior
> >
> >
> >
>
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Sree V <sr...@yahoo.com.INVALID>.
+1separate main binaries, connectors and samples/examples.
Thanking you.
With Regards
Sree
On Thursday, March 2, 2017 8:02 AM, Satish Duggana <sa...@gmail.com> wrote:
Agree that such huge binaries may not be acceptable. We should really think
about the options proposed earlier like excluding some of the external
connectors from binary and update the documentation respectively.
Thanks,
Satish.
On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> Adding my observation to my last mail:
> I just give STORM-2249 a try (against master branch) and compare before vs
> after.
> It adds more than 300 MB, and the final archive is more than 550 MB. I
> guess it would be similar for 1.x branch.
>
> Before ----------
> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> pom
> -rw-r--r-- 1 jlim staff 473B 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> pom.asc
> -rw-r--r-- 1 jlim staff 264M 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> tar.gz
> -rw-r--r-- 1 jlim staff 473B 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> tar.gz.asc
> -rw-r--r-- 1 jlim staff 264M 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> zip
> -rw-r--r-- 1 jlim staff 473B 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> zip.asc
>
> After -----------
> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> pom
> -rw-r--r-- 1 jlim staff 473B 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> pom.asc
> -rw-r--r-- 1 jlim staff 564M 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> tar.gz
> -rw-r--r-- 1 jlim staff 473B 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> tar.gz.asc
> -rw-r--r-- 1 jlim staff 565M 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> zip
> -rw-r--r-- 1 jlim staff 473B 3 2 19:47 apache-storm-2.0.0-SNAPSHOT.
> zip.asc
>
> While 264M is already a bit huge for me, 564M is not that I can accept.
> (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka 0.10.2 is
> 37M.)
>
> Btw, we're including source code of examples, and "mvn clean package" will
> work for every example modules.
>
> 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim <ka...@gmail.com>님이 작성:
>
> > I guess it might be good time to think why we add all connectors to the
> > binary distribution.
> >
> > Spark and Flink don't include them to binary dist. They even moved some
> or
> > most of connectors out of repo, have been maintaining them in Apache
> Bahir.
> > (Personally this is something I'm in favor of. We have lots of connectors
> > and many of them are outdated - clear example is storm-elasticsearch.)
> >
> > If we are assuming online then we don't even need to think about users
> > touching binary dist. version of connectors. Users have been including
> them
> > via build tools' dependency management, or even starting 1.1.0, users can
> > include them via '--artifact' option.
> >
> > I also was just one of users for Storm, and I haven't use them directly.
> > How much UX gets worse when we remove connectors to binary dist? It only
> > helps some users who are not connected to the internet, and IMHO it's a
> > rare case.
> >
> > I would like to see the opposite approach, removing all connectors (or
> > just keeping storm-kafka/storm-kafka-client and some more preferred
> things)
> > and its relevant examples from binary dist.
> >
> > What do you think about it?
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> >
> > 2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ro...@hortonworks.com>님이 작성:
> >
> > Once all of the shaded examples are included the size will go up further.
> >
> > But currently as they are not part of the tar.gz … something else is the
> > culprit for the bloat.
> >
> >
> >
> > Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases .. of
> > files that are larger than 4MB.
> >
> >
> >
> >
> >
> > @Jungtaek Lim<ma...@hortonworks.com> : I am thinking, since the
> > code for the examples can be easily viewed online … it would be valuable
> to
> > have the executable topologies made available to the user as part of the
> > binary release … rather have them figure out how to build it correctly
> > before trying them out.
> >
> >
> >
> > -roshan
> >
> >
> >
> >
> >
> > ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh {} \;
> >
> > -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33
> ./examples/storm-starter/
> > storm-starter-topologies-1.0.3.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30
> > ./external/flux/flux-examples-1.0.3.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32
> > ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32
> > ./external/sql/storm-sql-core/guava-16.0.1.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30
> > ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-
> with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33
> > ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33
> > ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
> >
> > -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar
> >
> > -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26
> ./lib/storm-core-1.0.3.jar
> >
> > -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26
> > ./lib/storm-rename-hack-1.0.3.jar
> >
> >
> >
> > ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh {} \;
> >
> > -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23 ./examples/storm-pmml-
> > examples/storm-pmml-examples-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20
> ./examples/storm-starter/
> > storm-starter-topologies-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11
> > ./external/flux/flux-examples-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16
> > ./external/sql/storm-sql-core/calcite-core-1.11.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16
> > ./external/sql/storm-sql-core/guava-16.0.1.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12
> ./external/sql/storm-sql-
> > runtime/calcite-core-1.11.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12
> ./external/sql/storm-sql-
> > runtime/guava-16.0.1.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18
> > ./external/storm-druid/storm-druid-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11
> > ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-
> with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20
> > ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18
> > ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07
> ./lib/storm-core-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07
> > ./lib/storm-rename-hack-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19
> > ./toollib/storm-kafka-monitor-1.1.0.jar
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >
> >
> >
> > About STORM-2249, since examples are shading their dependencies,
> binary
> >
> > dist will grow much bigger. I've left some comments regarding that.
> >
> > Btw, I have another view of this. Showing example codes is more
> > important
> >
> > than just let users execute some topologies. That's what example
> > modules
> >
> > are for. We need to include source as well. If we need to pick one,
> > source
> >
> > code would be better.
> >
> >
> >
> > STORM-2343 seems better to add to 1.1.0. I just am not enough
> familiar
> > with
> >
> > storm-kafka-client so not sure I can review that, but I'll try to. I
> > feel
> >
> > it's not that make release dragged. Let's add to 1.1.0 epic.
> >
> >
> >
> > Let's make minimum merge before another RC vote. Addressing
> STORM-2389
> > (and
> >
> > maybe STORM-2343) is enough for me. Others are not that critical.
> >
> >
> >
> > Thanks,
> >
> > Jungtaek Lim (HeartSaVioR)
> >
> >
> >
> >
> >
> > On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <
> > hlouro@hortonworks.com>
> >
> > wrote:
> >
> >
> >
> > > Roshan, does this PR<https://github.com/apache/storm/pull/1831>
> and
> > JIRA<
> >
> > > https://issues.apache.org/jira/browse/STORM-2249> address the
> > missing
> >
> > > jars problem that you mentioned. I had created it in December 2016,
> > but
> >
> > > there is an ongoing discussion if we should indeed put the jars in
> > the
> >
> > > examples location or not.
> >
> > >
> >
> > > On a different note, this storm-kafka-client/KafkaSpout PR<
> >
> > > https://github.com/apache/storm/pull/1924> fixes a bug with the
> > number of
> >
> > > uncommitted offsets that is quite important. It is not a blocker,
> > but it is
> >
> > > quite critical. I am going to do one last pass reviewing today. It
> > would be
> >
> > > good if we could have this PR included with the release. Can anyone
> > else
> >
> > > review it as well ?
> >
> > >
> >
> > > Thanks,
> >
> > > Hugo
> >
> > >
> >
> > > On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <ptgoetz@gmail.com
> > <mailto:ptgo
> >
> > > etz@gmail.com>> wrote:
> >
> > >
> >
> > > Yeah, I don’t think the file size is a killer/blocker. It’s largely
> > due to
> >
> > > shaded examples, etc. But it’s something to keep an eye on. Our
> > binary
> >
> > > releases shouldn’t have to be that big.
> >
> > >
> >
> > > -Taylor
> >
> > >
> >
> > > On Mar 1, 2017, at 12:09 PM, Roshan Naik <roshan@hortonworks.com<
> > mailto:
> >
> > > roshan@hortonworks.com>> wrote:
> >
> > >
> >
> > > Have filed Jiras so for the 3 issues mentioned. Not sure if we need
> > a JIRA
> >
> > > for the file size getting bloated by that much.
> >
> > > Somebody better familiar with the matter may want to take about
> that?
> >
> > > -roshan
> >
> > >
> >
> > >
> >
> > > On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com<mailto:
> ptgo
> >
> > > etz@gmail.com>> wrote:
> >
> > >
> >
> > > Thanks for bringing these up Roshan. Feel free to file JIRA
> > tickets for
> >
> > > these issues and assign the “Release Apache Storm 1.1.0” epic so
> > they can
> >
> > > be tracked for this release.
> >
> > >
> >
> > > -Taylor
> >
> > >
> >
> > > On Mar 1, 2017, at 9:27 AM, Roshan Naik <roshan@hortonworks.com<
> > mailto:
> >
> > > roshan@hortonworks.com>> wrote:
> >
> > >
> >
> > > Found these additional issues:
> >
> > >
> >
> > >
> >
> > >
> >
> > > 1- BUG: Even if topology.eventlogger.executors=0, the
> event_logger
> > bolt
> >
> > > is instantiated … previously observed to cause ~10% degradation in
> > perf
> >
> > > even with logging disabled.
> >
> > >
> >
> > > 2- Missing Jars: The storm-*-examples jars are missing in the
> binary
> >
> > > distro (other than a storm-pmml-examples.jar,
> storm-jms-examples.jar
> > &
> >
> > > flux-examples.jar).
> >
> > >
> >
> > > 3- Minor: HdfsSpoutTopology example has not been moved into
> >
> > > storm-hdfs-examples from storm-starter
> >
> > >
> >
> > >
> >
> > >
> >
> > > Another side observation … v1.0.3 tar.gz downloadable was 190MB.
> > This
> >
> > > v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the
> > example
> >
> > > topologies didn’t make it.
> >
> > >
> >
> > >
> >
> > >
> >
> > > -roshan
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> >
> >
> >
> >
> > --
> >
> > Name : Jungtaek Lim
> >
> > Blog : http://medium.com/@heartsavior
> >
> > Twitter : http://twitter.com/heartsavior
> >
> > LinkedIn : http://www.linkedin.com/in/heartsavior
> >
> >
> >
>
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Satish Duggana <sa...@gmail.com>.
Agree that such huge binaries may not be acceptable. We should really think
about the options proposed earlier like excluding some of the external
connectors from binary and update the documentation respectively.
Thanks,
Satish.
On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> Adding my observation to my last mail:
> I just give STORM-2249 a try (against master branch) and compare before vs
> after.
> It adds more than 300 MB, and the final archive is more than 550 MB. I
> guess it would be similar for 1.x branch.
>
> Before ----------
> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> pom
> -rw-r--r-- 1 jlim staff 473B 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> pom.asc
> -rw-r--r-- 1 jlim staff 264M 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> tar.gz
> -rw-r--r-- 1 jlim staff 473B 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> tar.gz.asc
> -rw-r--r-- 1 jlim staff 264M 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> zip
> -rw-r--r-- 1 jlim staff 473B 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
> zip.asc
>
> After -----------
> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> pom
> -rw-r--r-- 1 jlim staff 473B 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> pom.asc
> -rw-r--r-- 1 jlim staff 564M 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> tar.gz
> -rw-r--r-- 1 jlim staff 473B 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> tar.gz.asc
> -rw-r--r-- 1 jlim staff 565M 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
> zip
> -rw-r--r-- 1 jlim staff 473B 3 2 19:47 apache-storm-2.0.0-SNAPSHOT.
> zip.asc
>
> While 264M is already a bit huge for me, 564M is not that I can accept.
> (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka 0.10.2 is
> 37M.)
>
> Btw, we're including source code of examples, and "mvn clean package" will
> work for every example modules.
>
> 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim <ka...@gmail.com>님이 작성:
>
> > I guess it might be good time to think why we add all connectors to the
> > binary distribution.
> >
> > Spark and Flink don't include them to binary dist. They even moved some
> or
> > most of connectors out of repo, have been maintaining them in Apache
> Bahir.
> > (Personally this is something I'm in favor of. We have lots of connectors
> > and many of them are outdated - clear example is storm-elasticsearch.)
> >
> > If we are assuming online then we don't even need to think about users
> > touching binary dist. version of connectors. Users have been including
> them
> > via build tools' dependency management, or even starting 1.1.0, users can
> > include them via '--artifact' option.
> >
> > I also was just one of users for Storm, and I haven't use them directly.
> > How much UX gets worse when we remove connectors to binary dist? It only
> > helps some users who are not connected to the internet, and IMHO it's a
> > rare case.
> >
> > I would like to see the opposite approach, removing all connectors (or
> > just keeping storm-kafka/storm-kafka-client and some more preferred
> things)
> > and its relevant examples from binary dist.
> >
> > What do you think about it?
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> >
> > 2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ro...@hortonworks.com>님이 작성:
> >
> > Once all of the shaded examples are included the size will go up further.
> >
> > But currently as they are not part of the tar.gz … something else is the
> > culprit for the bloat.
> >
> >
> >
> > Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases .. of
> > files that are larger than 4MB.
> >
> >
> >
> >
> >
> > @Jungtaek Lim<ma...@hortonworks.com> : I am thinking, since the
> > code for the examples can be easily viewed online … it would be valuable
> to
> > have the executable topologies made available to the user as part of the
> > binary release … rather have them figure out how to build it correctly
> > before trying them out.
> >
> >
> >
> > -roshan
> >
> >
> >
> >
> >
> > ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh {} \;
> >
> > -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33
> ./examples/storm-starter/
> > storm-starter-topologies-1.0.3.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30
> > ./external/flux/flux-examples-1.0.3.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32
> > ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32
> > ./external/sql/storm-sql-core/guava-16.0.1.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30
> > ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-
> with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33
> > ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33
> > ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
> >
> > -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar
> >
> > -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26
> ./lib/storm-core-1.0.3.jar
> >
> > -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26
> > ./lib/storm-rename-hack-1.0.3.jar
> >
> >
> >
> > ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh {} \;
> >
> > -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23 ./examples/storm-pmml-
> > examples/storm-pmml-examples-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20
> ./examples/storm-starter/
> > storm-starter-topologies-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11
> > ./external/flux/flux-examples-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16
> > ./external/sql/storm-sql-core/calcite-core-1.11.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16
> > ./external/sql/storm-sql-core/guava-16.0.1.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12
> ./external/sql/storm-sql-
> > runtime/calcite-core-1.11.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12
> ./external/sql/storm-sql-
> > runtime/guava-16.0.1.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18
> > ./external/storm-druid/storm-druid-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11
> > ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-
> with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20
> > ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18
> > ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07
> ./lib/storm-core-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07
> > ./lib/storm-rename-hack-1.1.0.jar
> >
> > -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19
> > ./toollib/storm-kafka-monitor-1.1.0.jar
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >
> >
> >
> > About STORM-2249, since examples are shading their dependencies,
> binary
> >
> > dist will grow much bigger. I've left some comments regarding that.
> >
> > Btw, I have another view of this. Showing example codes is more
> > important
> >
> > than just let users execute some topologies. That's what example
> > modules
> >
> > are for. We need to include source as well. If we need to pick one,
> > source
> >
> > code would be better.
> >
> >
> >
> > STORM-2343 seems better to add to 1.1.0. I just am not enough
> familiar
> > with
> >
> > storm-kafka-client so not sure I can review that, but I'll try to. I
> > feel
> >
> > it's not that make release dragged. Let's add to 1.1.0 epic.
> >
> >
> >
> > Let's make minimum merge before another RC vote. Addressing
> STORM-2389
> > (and
> >
> > maybe STORM-2343) is enough for me. Others are not that critical.
> >
> >
> >
> > Thanks,
> >
> > Jungtaek Lim (HeartSaVioR)
> >
> >
> >
> >
> >
> > On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <
> > hlouro@hortonworks.com>
> >
> > wrote:
> >
> >
> >
> > > Roshan, does this PR<https://github.com/apache/storm/pull/1831>
> and
> > JIRA<
> >
> > > https://issues.apache.org/jira/browse/STORM-2249> address the
> > missing
> >
> > > jars problem that you mentioned. I had created it in December 2016,
> > but
> >
> > > there is an ongoing discussion if we should indeed put the jars in
> > the
> >
> > > examples location or not.
> >
> > >
> >
> > > On a different note, this storm-kafka-client/KafkaSpout PR<
> >
> > > https://github.com/apache/storm/pull/1924> fixes a bug with the
> > number of
> >
> > > uncommitted offsets that is quite important. It is not a blocker,
> > but it is
> >
> > > quite critical. I am going to do one last pass reviewing today. It
> > would be
> >
> > > good if we could have this PR included with the release. Can anyone
> > else
> >
> > > review it as well ?
> >
> > >
> >
> > > Thanks,
> >
> > > Hugo
> >
> > >
> >
> > > On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <ptgoetz@gmail.com
> > <mailto:ptgo
> >
> > > etz@gmail.com>> wrote:
> >
> > >
> >
> > > Yeah, I don’t think the file size is a killer/blocker. It’s largely
> > due to
> >
> > > shaded examples, etc. But it’s something to keep an eye on. Our
> > binary
> >
> > > releases shouldn’t have to be that big.
> >
> > >
> >
> > > -Taylor
> >
> > >
> >
> > > On Mar 1, 2017, at 12:09 PM, Roshan Naik <roshan@hortonworks.com<
> > mailto:
> >
> > > roshan@hortonworks.com>> wrote:
> >
> > >
> >
> > > Have filed Jiras so for the 3 issues mentioned. Not sure if we need
> > a JIRA
> >
> > > for the file size getting bloated by that much.
> >
> > > Somebody better familiar with the matter may want to take about
> that?
> >
> > > -roshan
> >
> > >
> >
> > >
> >
> > > On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com<mailto:
> ptgo
> >
> > > etz@gmail.com>> wrote:
> >
> > >
> >
> > > Thanks for bringing these up Roshan. Feel free to file JIRA
> > tickets for
> >
> > > these issues and assign the “Release Apache Storm 1.1.0” epic so
> > they can
> >
> > > be tracked for this release.
> >
> > >
> >
> > > -Taylor
> >
> > >
> >
> > > On Mar 1, 2017, at 9:27 AM, Roshan Naik <roshan@hortonworks.com<
> > mailto:
> >
> > > roshan@hortonworks.com>> wrote:
> >
> > >
> >
> > > Found these additional issues:
> >
> > >
> >
> > >
> >
> > >
> >
> > > 1- BUG: Even if topology.eventlogger.executors=0, the
> event_logger
> > bolt
> >
> > > is instantiated … previously observed to cause ~10% degradation in
> > perf
> >
> > > even with logging disabled.
> >
> > >
> >
> > > 2- Missing Jars: The storm-*-examples jars are missing in the
> binary
> >
> > > distro (other than a storm-pmml-examples.jar,
> storm-jms-examples.jar
> > &
> >
> > > flux-examples.jar).
> >
> > >
> >
> > > 3- Minor: HdfsSpoutTopology example has not been moved into
> >
> > > storm-hdfs-examples from storm-starter
> >
> > >
> >
> > >
> >
> > >
> >
> > > Another side observation … v1.0.3 tar.gz downloadable was 190MB.
> > This
> >
> > > v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the
> > example
> >
> > > topologies didn’t make it.
> >
> > >
> >
> > >
> >
> > >
> >
> > > -roshan
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> >
> >
> >
> >
> > --
> >
> > Name : Jungtaek Lim
> >
> > Blog : http://medium.com/@heartsavior
> >
> > Twitter : http://twitter.com/heartsavior
> >
> > LinkedIn : http://www.linkedin.com/in/heartsavior
> >
> >
> >
>
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Jungtaek Lim <ka...@gmail.com>.
Adding my observation to my last mail:
I just give STORM-2249 a try (against master branch) and compare before vs
after.
It adds more than 300 MB, and the final archive is more than 550 MB. I
guess it would be similar for 1.x branch.
Before ----------
-rw-r--r-- 1 jlim staff 3.3K 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
pom
-rw-r--r-- 1 jlim staff 473B 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
pom.asc
-rw-r--r-- 1 jlim staff 264M 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
tar.gz
-rw-r--r-- 1 jlim staff 473B 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
tar.gz.asc
-rw-r--r-- 1 jlim staff 264M 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
zip
-rw-r--r-- 1 jlim staff 473B 3 2 19:41 apache-storm-2.0.0-SNAPSHOT.
zip.asc
After -----------
-rw-r--r-- 1 jlim staff 3.3K 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
pom
-rw-r--r-- 1 jlim staff 473B 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
pom.asc
-rw-r--r-- 1 jlim staff 564M 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
tar.gz
-rw-r--r-- 1 jlim staff 473B 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
tar.gz.asc
-rw-r--r-- 1 jlim staff 565M 3 2 19:46 apache-storm-2.0.0-SNAPSHOT.
zip
-rw-r--r-- 1 jlim staff 473B 3 2 19:47 apache-storm-2.0.0-SNAPSHOT.
zip.asc
While 264M is already a bit huge for me, 564M is not that I can accept.
(Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka 0.10.2 is
37M.)
Btw, we're including source code of examples, and "mvn clean package" will
work for every example modules.
2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim <ka...@gmail.com>님이 작성:
> I guess it might be good time to think why we add all connectors to the
> binary distribution.
>
> Spark and Flink don't include them to binary dist. They even moved some or
> most of connectors out of repo, have been maintaining them in Apache Bahir.
> (Personally this is something I'm in favor of. We have lots of connectors
> and many of them are outdated - clear example is storm-elasticsearch.)
>
> If we are assuming online then we don't even need to think about users
> touching binary dist. version of connectors. Users have been including them
> via build tools' dependency management, or even starting 1.1.0, users can
> include them via '--artifact' option.
>
> I also was just one of users for Storm, and I haven't use them directly.
> How much UX gets worse when we remove connectors to binary dist? It only
> helps some users who are not connected to the internet, and IMHO it's a
> rare case.
>
> I would like to see the opposite approach, removing all connectors (or
> just keeping storm-kafka/storm-kafka-client and some more preferred things)
> and its relevant examples from binary dist.
>
> What do you think about it?
>
> - Jungtaek Lim (HeartSaVioR)
>
>
> 2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ro...@hortonworks.com>님이 작성:
>
> Once all of the shaded examples are included the size will go up further.
>
> But currently as they are not part of the tar.gz … something else is the
> culprit for the bloat.
>
>
>
> Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases .. of
> files that are larger than 4MB.
>
>
>
>
>
> @Jungtaek Lim<ma...@hortonworks.com> : I am thinking, since the
> code for the examples can be easily viewed online … it would be valuable to
> have the executable topologies made available to the user as part of the
> binary release … rather have them figure out how to build it correctly
> before trying them out.
>
>
>
> -roshan
>
>
>
>
>
> ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh {} \;
>
> -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33 ./examples/storm-starter/
> storm-starter-topologies-1.0.3.jar
>
> -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30
> ./external/flux/flux-examples-1.0.3.jar
>
> -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32
> ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
>
> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32
> ./external/sql/storm-sql-core/guava-16.0.1.jar
>
> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30
> ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-with-dependencies.jar
>
> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33
> ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
>
> -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33
> ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
>
> -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar
>
> -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26 ./lib/storm-core-1.0.3.jar
>
> -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26
> ./lib/storm-rename-hack-1.0.3.jar
>
>
>
> ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh {} \;
>
> -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23 ./examples/storm-pmml-
> examples/storm-pmml-examples-1.1.0.jar
>
> -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20 ./examples/storm-starter/
> storm-starter-topologies-1.1.0.jar
>
> -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11
> ./external/flux/flux-examples-1.1.0.jar
>
> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16
> ./external/sql/storm-sql-core/calcite-core-1.11.0.jar
>
> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16
> ./external/sql/storm-sql-core/guava-16.0.1.jar
>
> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12 ./external/sql/storm-sql-
> runtime/calcite-core-1.11.0.jar
>
> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12 ./external/sql/storm-sql-
> runtime/guava-16.0.1.jar
>
> -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18
> ./external/storm-druid/storm-druid-1.1.0.jar
>
> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11
> ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-with-dependencies.jar
>
> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20
> ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
>
> -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18
> ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
>
> -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar
>
> -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07 ./lib/storm-core-1.1.0.jar
>
> -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07
> ./lib/storm-rename-hack-1.1.0.jar
>
> -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19
> ./toollib/storm-kafka-monitor-1.1.0.jar
>
>
>
>
>
>
>
>
>
>
>
>
>
> On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>
>
>
> About STORM-2249, since examples are shading their dependencies, binary
>
> dist will grow much bigger. I've left some comments regarding that.
>
> Btw, I have another view of this. Showing example codes is more
> important
>
> than just let users execute some topologies. That's what example
> modules
>
> are for. We need to include source as well. If we need to pick one,
> source
>
> code would be better.
>
>
>
> STORM-2343 seems better to add to 1.1.0. I just am not enough familiar
> with
>
> storm-kafka-client so not sure I can review that, but I'll try to. I
> feel
>
> it's not that make release dragged. Let's add to 1.1.0 epic.
>
>
>
> Let's make minimum merge before another RC vote. Addressing STORM-2389
> (and
>
> maybe STORM-2343) is enough for me. Others are not that critical.
>
>
>
> Thanks,
>
> Jungtaek Lim (HeartSaVioR)
>
>
>
>
>
> On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <
> hlouro@hortonworks.com>
>
> wrote:
>
>
>
> > Roshan, does this PR<https://github.com/apache/storm/pull/1831> and
> JIRA<
>
> > https://issues.apache.org/jira/browse/STORM-2249> address the
> missing
>
> > jars problem that you mentioned. I had created it in December 2016,
> but
>
> > there is an ongoing discussion if we should indeed put the jars in
> the
>
> > examples location or not.
>
> >
>
> > On a different note, this storm-kafka-client/KafkaSpout PR<
>
> > https://github.com/apache/storm/pull/1924> fixes a bug with the
> number of
>
> > uncommitted offsets that is quite important. It is not a blocker,
> but it is
>
> > quite critical. I am going to do one last pass reviewing today. It
> would be
>
> > good if we could have this PR included with the release. Can anyone
> else
>
> > review it as well ?
>
> >
>
> > Thanks,
>
> > Hugo
>
> >
>
> > On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <ptgoetz@gmail.com
> <mailto:ptgo
>
> > etz@gmail.com>> wrote:
>
> >
>
> > Yeah, I don’t think the file size is a killer/blocker. It’s largely
> due to
>
> > shaded examples, etc. But it’s something to keep an eye on. Our
> binary
>
> > releases shouldn’t have to be that big.
>
> >
>
> > -Taylor
>
> >
>
> > On Mar 1, 2017, at 12:09 PM, Roshan Naik <roshan@hortonworks.com<
> mailto:
>
> > roshan@hortonworks.com>> wrote:
>
> >
>
> > Have filed Jiras so for the 3 issues mentioned. Not sure if we need
> a JIRA
>
> > for the file size getting bloated by that much.
>
> > Somebody better familiar with the matter may want to take about that?
>
> > -roshan
>
> >
>
> >
>
> > On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com<mailto:ptgo
>
> > etz@gmail.com>> wrote:
>
> >
>
> > Thanks for bringing these up Roshan. Feel free to file JIRA
> tickets for
>
> > these issues and assign the “Release Apache Storm 1.1.0” epic so
> they can
>
> > be tracked for this release.
>
> >
>
> > -Taylor
>
> >
>
> > On Mar 1, 2017, at 9:27 AM, Roshan Naik <roshan@hortonworks.com<
> mailto:
>
> > roshan@hortonworks.com>> wrote:
>
> >
>
> > Found these additional issues:
>
> >
>
> >
>
> >
>
> > 1- BUG: Even if topology.eventlogger.executors=0, the event_logger
> bolt
>
> > is instantiated … previously observed to cause ~10% degradation in
> perf
>
> > even with logging disabled.
>
> >
>
> > 2- Missing Jars: The storm-*-examples jars are missing in the binary
>
> > distro (other than a storm-pmml-examples.jar, storm-jms-examples.jar
> &
>
> > flux-examples.jar).
>
> >
>
> > 3- Minor: HdfsSpoutTopology example has not been moved into
>
> > storm-hdfs-examples from storm-starter
>
> >
>
> >
>
> >
>
> > Another side observation … v1.0.3 tar.gz downloadable was 190MB.
> This
>
> > v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the
> example
>
> > topologies didn’t make it.
>
> >
>
> >
>
> >
>
> > -roshan
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
>
>
>
>
> --
>
> Name : Jungtaek Lim
>
> Blog : http://medium.com/@heartsavior
>
> Twitter : http://twitter.com/heartsavior
>
> LinkedIn : http://www.linkedin.com/in/heartsavior
>
>
>
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Jungtaek Lim <ka...@gmail.com>.
I guess it might be good time to think why we add all connectors to the
binary distribution.
Spark and Flink don't include them to binary dist. They even moved some or
most of connectors out of repo, have been maintaining them in Apache Bahir.
(Personally this is something I'm in favor of. We have lots of connectors
and many of them are outdated - clear example is storm-elasticsearch.)
If we are assuming online then we don't even need to think about users
touching binary dist. version of connectors. Users have been including them
via build tools' dependency management, or even starting 1.1.0, users can
include them via '--artifact' option.
I also was just one of users for Storm, and I haven't use them directly.
How much UX gets worse when we remove connectors to binary dist? It only
helps some users who are not connected to the internet, and IMHO it's a
rare case.
I would like to see the opposite approach, removing all connectors (or just
keeping storm-kafka/storm-kafka-client and some more preferred things) and
its relevant examples from binary dist.
What do you think about it?
- Jungtaek Lim (HeartSaVioR)
2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ro...@hortonworks.com>님이 작성:
Once all of the shaded examples are included the size will go up further.
But currently as they are not part of the tar.gz … something else is the
culprit for the bloat.
Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases .. of
files that are larger than 4MB.
@Jungtaek Lim<ma...@hortonworks.com> : I am thinking, since the code
for the examples can be easily viewed online … it would be valuable to have
the executable topologies made available to the user as part of the binary
release … rather have them figure out how to build it correctly before
trying them out.
-roshan
➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh {} \;
-rw-r--r--@ 1 roshan staff 70M Feb 7 12:33
./examples/storm-starter/storm-starter-topologies-1.0.3.jar
-rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30
./external/flux/flux-examples-1.0.3.jar
-rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32
./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
-rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32
./external/sql/storm-sql-core/guava-16.0.1.jar
-rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30
./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-with-dependencies.jar
-rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33
./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
-rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33
./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
-rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar
-rw-r--r--@ 1 roshan staff 19M Feb 7 12:26 ./lib/storm-core-1.0.3.jar
-rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26
./lib/storm-rename-hack-1.0.3.jar
➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh {} \;
-rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23
./examples/storm-pmml-examples/storm-pmml-examples-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20
./examples/storm-starter/storm-starter-topologies-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11
./external/flux/flux-examples-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16
./external/sql/storm-sql-core/calcite-core-1.11.0.jar
-rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16
./external/sql/storm-sql-core/guava-16.0.1.jar
-rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12
./external/sql/storm-sql-runtime/calcite-core-1.11.0.jar
-rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12
./external/sql/storm-sql-runtime/guava-16.0.1.jar
-rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18
./external/storm-druid/storm-druid-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11
./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-with-dependencies.jar
-rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20
./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
-rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18
./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar
-rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07 ./lib/storm-core-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07
./lib/storm-rename-hack-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19
./toollib/storm-kafka-monitor-1.1.0.jar
On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
About STORM-2249, since examples are shading their dependencies, binary
dist will grow much bigger. I've left some comments regarding that.
Btw, I have another view of this. Showing example codes is more
important
than just let users execute some topologies. That's what example modules
are for. We need to include source as well. If we need to pick one,
source
code would be better.
STORM-2343 seems better to add to 1.1.0. I just am not enough familiar
with
storm-kafka-client so not sure I can review that, but I'll try to. I
feel
it's not that make release dragged. Let's add to 1.1.0 epic.
Let's make minimum merge before another RC vote. Addressing STORM-2389
(and
maybe STORM-2343) is enough for me. Others are not that critical.
Thanks,
Jungtaek Lim (HeartSaVioR)
On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <
hlouro@hortonworks.com>
wrote:
> Roshan, does this PR<https://github.com/apache/storm/pull/1831> and
JIRA<
> https://issues.apache.org/jira/browse/STORM-2249> address the missing
> jars problem that you mentioned. I had created it in December 2016,
but
> there is an ongoing discussion if we should indeed put the jars in the
> examples location or not.
>
> On a different note, this storm-kafka-client/KafkaSpout PR<
> https://github.com/apache/storm/pull/1924> fixes a bug with the
number of
> uncommitted offsets that is quite important. It is not a blocker, but
it is
> quite critical. I am going to do one last pass reviewing today. It
would be
> good if we could have this PR included with the release. Can anyone
else
> review it as well ?
>
> Thanks,
> Hugo
>
> On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <ptgoetz@gmail.com<mailto:
ptgo
> etz@gmail.com>> wrote:
>
> Yeah, I don’t think the file size is a killer/blocker. It’s largely
due to
> shaded examples, etc. But it’s something to keep an eye on. Our binary
> releases shouldn’t have to be that big.
>
> -Taylor
>
> On Mar 1, 2017, at 12:09 PM, Roshan Naik <roshan@hortonworks.com
<mailto:
> roshan@hortonworks.com>> wrote:
>
> Have filed Jiras so for the 3 issues mentioned. Not sure if we need a
JIRA
> for the file size getting bloated by that much.
> Somebody better familiar with the matter may want to take about that?
> -roshan
>
>
> On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com<mailto:ptgo
> etz@gmail.com>> wrote:
>
> Thanks for bringing these up Roshan. Feel free to file JIRA tickets
for
> these issues and assign the “Release Apache Storm 1.1.0” epic so they
can
> be tracked for this release.
>
> -Taylor
>
> On Mar 1, 2017, at 9:27 AM, Roshan Naik <roshan@hortonworks.com
<mailto:
> roshan@hortonworks.com>> wrote:
>
> Found these additional issues:
>
>
>
> 1- BUG: Even if topology.eventlogger.executors=0, the event_logger
bolt
> is instantiated … previously observed to cause ~10% degradation in
perf
> even with logging disabled.
>
> 2- Missing Jars: The storm-*-examples jars are missing in the binary
> distro (other than a storm-pmml-examples.jar, storm-jms-examples.jar &
> flux-examples.jar).
>
> 3- Minor: HdfsSpoutTopology example has not been moved into
> storm-hdfs-examples from storm-starter
>
>
>
> Another side observation … v1.0.3 tar.gz downloadable was 190MB.
This
> v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the
example
> topologies didn’t make it.
>
>
>
> -roshan
>
>
>
>
>
>
>
>
--
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Roshan Naik <ro...@hortonworks.com>.
Looks like the formatting on my prev email got dropped. But observation is that storm-kafka-monitor and storm-druid are the biggest new contributors.
-roshan
On 3/1/17, 5:23 PM, "Roshan Naik" <ro...@hortonworks.com> wrote:
Once all of the shaded examples are included the size will go up further.
But currently as they are not part of the tar.gz … something else is the culprit for the bloat.
Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases .. of files that are larger than 4MB.
@Jungtaek Lim<ma...@hortonworks.com> : I am thinking, since the code for the examples can be easily viewed online … it would be valuable to have the executable topologies made available to the user as part of the binary release … rather have them figure out how to build it correctly before trying them out.
-roshan
➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh {} \;
-rw-r--r--@ 1 roshan staff 70M Feb 7 12:33 ./examples/storm-starter/storm-starter-topologies-1.0.3.jar
-rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30 ./external/flux/flux-examples-1.0.3.jar
-rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32 ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
-rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32 ./external/sql/storm-sql-core/guava-16.0.1.jar
-rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30 ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-with-dependencies.jar
-rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33 ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
-rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33 ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
-rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar
-rw-r--r--@ 1 roshan staff 19M Feb 7 12:26 ./lib/storm-core-1.0.3.jar
-rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26 ./lib/storm-rename-hack-1.0.3.jar
➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh {} \;
-rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23 ./examples/storm-pmml-examples/storm-pmml-examples-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20 ./examples/storm-starter/storm-starter-topologies-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11 ./external/flux/flux-examples-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16 ./external/sql/storm-sql-core/calcite-core-1.11.0.jar
-rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16 ./external/sql/storm-sql-core/guava-16.0.1.jar
-rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12 ./external/sql/storm-sql-runtime/calcite-core-1.11.0.jar
-rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12 ./external/sql/storm-sql-runtime/guava-16.0.1.jar
-rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18 ./external/storm-druid/storm-druid-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11 ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-with-dependencies.jar
-rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20 ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
-rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18 ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar
-rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07 ./lib/storm-core-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07 ./lib/storm-rename-hack-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19 ./toollib/storm-kafka-monitor-1.1.0.jar
On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
About STORM-2249, since examples are shading their dependencies, binary
dist will grow much bigger. I've left some comments regarding that.
Btw, I have another view of this. Showing example codes is more important
than just let users execute some topologies. That's what example modules
are for. We need to include source as well. If we need to pick one, source
code would be better.
STORM-2343 seems better to add to 1.1.0. I just am not enough familiar with
storm-kafka-client so not sure I can review that, but I'll try to. I feel
it's not that make release dragged. Let's add to 1.1.0 epic.
Let's make minimum merge before another RC vote. Addressing STORM-2389 (and
maybe STORM-2343) is enough for me. Others are not that critical.
Thanks,
Jungtaek Lim (HeartSaVioR)
On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <hl...@hortonworks.com>
wrote:
> Roshan, does this PR<https://github.com/apache/storm/pull/1831> and JIRA<
> https://issues.apache.org/jira/browse/STORM-2249> address the missing
> jars problem that you mentioned. I had created it in December 2016, but
> there is an ongoing discussion if we should indeed put the jars in the
> examples location or not.
>
> On a different note, this storm-kafka-client/KafkaSpout PR<
> https://github.com/apache/storm/pull/1924> fixes a bug with the number of
> uncommitted offsets that is quite important. It is not a blocker, but it is
> quite critical. I am going to do one last pass reviewing today. It would be
> good if we could have this PR included with the release. Can anyone else
> review it as well ?
>
> Thanks,
> Hugo
>
> On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <ptgoetz@gmail.com<mailto:ptgo
> etz@gmail.com>> wrote:
>
> Yeah, I don’t think the file size is a killer/blocker. It’s largely due to
> shaded examples, etc. But it’s something to keep an eye on. Our binary
> releases shouldn’t have to be that big.
>
> -Taylor
>
> On Mar 1, 2017, at 12:09 PM, Roshan Naik <roshan@hortonworks.com<mailto:
> roshan@hortonworks.com>> wrote:
>
> Have filed Jiras so for the 3 issues mentioned. Not sure if we need a JIRA
> for the file size getting bloated by that much.
> Somebody better familiar with the matter may want to take about that?
> -roshan
>
>
> On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com<mailto:ptgo
> etz@gmail.com>> wrote:
>
> Thanks for bringing these up Roshan. Feel free to file JIRA tickets for
> these issues and assign the “Release Apache Storm 1.1.0” epic so they can
> be tracked for this release.
>
> -Taylor
>
> On Mar 1, 2017, at 9:27 AM, Roshan Naik <roshan@hortonworks.com<mailto:
> roshan@hortonworks.com>> wrote:
>
> Found these additional issues:
>
>
>
> 1- BUG: Even if topology.eventlogger.executors=0, the event_logger bolt
> is instantiated … previously observed to cause ~10% degradation in perf
> even with logging disabled.
>
> 2- Missing Jars: The storm-*-examples jars are missing in the binary
> distro (other than a storm-pmml-examples.jar, storm-jms-examples.jar &
> flux-examples.jar).
>
> 3- Minor: HdfsSpoutTopology example has not been moved into
> storm-hdfs-examples from storm-starter
>
>
>
> Another side observation … v1.0.3 tar.gz downloadable was 190MB. This
> v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the example
> topologies didn’t make it.
>
>
>
> -roshan
>
>
>
>
>
>
>
>
--
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Roshan Naik <ro...@hortonworks.com>.
Once all of the shaded examples are included the size will go up further.
But currently as they are not part of the tar.gz … something else is the culprit for the bloat.
Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases .. of files that are larger than 4MB.
@Jungtaek Lim<ma...@hortonworks.com> : I am thinking, since the code for the examples can be easily viewed online … it would be valuable to have the executable topologies made available to the user as part of the binary release … rather have them figure out how to build it correctly before trying them out.
-roshan
➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh {} \;
-rw-r--r--@ 1 roshan staff 70M Feb 7 12:33 ./examples/storm-starter/storm-starter-topologies-1.0.3.jar
-rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30 ./external/flux/flux-examples-1.0.3.jar
-rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32 ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
-rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32 ./external/sql/storm-sql-core/guava-16.0.1.jar
-rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30 ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-with-dependencies.jar
-rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33 ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
-rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33 ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
-rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar
-rw-r--r--@ 1 roshan staff 19M Feb 7 12:26 ./lib/storm-core-1.0.3.jar
-rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26 ./lib/storm-rename-hack-1.0.3.jar
➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh {} \;
-rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23 ./examples/storm-pmml-examples/storm-pmml-examples-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20 ./examples/storm-starter/storm-starter-topologies-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11 ./external/flux/flux-examples-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16 ./external/sql/storm-sql-core/calcite-core-1.11.0.jar
-rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16 ./external/sql/storm-sql-core/guava-16.0.1.jar
-rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12 ./external/sql/storm-sql-runtime/calcite-core-1.11.0.jar
-rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12 ./external/sql/storm-sql-runtime/guava-16.0.1.jar
-rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18 ./external/storm-druid/storm-druid-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11 ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-with-dependencies.jar
-rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20 ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar
-rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18 ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar
-rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07 ./lib/storm-core-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07 ./lib/storm-rename-hack-1.1.0.jar
-rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19 ./toollib/storm-kafka-monitor-1.1.0.jar
On 3/1/17, 4:43 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
About STORM-2249, since examples are shading their dependencies, binary
dist will grow much bigger. I've left some comments regarding that.
Btw, I have another view of this. Showing example codes is more important
than just let users execute some topologies. That's what example modules
are for. We need to include source as well. If we need to pick one, source
code would be better.
STORM-2343 seems better to add to 1.1.0. I just am not enough familiar with
storm-kafka-client so not sure I can review that, but I'll try to. I feel
it's not that make release dragged. Let's add to 1.1.0 epic.
Let's make minimum merge before another RC vote. Addressing STORM-2389 (and
maybe STORM-2343) is enough for me. Others are not that critical.
Thanks,
Jungtaek Lim (HeartSaVioR)
On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <hl...@hortonworks.com>
wrote:
> Roshan, does this PR<https://github.com/apache/storm/pull/1831> and JIRA<
> https://issues.apache.org/jira/browse/STORM-2249> address the missing
> jars problem that you mentioned. I had created it in December 2016, but
> there is an ongoing discussion if we should indeed put the jars in the
> examples location or not.
>
> On a different note, this storm-kafka-client/KafkaSpout PR<
> https://github.com/apache/storm/pull/1924> fixes a bug with the number of
> uncommitted offsets that is quite important. It is not a blocker, but it is
> quite critical. I am going to do one last pass reviewing today. It would be
> good if we could have this PR included with the release. Can anyone else
> review it as well ?
>
> Thanks,
> Hugo
>
> On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <ptgoetz@gmail.com<mailto:ptgo
> etz@gmail.com>> wrote:
>
> Yeah, I don’t think the file size is a killer/blocker. It’s largely due to
> shaded examples, etc. But it’s something to keep an eye on. Our binary
> releases shouldn’t have to be that big.
>
> -Taylor
>
> On Mar 1, 2017, at 12:09 PM, Roshan Naik <roshan@hortonworks.com<mailto:
> roshan@hortonworks.com>> wrote:
>
> Have filed Jiras so for the 3 issues mentioned. Not sure if we need a JIRA
> for the file size getting bloated by that much.
> Somebody better familiar with the matter may want to take about that?
> -roshan
>
>
> On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com<mailto:ptgo
> etz@gmail.com>> wrote:
>
> Thanks for bringing these up Roshan. Feel free to file JIRA tickets for
> these issues and assign the “Release Apache Storm 1.1.0” epic so they can
> be tracked for this release.
>
> -Taylor
>
> On Mar 1, 2017, at 9:27 AM, Roshan Naik <roshan@hortonworks.com<mailto:
> roshan@hortonworks.com>> wrote:
>
> Found these additional issues:
>
>
>
> 1- BUG: Even if topology.eventlogger.executors=0, the event_logger bolt
> is instantiated … previously observed to cause ~10% degradation in perf
> even with logging disabled.
>
> 2- Missing Jars: The storm-*-examples jars are missing in the binary
> distro (other than a storm-pmml-examples.jar, storm-jms-examples.jar &
> flux-examples.jar).
>
> 3- Minor: HdfsSpoutTopology example has not been moved into
> storm-hdfs-examples from storm-starter
>
>
>
> Another side observation … v1.0.3 tar.gz downloadable was 190MB. This
> v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the example
> topologies didn’t make it.
>
>
>
> -roshan
>
>
>
>
>
>
>
>
--
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Jungtaek Lim <ka...@gmail.com>.
About STORM-2249, since examples are shading their dependencies, binary
dist will grow much bigger. I've left some comments regarding that.
Btw, I have another view of this. Showing example codes is more important
than just let users execute some topologies. That's what example modules
are for. We need to include source as well. If we need to pick one, source
code would be better.
STORM-2343 seems better to add to 1.1.0. I just am not enough familiar with
storm-kafka-client so not sure I can review that, but I'll try to. I feel
it's not that make release dragged. Let's add to 1.1.0 epic.
Let's make minimum merge before another RC vote. Addressing STORM-2389 (and
maybe STORM-2343) is enough for me. Others are not that critical.
Thanks,
Jungtaek Lim (HeartSaVioR)
On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <hl...@hortonworks.com>
wrote:
> Roshan, does this PR<https://github.com/apache/storm/pull/1831> and JIRA<
> https://issues.apache.org/jira/browse/STORM-2249> address the missing
> jars problem that you mentioned. I had created it in December 2016, but
> there is an ongoing discussion if we should indeed put the jars in the
> examples location or not.
>
> On a different note, this storm-kafka-client/KafkaSpout PR<
> https://github.com/apache/storm/pull/1924> fixes a bug with the number of
> uncommitted offsets that is quite important. It is not a blocker, but it is
> quite critical. I am going to do one last pass reviewing today. It would be
> good if we could have this PR included with the release. Can anyone else
> review it as well ?
>
> Thanks,
> Hugo
>
> On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <ptgoetz@gmail.com<mailto:ptgo
> etz@gmail.com>> wrote:
>
> Yeah, I don’t think the file size is a killer/blocker. It’s largely due to
> shaded examples, etc. But it’s something to keep an eye on. Our binary
> releases shouldn’t have to be that big.
>
> -Taylor
>
> On Mar 1, 2017, at 12:09 PM, Roshan Naik <roshan@hortonworks.com<mailto:
> roshan@hortonworks.com>> wrote:
>
> Have filed Jiras so for the 3 issues mentioned. Not sure if we need a JIRA
> for the file size getting bloated by that much.
> Somebody better familiar with the matter may want to take about that?
> -roshan
>
>
> On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgoetz@gmail.com<mailto:ptgo
> etz@gmail.com>> wrote:
>
> Thanks for bringing these up Roshan. Feel free to file JIRA tickets for
> these issues and assign the “Release Apache Storm 1.1.0” epic so they can
> be tracked for this release.
>
> -Taylor
>
> On Mar 1, 2017, at 9:27 AM, Roshan Naik <roshan@hortonworks.com<mailto:
> roshan@hortonworks.com>> wrote:
>
> Found these additional issues:
>
>
>
> 1- BUG: Even if topology.eventlogger.executors=0, the event_logger bolt
> is instantiated … previously observed to cause ~10% degradation in perf
> even with logging disabled.
>
> 2- Missing Jars: The storm-*-examples jars are missing in the binary
> distro (other than a storm-pmml-examples.jar, storm-jms-examples.jar &
> flux-examples.jar).
>
> 3- Minor: HdfsSpoutTopology example has not been moved into
> storm-hdfs-examples from storm-starter
>
>
>
> Another side observation … v1.0.3 tar.gz downloadable was 190MB. This
> v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the example
> topologies didn’t make it.
>
>
>
> -roshan
>
>
>
>
>
>
>
>
--
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Hugo Da Cruz Louro <hl...@hortonworks.com>.
Roshan, does this PR<https://github.com/apache/storm/pull/1831> and JIRA<https://issues.apache.org/jira/browse/STORM-2249> address the missing jars problem that you mentioned. I had created it in December 2016, but there is an ongoing discussion if we should indeed put the jars in the examples location or not.
On a different note, this storm-kafka-client/KafkaSpout PR<https://github.com/apache/storm/pull/1924> fixes a bug with the number of uncommitted offsets that is quite important. It is not a blocker, but it is quite critical. I am going to do one last pass reviewing today. It would be good if we could have this PR included with the release. Can anyone else review it as well ?
Thanks,
Hugo
On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <pt...@gmail.com>> wrote:
Yeah, I don’t think the file size is a killer/blocker. It’s largely due to shaded examples, etc. But it’s something to keep an eye on. Our binary releases shouldn’t have to be that big.
-Taylor
On Mar 1, 2017, at 12:09 PM, Roshan Naik <ro...@hortonworks.com>> wrote:
Have filed Jiras so for the 3 issues mentioned. Not sure if we need a JIRA for the file size getting bloated by that much.
Somebody better familiar with the matter may want to take about that?
-roshan
On 3/1/17, 8:13 AM, "P. Taylor Goetz" <pt...@gmail.com>> wrote:
Thanks for bringing these up Roshan. Feel free to file JIRA tickets for these issues and assign the “Release Apache Storm 1.1.0” epic so they can be tracked for this release.
-Taylor
On Mar 1, 2017, at 9:27 AM, Roshan Naik <ro...@hortonworks.com>> wrote:
Found these additional issues:
1- BUG: Even if topology.eventlogger.executors=0, the event_logger bolt is instantiated … previously observed to cause ~10% degradation in perf even with logging disabled.
2- Missing Jars: The storm-*-examples jars are missing in the binary distro (other than a storm-pmml-examples.jar, storm-jms-examples.jar & flux-examples.jar).
3- Minor: HdfsSpoutTopology example has not been moved into storm-hdfs-examples from storm-starter
Another side observation … v1.0.3 tar.gz downloadable was 190MB. This v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the example topologies didn’t make it.
-roshan
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Yeah, I don’t think the file size is a killer/blocker. It’s largely due to shaded examples, etc. But it’s something to keep an eye on. Our binary releases shouldn’t have to be that big.
-Taylor
> On Mar 1, 2017, at 12:09 PM, Roshan Naik <ro...@hortonworks.com> wrote:
>
> Have filed Jiras so for the 3 issues mentioned. Not sure if we need a JIRA for the file size getting bloated by that much.
> Somebody better familiar with the matter may want to take about that?
> -roshan
>
>
> On 3/1/17, 8:13 AM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
>
> Thanks for bringing these up Roshan. Feel free to file JIRA tickets for these issues and assign the “Release Apache Storm 1.1.0” epic so they can be tracked for this release.
>
> -Taylor
>
>> On Mar 1, 2017, at 9:27 AM, Roshan Naik <ro...@hortonworks.com> wrote:
>>
>> Found these additional issues:
>>
>>
>>
>> 1- BUG: Even if topology.eventlogger.executors=0, the event_logger bolt is instantiated … previously observed to cause ~10% degradation in perf even with logging disabled.
>>
>> 2- Missing Jars: The storm-*-examples jars are missing in the binary distro (other than a storm-pmml-examples.jar, storm-jms-examples.jar & flux-examples.jar).
>>
>> 3- Minor: HdfsSpoutTopology example has not been moved into storm-hdfs-examples from storm-starter
>>
>>
>>
>> Another side observation … v1.0.3 tar.gz downloadable was 190MB. This v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the example topologies didn’t make it.
>>
>>
>>
>> -roshan
>
>
>
>
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Roshan Naik <ro...@hortonworks.com>.
Have filed Jiras so for the 3 issues mentioned. Not sure if we need a JIRA for the file size getting bloated by that much.
Somebody better familiar with the matter may want to take about that?
-roshan
On 3/1/17, 8:13 AM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
Thanks for bringing these up Roshan. Feel free to file JIRA tickets for these issues and assign the “Release Apache Storm 1.1.0” epic so they can be tracked for this release.
-Taylor
> On Mar 1, 2017, at 9:27 AM, Roshan Naik <ro...@hortonworks.com> wrote:
>
> Found these additional issues:
>
>
>
> 1- BUG: Even if topology.eventlogger.executors=0, the event_logger bolt is instantiated … previously observed to cause ~10% degradation in perf even with logging disabled.
>
> 2- Missing Jars: The storm-*-examples jars are missing in the binary distro (other than a storm-pmml-examples.jar, storm-jms-examples.jar & flux-examples.jar).
>
> 3- Minor: HdfsSpoutTopology example has not been moved into storm-hdfs-examples from storm-starter
>
>
>
> Another side observation … v1.0.3 tar.gz downloadable was 190MB. This v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the example topologies didn’t make it.
>
>
>
> -roshan
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Thanks for bringing these up Roshan. Feel free to file JIRA tickets for these issues and assign the “Release Apache Storm 1.1.0” epic so they can be tracked for this release.
-Taylor
> On Mar 1, 2017, at 9:27 AM, Roshan Naik <ro...@hortonworks.com> wrote:
>
> Found these additional issues:
>
>
>
> 1- BUG: Even if topology.eventlogger.executors=0, the event_logger bolt is instantiated … previously observed to cause ~10% degradation in perf even with logging disabled.
>
> 2- Missing Jars: The storm-*-examples jars are missing in the binary distro (other than a storm-pmml-examples.jar, storm-jms-examples.jar & flux-examples.jar).
>
> 3- Minor: HdfsSpoutTopology example has not been moved into storm-hdfs-examples from storm-starter
>
>
>
> Another side observation … v1.0.3 tar.gz downloadable was 190MB. This v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the example topologies didn’t make it.
>
>
>
> -roshan
Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Roshan Naik <ro...@hortonworks.com>.
Found these additional issues:
1- BUG: Even if topology.eventlogger.executors=0, the event_logger bolt is instantiated … previously observed to cause ~10% degradation in perf even with logging disabled.
2- Missing Jars: The storm-*-examples jars are missing in the binary distro (other than a storm-pmml-examples.jar, storm-jms-examples.jar & flux-examples.jar).
3- Minor: HdfsSpoutTopology example has not been moved into storm-hdfs-examples from storm-starter
Another side observation … v1.0.3 tar.gz downloadable was 190MB. This v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the example topologies didn’t make it.
-roshan
[CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by "P. Taylor Goetz" <pt...@apache.org>.
Canceling to include fix for JDK7 compatibility issue.
-Taylor
> On Feb 24, 2017, at 3:40 PM, P. Taylor Goetz <pt...@apache.org> wrote:
>
> This is a call to vote on releasing Apache Storm 1.1.0 (rc2)
>
> Full list of changes in this release:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=17a2017fb644e353fb2a0f5bf50d400ee28036ba
>
> The tag/commit to be voted upon is v1.1.0:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob;f=CHANGELOG.md;h=0352f6d27dad1c88c24bb99fd894a211ea6c1e71;hb=7c8ea09e09160d36bba7e363ed53781816b46597
>
> The source archive being voted upon can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc2/apache-storm-1.1.0-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc2/
>
> The release artifacts are signed with the following key:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1046
>
> Please vote on releasing this package as Apache Storm 1.1.0.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.1.0
> [ ] 0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
Re: [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by S G <sg...@gmail.com>.
Zookeeper version being used by Storm is 3.4.6 and it has a pretty serious
bug when hostnames are used instead of IPs.
The fix for the same is pretty small (See the PRs in
https://issues.apache.org/jira/browse/STORM-2290)
It would be good to include this for 1.1
-SG
On Tue, Feb 28, 2017 at 1:27 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> I just found that source doesn't compile against JDK 7.
> (Unfortunately I missed building source with JDK7. It was built with JDK8.)
>
> I filed STORM-2388 <https://issues.apache.org/jira/browse/STORM-2388> and
> submitted a PR <https://github.com/apache/storm/pull/1977> for this.
>
> Since Storm 1.x should be guaranteed to be compatible with JDK 7, I think
> it's blocker.
> I'm changing my vote to -1 (binding).
>
> We might need to have code-freeze period for 1.1.0 release.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 2월 28일 (화) 오전 8:33, Priyank Shah <ps...@hortonworks.com>님이 작성:
>
> > +1(non-binding)
> >
> > - Verified signatures, md5s and shas for all files
> > - Mvn clean install on source successsully
> > - Started daemons and killed them successfully for both source and binary
> > - Submitted WodCountTopology and FastWordCountTopology for both as well
> > - Tried out different commands from UI like Deactivate, Activate,
> > Rebalance and Kill
> > - All of test runs were done on single node cluster on my local machine
> > with each daemon running in its own jvm
> >
> >
> > On 2/27/17, 12:26 PM, "Alexandre Vermeerbergen" <
> avermeerbergen@gmail.com>
> > wrote:
> >
> > Hello,
> >
> > +1 (non-binding)
> >
> > My first attempt to deploy our Supervision app (made of 10 topologies
> > with
> > a Redis server for sharing memory) intto a small Storm 1.1.0rc2
> > cluster (1
> > nimbus node, 2 supervisors, 3 zookeeper VMs) failed because
> operations
> > on
> > Nimbus VM were taking too long.
> >
> > I realized that this was because my Nimbus VM was a "m1.small"
> > (provided by
> > OUTSCALE Europe 2 datacenter) and it was too slow. I have the
> > impression
> > that this setup was OK with Storm 1.0.2, but I have no proofs. I
> > re-did my
> > 1.1.0rc2 with a m1.medium VM for Nimbus and everything works like a
> > charm.
> >
> > One detail: on our Kafka spouts, I got the following message instead
> > of a
> > lag information:
> >
> >
> > Offset lags for kafka not supported for older versions. Please update
> > kafka
> > spout to latest version.
> > Is there some documentation which provides the clear requirements to
> > get
> > Kafka monitoring from Nimbus UI?
> > PS: I'm using Kafka Spout (not yet migrated to Kafka Client Spout
> with
> > Kafka 0.9.01 libraries.
> >
> > Best regards,
> > Alexandre
> >
> >
> >
> > 2017-02-27 14:13 GMT+01:00 Alexandre Vermeerbergen <
> > avermeerbergen@gmail.com
> > >:
> >
> > > Hello,
> > >
> > > Tried with our supervision topologies on a small cluster (1 Nimbus
> > VM, 3
> > > Zookeeper VMs, 2 Supervisor VMs).
> > >
> > > For the moment I have trouble with Nimbus & Nimbus UI startup time,
> > the
> > > "bin/storm nimbus" starts the following process with runs for quite
> > a long
> > > time (5 to 10 minutes) before the actual Nimbus process (which has
> a
> > "java
> > > -server" pattern instead of "java -client" in ps aux list) is
> > actually
> > > started:
> > >
> > > root 29178 81.0 0.9 1965584 18404 pts/0 Sl 13:04 0:00
> > > /usr/local/jdk/bin/java -client -Dstorm.options= -Dstorm.conf.file=
> > -cp
> > > /usr/local/Storm/storm-stable/lib/servlet-api-2.5.jar:/usr/l
> > > ocal/Storm/storm-stable/lib/log4j-slf4j-impl-2.8.jar:/usr/lo
> > > cal/Storm/storm-stable/lib/minlog-1.3.0.jar:/usr/local/Storm
> > > /storm-stable/lib/storm-rename-hack-1.1.0.jar:/usr/local/
> > > Storm/storm-stable/lib/slf4j-api-1.7.21.jar:/usr/local/
> > > Storm/storm-stable/lib/disruptor-3.3.2.jar:/usr/local/Storm/
> > > storm-stable/lib/log4j-core-2.8.jar:/usr/local/Storm/storm-
> > > stable/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/Storm/
> > > storm-stable/lib/reflectasm-1.10.1.jar:/usr/local/Storm/
> > > storm-stable/lib/ring-cors-0.1.5.jar:/usr/local/Storm/
> > > storm-stable/lib/clojure-1.7.0.jar:/usr/local/Storm/storm-
> > > stable/lib/kryo-3.0.3.jar:/usr/local/Storm/storm-stable/
> > > lib/asm-5.0.3.jar:/usr/local/Storm/storm-stable/lib/
> > > objenesis-2.1.jar:/usr/local/Storm/storm-stable/lib/log4j-
> > > api-2.8.jar:/usr/local/Storm/storm-stable/lib/storm-core-1.
> > > 1.0.jar:/usr/local/Storm/storm-stable/extlib/flux-
> > > wrappers-1.1.0.jar:/usr/local/Storm/storm-stable/extlib/
> > > flux-core-1.1.0.jar:/usr/local/Storm/storm-stable/conf
> > > org.apache.storm.command.config_value java.library.path
> > >
> > > I have no logs generated by this first process.
> > > Is there a way to turn on some logs to understand what takes so
> long?
> > >
> > > Best regards,
> > > Alexandre
> > >
> > >
> > > 2017-02-24 21:40 GMT+01:00 P. Taylor Goetz <pt...@apache.org>:
> > >
> > >> This is a call to vote on releasing Apache Storm 1.1.0 (rc2)
> > >>
> > >> Full list of changes in this release:
> > >>
> > >> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_p
> > >> lain;f=CHANGELOG.md;hb=17a2017fb644e353fb2a0f5bf50d400ee28036ba
> > >>
> > >> The tag/commit to be voted upon is v1.1.0:
> > >>
> > >> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob;f
> > >> =CHANGELOG.md;h=0352f6d27dad1c88c24bb99fd894a211ea6c1e71;hb=
> > >> 7c8ea09e09160d36bba7e363ed53781816b46597
> > >>
> > >> The source archive being voted upon can be found here:
> > >>
> > >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > >> 1.0-rc2/apache-storm-1.1.0-src.tar.gz
> > >>
> > >> Other release files, signatures and digests can be found here:
> > >>
> > >>
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc2/
> > >>
> > >> The release artifacts are signed with the following key:
> > >>
> > >> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_p
> > >> lain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > >>
> > >> The Nexus staging repository for this release is:
> > >>
> > >>
> > https://repository.apache.org/content/repositories/orgapachestorm-1046
> > >>
> > >> Please vote on releasing this package as Apache Storm 1.1.0.
> > >>
> > >> When voting, please list the actions taken to verify the release.
> > >>
> > >> This vote will be open for at least 72 hours.
> > >>
> > >> [ ] +1 Release this package as Apache Storm 1.1.0
> > >> [ ] 0 No opinion
> > >> [ ] -1 Do not release this package because...
> > >>
> > >> Thanks to everyone who contributed to this release.
> > >>
> > >> -Taylor
> > >
> > >
> > >
> >
> >
> >
>
Re: [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Jungtaek Lim <ka...@gmail.com>.
I just found that source doesn't compile against JDK 7.
(Unfortunately I missed building source with JDK7. It was built with JDK8.)
I filed STORM-2388 <https://issues.apache.org/jira/browse/STORM-2388> and
submitted a PR <https://github.com/apache/storm/pull/1977> for this.
Since Storm 1.x should be guaranteed to be compatible with JDK 7, I think
it's blocker.
I'm changing my vote to -1 (binding).
We might need to have code-freeze period for 1.1.0 release.
- Jungtaek Lim (HeartSaVioR)
2017년 2월 28일 (화) 오전 8:33, Priyank Shah <ps...@hortonworks.com>님이 작성:
> +1(non-binding)
>
> - Verified signatures, md5s and shas for all files
> - Mvn clean install on source successsully
> - Started daemons and killed them successfully for both source and binary
> - Submitted WodCountTopology and FastWordCountTopology for both as well
> - Tried out different commands from UI like Deactivate, Activate,
> Rebalance and Kill
> - All of test runs were done on single node cluster on my local machine
> with each daemon running in its own jvm
>
>
> On 2/27/17, 12:26 PM, "Alexandre Vermeerbergen" <av...@gmail.com>
> wrote:
>
> Hello,
>
> +1 (non-binding)
>
> My first attempt to deploy our Supervision app (made of 10 topologies
> with
> a Redis server for sharing memory) intto a small Storm 1.1.0rc2
> cluster (1
> nimbus node, 2 supervisors, 3 zookeeper VMs) failed because operations
> on
> Nimbus VM were taking too long.
>
> I realized that this was because my Nimbus VM was a "m1.small"
> (provided by
> OUTSCALE Europe 2 datacenter) and it was too slow. I have the
> impression
> that this setup was OK with Storm 1.0.2, but I have no proofs. I
> re-did my
> 1.1.0rc2 with a m1.medium VM for Nimbus and everything works like a
> charm.
>
> One detail: on our Kafka spouts, I got the following message instead
> of a
> lag information:
>
>
> Offset lags for kafka not supported for older versions. Please update
> kafka
> spout to latest version.
> Is there some documentation which provides the clear requirements to
> get
> Kafka monitoring from Nimbus UI?
> PS: I'm using Kafka Spout (not yet migrated to Kafka Client Spout with
> Kafka 0.9.01 libraries.
>
> Best regards,
> Alexandre
>
>
>
> 2017-02-27 14:13 GMT+01:00 Alexandre Vermeerbergen <
> avermeerbergen@gmail.com
> >:
>
> > Hello,
> >
> > Tried with our supervision topologies on a small cluster (1 Nimbus
> VM, 3
> > Zookeeper VMs, 2 Supervisor VMs).
> >
> > For the moment I have trouble with Nimbus & Nimbus UI startup time,
> the
> > "bin/storm nimbus" starts the following process with runs for quite
> a long
> > time (5 to 10 minutes) before the actual Nimbus process (which has a
> "java
> > -server" pattern instead of "java -client" in ps aux list) is
> actually
> > started:
> >
> > root 29178 81.0 0.9 1965584 18404 pts/0 Sl 13:04 0:00
> > /usr/local/jdk/bin/java -client -Dstorm.options= -Dstorm.conf.file=
> -cp
> > /usr/local/Storm/storm-stable/lib/servlet-api-2.5.jar:/usr/l
> > ocal/Storm/storm-stable/lib/log4j-slf4j-impl-2.8.jar:/usr/lo
> > cal/Storm/storm-stable/lib/minlog-1.3.0.jar:/usr/local/Storm
> > /storm-stable/lib/storm-rename-hack-1.1.0.jar:/usr/local/
> > Storm/storm-stable/lib/slf4j-api-1.7.21.jar:/usr/local/
> > Storm/storm-stable/lib/disruptor-3.3.2.jar:/usr/local/Storm/
> > storm-stable/lib/log4j-core-2.8.jar:/usr/local/Storm/storm-
> > stable/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/Storm/
> > storm-stable/lib/reflectasm-1.10.1.jar:/usr/local/Storm/
> > storm-stable/lib/ring-cors-0.1.5.jar:/usr/local/Storm/
> > storm-stable/lib/clojure-1.7.0.jar:/usr/local/Storm/storm-
> > stable/lib/kryo-3.0.3.jar:/usr/local/Storm/storm-stable/
> > lib/asm-5.0.3.jar:/usr/local/Storm/storm-stable/lib/
> > objenesis-2.1.jar:/usr/local/Storm/storm-stable/lib/log4j-
> > api-2.8.jar:/usr/local/Storm/storm-stable/lib/storm-core-1.
> > 1.0.jar:/usr/local/Storm/storm-stable/extlib/flux-
> > wrappers-1.1.0.jar:/usr/local/Storm/storm-stable/extlib/
> > flux-core-1.1.0.jar:/usr/local/Storm/storm-stable/conf
> > org.apache.storm.command.config_value java.library.path
> >
> > I have no logs generated by this first process.
> > Is there a way to turn on some logs to understand what takes so long?
> >
> > Best regards,
> > Alexandre
> >
> >
> > 2017-02-24 21:40 GMT+01:00 P. Taylor Goetz <pt...@apache.org>:
> >
> >> This is a call to vote on releasing Apache Storm 1.1.0 (rc2)
> >>
> >> Full list of changes in this release:
> >>
> >> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_p
> >> lain;f=CHANGELOG.md;hb=17a2017fb644e353fb2a0f5bf50d400ee28036ba
> >>
> >> The tag/commit to be voted upon is v1.1.0:
> >>
> >> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob;f
> >> =CHANGELOG.md;h=0352f6d27dad1c88c24bb99fd894a211ea6c1e71;hb=
> >> 7c8ea09e09160d36bba7e363ed53781816b46597
> >>
> >> The source archive being voted upon can be found here:
> >>
> >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> >> 1.0-rc2/apache-storm-1.1.0-src.tar.gz
> >>
> >> Other release files, signatures and digests can be found here:
> >>
> >>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc2/
> >>
> >> The release artifacts are signed with the following key:
> >>
> >> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_p
> >> lain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >>
> >> The Nexus staging repository for this release is:
> >>
> >>
> https://repository.apache.org/content/repositories/orgapachestorm-1046
> >>
> >> Please vote on releasing this package as Apache Storm 1.1.0.
> >>
> >> When voting, please list the actions taken to verify the release.
> >>
> >> This vote will be open for at least 72 hours.
> >>
> >> [ ] +1 Release this package as Apache Storm 1.1.0
> >> [ ] 0 No opinion
> >> [ ] -1 Do not release this package because...
> >>
> >> Thanks to everyone who contributed to this release.
> >>
> >> -Taylor
> >
> >
> >
>
>
>
Re: [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Priyank Shah <ps...@hortonworks.com>.
+1(non-binding)
- Verified signatures, md5s and shas for all files
- Mvn clean install on source successsully
- Started daemons and killed them successfully for both source and binary
- Submitted WodCountTopology and FastWordCountTopology for both as well
- Tried out different commands from UI like Deactivate, Activate, Rebalance and Kill
- All of test runs were done on single node cluster on my local machine with each daemon running in its own jvm
On 2/27/17, 12:26 PM, "Alexandre Vermeerbergen" <av...@gmail.com> wrote:
Hello,
+1 (non-binding)
My first attempt to deploy our Supervision app (made of 10 topologies with
a Redis server for sharing memory) intto a small Storm 1.1.0rc2 cluster (1
nimbus node, 2 supervisors, 3 zookeeper VMs) failed because operations on
Nimbus VM were taking too long.
I realized that this was because my Nimbus VM was a "m1.small" (provided by
OUTSCALE Europe 2 datacenter) and it was too slow. I have the impression
that this setup was OK with Storm 1.0.2, but I have no proofs. I re-did my
1.1.0rc2 with a m1.medium VM for Nimbus and everything works like a charm.
One detail: on our Kafka spouts, I got the following message instead of a
lag information:
Offset lags for kafka not supported for older versions. Please update kafka
spout to latest version.
Is there some documentation which provides the clear requirements to get
Kafka monitoring from Nimbus UI?
PS: I'm using Kafka Spout (not yet migrated to Kafka Client Spout with
Kafka 0.9.01 libraries.
Best regards,
Alexandre
2017-02-27 14:13 GMT+01:00 Alexandre Vermeerbergen <avermeerbergen@gmail.com
>:
> Hello,
>
> Tried with our supervision topologies on a small cluster (1 Nimbus VM, 3
> Zookeeper VMs, 2 Supervisor VMs).
>
> For the moment I have trouble with Nimbus & Nimbus UI startup time, the
> "bin/storm nimbus" starts the following process with runs for quite a long
> time (5 to 10 minutes) before the actual Nimbus process (which has a "java
> -server" pattern instead of "java -client" in ps aux list) is actually
> started:
>
> root 29178 81.0 0.9 1965584 18404 pts/0 Sl 13:04 0:00
> /usr/local/jdk/bin/java -client -Dstorm.options= -Dstorm.conf.file= -cp
> /usr/local/Storm/storm-stable/lib/servlet-api-2.5.jar:/usr/l
> ocal/Storm/storm-stable/lib/log4j-slf4j-impl-2.8.jar:/usr/lo
> cal/Storm/storm-stable/lib/minlog-1.3.0.jar:/usr/local/Storm
> /storm-stable/lib/storm-rename-hack-1.1.0.jar:/usr/local/
> Storm/storm-stable/lib/slf4j-api-1.7.21.jar:/usr/local/
> Storm/storm-stable/lib/disruptor-3.3.2.jar:/usr/local/Storm/
> storm-stable/lib/log4j-core-2.8.jar:/usr/local/Storm/storm-
> stable/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/Storm/
> storm-stable/lib/reflectasm-1.10.1.jar:/usr/local/Storm/
> storm-stable/lib/ring-cors-0.1.5.jar:/usr/local/Storm/
> storm-stable/lib/clojure-1.7.0.jar:/usr/local/Storm/storm-
> stable/lib/kryo-3.0.3.jar:/usr/local/Storm/storm-stable/
> lib/asm-5.0.3.jar:/usr/local/Storm/storm-stable/lib/
> objenesis-2.1.jar:/usr/local/Storm/storm-stable/lib/log4j-
> api-2.8.jar:/usr/local/Storm/storm-stable/lib/storm-core-1.
> 1.0.jar:/usr/local/Storm/storm-stable/extlib/flux-
> wrappers-1.1.0.jar:/usr/local/Storm/storm-stable/extlib/
> flux-core-1.1.0.jar:/usr/local/Storm/storm-stable/conf
> org.apache.storm.command.config_value java.library.path
>
> I have no logs generated by this first process.
> Is there a way to turn on some logs to understand what takes so long?
>
> Best regards,
> Alexandre
>
>
> 2017-02-24 21:40 GMT+01:00 P. Taylor Goetz <pt...@apache.org>:
>
>> This is a call to vote on releasing Apache Storm 1.1.0 (rc2)
>>
>> Full list of changes in this release:
>>
>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_p
>> lain;f=CHANGELOG.md;hb=17a2017fb644e353fb2a0f5bf50d400ee28036ba
>>
>> The tag/commit to be voted upon is v1.1.0:
>>
>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob;f
>> =CHANGELOG.md;h=0352f6d27dad1c88c24bb99fd894a211ea6c1e71;hb=
>> 7c8ea09e09160d36bba7e363ed53781816b46597
>>
>> The source archive being voted upon can be found here:
>>
>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
>> 1.0-rc2/apache-storm-1.1.0-src.tar.gz
>>
>> Other release files, signatures and digests can be found here:
>>
>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc2/
>>
>> The release artifacts are signed with the following key:
>>
>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_p
>> lain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>>
>> The Nexus staging repository for this release is:
>>
>> https://repository.apache.org/content/repositories/orgapachestorm-1046
>>
>> Please vote on releasing this package as Apache Storm 1.1.0.
>>
>> When voting, please list the actions taken to verify the release.
>>
>> This vote will be open for at least 72 hours.
>>
>> [ ] +1 Release this package as Apache Storm 1.1.0
>> [ ] 0 No opinion
>> [ ] -1 Do not release this package because...
>>
>> Thanks to everyone who contributed to this release.
>>
>> -Taylor
>
>
>
Re: [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Alexandre Vermeerbergen <av...@gmail.com>.
Hello,
+1 (non-binding)
My first attempt to deploy our Supervision app (made of 10 topologies with
a Redis server for sharing memory) intto a small Storm 1.1.0rc2 cluster (1
nimbus node, 2 supervisors, 3 zookeeper VMs) failed because operations on
Nimbus VM were taking too long.
I realized that this was because my Nimbus VM was a "m1.small" (provided by
OUTSCALE Europe 2 datacenter) and it was too slow. I have the impression
that this setup was OK with Storm 1.0.2, but I have no proofs. I re-did my
1.1.0rc2 with a m1.medium VM for Nimbus and everything works like a charm.
One detail: on our Kafka spouts, I got the following message instead of a
lag information:
Offset lags for kafka not supported for older versions. Please update kafka
spout to latest version.
Is there some documentation which provides the clear requirements to get
Kafka monitoring from Nimbus UI?
PS: I'm using Kafka Spout (not yet migrated to Kafka Client Spout with
Kafka 0.9.01 libraries.
Best regards,
Alexandre
2017-02-27 14:13 GMT+01:00 Alexandre Vermeerbergen <avermeerbergen@gmail.com
>:
> Hello,
>
> Tried with our supervision topologies on a small cluster (1 Nimbus VM, 3
> Zookeeper VMs, 2 Supervisor VMs).
>
> For the moment I have trouble with Nimbus & Nimbus UI startup time, the
> "bin/storm nimbus" starts the following process with runs for quite a long
> time (5 to 10 minutes) before the actual Nimbus process (which has a "java
> -server" pattern instead of "java -client" in ps aux list) is actually
> started:
>
> root 29178 81.0 0.9 1965584 18404 pts/0 Sl 13:04 0:00
> /usr/local/jdk/bin/java -client -Dstorm.options= -Dstorm.conf.file= -cp
> /usr/local/Storm/storm-stable/lib/servlet-api-2.5.jar:/usr/l
> ocal/Storm/storm-stable/lib/log4j-slf4j-impl-2.8.jar:/usr/lo
> cal/Storm/storm-stable/lib/minlog-1.3.0.jar:/usr/local/Storm
> /storm-stable/lib/storm-rename-hack-1.1.0.jar:/usr/local/
> Storm/storm-stable/lib/slf4j-api-1.7.21.jar:/usr/local/
> Storm/storm-stable/lib/disruptor-3.3.2.jar:/usr/local/Storm/
> storm-stable/lib/log4j-core-2.8.jar:/usr/local/Storm/storm-
> stable/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/Storm/
> storm-stable/lib/reflectasm-1.10.1.jar:/usr/local/Storm/
> storm-stable/lib/ring-cors-0.1.5.jar:/usr/local/Storm/
> storm-stable/lib/clojure-1.7.0.jar:/usr/local/Storm/storm-
> stable/lib/kryo-3.0.3.jar:/usr/local/Storm/storm-stable/
> lib/asm-5.0.3.jar:/usr/local/Storm/storm-stable/lib/
> objenesis-2.1.jar:/usr/local/Storm/storm-stable/lib/log4j-
> api-2.8.jar:/usr/local/Storm/storm-stable/lib/storm-core-1.
> 1.0.jar:/usr/local/Storm/storm-stable/extlib/flux-
> wrappers-1.1.0.jar:/usr/local/Storm/storm-stable/extlib/
> flux-core-1.1.0.jar:/usr/local/Storm/storm-stable/conf
> org.apache.storm.command.config_value java.library.path
>
> I have no logs generated by this first process.
> Is there a way to turn on some logs to understand what takes so long?
>
> Best regards,
> Alexandre
>
>
> 2017-02-24 21:40 GMT+01:00 P. Taylor Goetz <pt...@apache.org>:
>
>> This is a call to vote on releasing Apache Storm 1.1.0 (rc2)
>>
>> Full list of changes in this release:
>>
>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_p
>> lain;f=CHANGELOG.md;hb=17a2017fb644e353fb2a0f5bf50d400ee28036ba
>>
>> The tag/commit to be voted upon is v1.1.0:
>>
>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob;f
>> =CHANGELOG.md;h=0352f6d27dad1c88c24bb99fd894a211ea6c1e71;hb=
>> 7c8ea09e09160d36bba7e363ed53781816b46597
>>
>> The source archive being voted upon can be found here:
>>
>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
>> 1.0-rc2/apache-storm-1.1.0-src.tar.gz
>>
>> Other release files, signatures and digests can be found here:
>>
>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc2/
>>
>> The release artifacts are signed with the following key:
>>
>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_p
>> lain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>>
>> The Nexus staging repository for this release is:
>>
>> https://repository.apache.org/content/repositories/orgapachestorm-1046
>>
>> Please vote on releasing this package as Apache Storm 1.1.0.
>>
>> When voting, please list the actions taken to verify the release.
>>
>> This vote will be open for at least 72 hours.
>>
>> [ ] +1 Release this package as Apache Storm 1.1.0
>> [ ] 0 No opinion
>> [ ] -1 Do not release this package because...
>>
>> Thanks to everyone who contributed to this release.
>>
>> -Taylor
>
>
>
Re: [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
+1 I rebuilt from the tag, all of the unit tests passed. I ran a small cluster and all of my simple tests passed there too.
Alexandre,
When the storm script is run it needs to read some config options. Sadly we have not reproduced the same process in python so to do this is run a special command to read a config option. This one is trying to read java.library.path so it can set up the command line properly. I have no idea why it would be taking so long. Usually it finishes very quickly. If it is reproducible for you, you could take a jstack of it, there should only be one thread, and see where it is stuck. My guess is that it is trying to read in a file, simply because that is all that this code really does.
- Bobby
On Monday, February 27, 2017, 7:13:28 AM CST, Alexandre Vermeerbergen <av...@gmail.com> wrote:Hello,
Tried with our supervision topologies on a small cluster (1 Nimbus VM, 3
Zookeeper VMs, 2 Supervisor VMs).
For the moment I have trouble with Nimbus & Nimbus UI startup time, the
"bin/storm nimbus" starts the following process with runs for quite a long
time (5 to 10 minutes) before the actual Nimbus process (which has a "java
-server" pattern instead of "java -client" in ps aux list) is actually
started:
root 29178 81.0 0.9 1965584 18404 pts/0 Sl 13:04 0:00
/usr/local/jdk/bin/java -client -Dstorm.options= -Dstorm.conf.file= -cp
/usr/local/Storm/storm-stable/lib/servlet-api-2.5.jar:/usr/
local/Storm/storm-stable/lib/log4j-slf4j-impl-2.8.jar:/usr/
local/Storm/storm-stable/lib/minlog-1.3.0.jar:/usr/local/
Storm/storm-stable/lib/storm-rename-hack-1.1.0.jar:/usr/
local/Storm/storm-stable/lib/slf4j-api-1.7.21.jar:/usr/
local/Storm/storm-stable/lib/disruptor-3.3.2.jar:/usr/
local/Storm/storm-stable/lib/log4j-core-2.8.jar:/usr/local/
Storm/storm-stable/lib/log4j-over-slf4j-1.6.6.jar:/usr/
local/Storm/storm-stable/lib/reflectasm-1.10.1.jar:/usr/
local/Storm/storm-stable/lib/ring-cors-0.1.5.jar:/usr/
local/Storm/storm-stable/lib/clojure-1.7.0.jar:/usr/local/
Storm/storm-stable/lib/kryo-3.0.3.jar:/usr/local/Storm/
storm-stable/lib/asm-5.0.3.jar:/usr/local/Storm/storm-
stable/lib/objenesis-2.1.jar:/usr/local/Storm/storm-stable/
lib/log4j-api-2.8.jar:/usr/local/Storm/storm-stable/lib/
storm-core-1.1.0.jar:/usr/local/Storm/storm-stable/
extlib/flux-wrappers-1.1.0.jar:/usr/local/Storm/storm-
stable/extlib/flux-core-1.1.0.jar:/usr/local/Storm/storm-stable/conf
org.apache.storm.command.config_value java.library.path
I have no logs generated by this first process.
Is there a way to turn on some logs to understand what takes so long?
Best regards,
Alexandre
2017-02-24 21:40 GMT+01:00 P. Taylor Goetz <pt...@apache.org>:
> This is a call to vote on releasing Apache Storm 1.1.0 (rc2)
>
> Full list of changes in this release:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> plain;f=CHANGELOG.md;hb=17a2017fb644e353fb2a0f5bf50d400ee28036ba
>
> The tag/commit to be voted upon is v1.1.0:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob;
> f=CHANGELOG.md;h=0352f6d27dad1c88c24bb99fd894a211ea6c1e71;hb=
> 7c8ea09e09160d36bba7e363ed53781816b46597
>
> The source archive being voted upon can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> 1.0-rc2/apache-storm-1.1.0-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc2/
>
> The release artifacts are signed with the following key:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1046
>
> Please vote on releasing this package as Apache Storm 1.1.0.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.1.0
> [ ] 0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
Re: [VOTE] Release Apache Storm 1.1.0 (RC2)
Posted by Alexandre Vermeerbergen <av...@gmail.com>.
Hello,
Tried with our supervision topologies on a small cluster (1 Nimbus VM, 3
Zookeeper VMs, 2 Supervisor VMs).
For the moment I have trouble with Nimbus & Nimbus UI startup time, the
"bin/storm nimbus" starts the following process with runs for quite a long
time (5 to 10 minutes) before the actual Nimbus process (which has a "java
-server" pattern instead of "java -client" in ps aux list) is actually
started:
root 29178 81.0 0.9 1965584 18404 pts/0 Sl 13:04 0:00
/usr/local/jdk/bin/java -client -Dstorm.options= -Dstorm.conf.file= -cp
/usr/local/Storm/storm-stable/lib/servlet-api-2.5.jar:/usr/
local/Storm/storm-stable/lib/log4j-slf4j-impl-2.8.jar:/usr/
local/Storm/storm-stable/lib/minlog-1.3.0.jar:/usr/local/
Storm/storm-stable/lib/storm-rename-hack-1.1.0.jar:/usr/
local/Storm/storm-stable/lib/slf4j-api-1.7.21.jar:/usr/
local/Storm/storm-stable/lib/disruptor-3.3.2.jar:/usr/
local/Storm/storm-stable/lib/log4j-core-2.8.jar:/usr/local/
Storm/storm-stable/lib/log4j-over-slf4j-1.6.6.jar:/usr/
local/Storm/storm-stable/lib/reflectasm-1.10.1.jar:/usr/
local/Storm/storm-stable/lib/ring-cors-0.1.5.jar:/usr/
local/Storm/storm-stable/lib/clojure-1.7.0.jar:/usr/local/
Storm/storm-stable/lib/kryo-3.0.3.jar:/usr/local/Storm/
storm-stable/lib/asm-5.0.3.jar:/usr/local/Storm/storm-
stable/lib/objenesis-2.1.jar:/usr/local/Storm/storm-stable/
lib/log4j-api-2.8.jar:/usr/local/Storm/storm-stable/lib/
storm-core-1.1.0.jar:/usr/local/Storm/storm-stable/
extlib/flux-wrappers-1.1.0.jar:/usr/local/Storm/storm-
stable/extlib/flux-core-1.1.0.jar:/usr/local/Storm/storm-stable/conf
org.apache.storm.command.config_value java.library.path
I have no logs generated by this first process.
Is there a way to turn on some logs to understand what takes so long?
Best regards,
Alexandre
2017-02-24 21:40 GMT+01:00 P. Taylor Goetz <pt...@apache.org>:
> This is a call to vote on releasing Apache Storm 1.1.0 (rc2)
>
> Full list of changes in this release:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> plain;f=CHANGELOG.md;hb=17a2017fb644e353fb2a0f5bf50d400ee28036ba
>
> The tag/commit to be voted upon is v1.1.0:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob;
> f=CHANGELOG.md;h=0352f6d27dad1c88c24bb99fd894a211ea6c1e71;hb=
> 7c8ea09e09160d36bba7e363ed53781816b46597
>
> The source archive being voted upon can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> 1.0-rc2/apache-storm-1.1.0-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc2/
>
> The release artifacts are signed with the following key:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1046
>
> Please vote on releasing this package as Apache Storm 1.1.0.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.1.0
> [ ] 0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor