You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Shinhyung Yang <sh...@yonsei.ac.kr> on 2018/02/26 16:35:36 UTC
Upgrading from 0.9.7 to 1.2.0
Dear Storm users and developers,
I have been running the Yahoo streaming benchmarks on Storm 0.9.7. The
entire Cloud infrastructure includes Zookeeper 3.4.6 on 3 nodes, 5 Kafka
2.10-0.8.2.1 nodes, 1 Storm 0.9.7 Nimbus node, 10 Storm 0.9.7 supervisor
nodes with 4 worker processes on each node, 10 kafka producer nodes, and 1
redis 3.0.5 node.
With the introduction of Storm 1.2.0, I decided to upgrade from 0.9.7 to
1.2.0. Currently I'm testing Yahoo streaming benchmark's topology on the
new setup and I end up getting the following exceptions:
#
streaming-benchmarks/apache-storm-1.2.0/logs/workers-artifacts/test-topo-1-1519628010/6700/worker.log
148 java.lang.RuntimeException:
org.apache.zookeeper.KeeperException$UnimplementedException:
KeeperErrorCode = Unimplemented for
/ad-events/7183b5b2-4971-41a1-b86d-0788f646bc64/partition_0
149 at org.apache.storm.kafka.ZkState.writeBytes(ZkState.java:84)
~[stormjar.jar:?]
150 at org.apache.storm.kafka.ZkState.writeJSON(ZkState.java:70)
~[stormjar.jar:?]
151 at
org.apache.storm.kafka.PartitionManager.commit(PartitionManager.java:361)
~[stormjar.jar:?]
152 at org.apache.storm.kafka.KafkaSpout.commit(KafkaSpout.java:257)
~[stormjar.jar:?]
153 at
org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:156)
~[stormjar.jar:?]
154 at
storm.benchmark.AdvertisingTopology$MyKafkaSpout.nextTuple(AdvertisingTopology.java:181)
~[stormjar.jar:?]
155 at
org.apache.storm.daemon.executor$fn__9610$fn__9625$fn__9656.invoke(executor.clj:654)
~[storm-core-1.2.0.jar:1.2.0]
156 at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
[storm-core-1.2.0.jar:1.2.0]
157 at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
158 at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
159 Caused by:
org.apache.zookeeper.KeeperException$UnimplementedException:
KeeperErrorCode = Unimplemented for
/ad-events/7183b5b2-4971-41a1-b86d-0788f646bc64/partition_0
160 at
org.apache.zookeeper.KeeperException.create(KeeperException.java:103)
~[stormjar.jar:?]
161 at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
~[stormjar.jar:?]
162 at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1525)
~[stormjar.jar:?]
163 at
org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1189)
~[stormjar.jar:?]
164 at
org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1166)
~[stormjar.jar:?]
165 at
org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64)
~[stormjar.jar:?]
166 at
org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100)
~[stormjar.jar:?]
167 at
org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:1163)
~[stormjar.jar:?]
168 at
org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:603)
~[stormjar.jar:?]
169 at
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:593)
~[stormjar.jar:?]
170 at
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:47)
~[stormjar.jar:?]
171 at org.apache.storm.kafka.ZkState.writeBytes(ZkState.java:79)
~[stormjar.jar:?]
172 ... 9 more
For each instance of benchmark execution, more than one of the spout
instances will eventually generate exception messages in their log files
like above.
Some people suggest that it's probably the compatibility issue between
Zookeeper and Curator:
https://stackoverflow.com/a/35753999
> Apparently this error can occur if you're using the wrong combination of
> Curator in combination with Zookeeper. From curator.apache.org :
>> Curator 2.x.x - compatible with both ZooKeeper 3.4.x and ZooKeeper 3.5.x
>> Curator 3.x.x - compatible only with ZooKeeper 3.5.x and includes
>> support for new features such as dynamic reconfiguration, etc.
http://curator.apache.org
> Apache Curator is meant to be used with ZooKeeper 3.5+. However, it
> is also compatible with ZooKeeper 3.4.x. See Compatibility for details.
http://curator.apache.org/zk-compatibility.html
> While ZooKeeper 3.5.x is still considered "beta" by the ZooKeeper
> development team, the reality is that it is used in production by many
users.
> However, ZooKeeper 3.4.x is also used in production. Prior to Apache
Curator 4.0,
> both versions of ZooKeeper were supported via two versions of Apache
Curator.
> Starting with Curator 4.0 both versions of ZooKeeper are supported via
> the same Curator libraries.
>
> ZooKeeper 3.5.x
> Curator 4.0 has a hard dependency on ZooKeeper 3.5.x
> If you are using ZooKeeper 3.5.x there's nothing additional to do -
> just use Curator 4.0
>
> ZooKeeper 3.4.x
> Curator 4.0 supports ZooKeeper 3.4.x ensembles in a soft-compatibility
> mode. To use this mode you must exclude ZooKeeper when adding
> Curator to your dependency management tool.
Could it be possible that something went wrong when Curator 4.0.0 is
working against Zookeeper 3.4.6? (I'm really wondering about this
soft-compatibility mode introduced in Curator 4.0)
Thank you for reading.
Best regards,
Shinhyung Yang
Re: Upgrading from 0.9.7 to 1.2.0
Posted by Shinhyung Yang <sh...@gmail.com>.
Dear Stig Rohde,
Thank you for fast reply!
In my pom, I added the following lines and now the topology runs as it
always did:
60 <dependency>
61 <groupId>org.apache.zookeeper</groupId>
62 <artifactId>zookeeper</artifactId>
63 <version>3.4.6</version>
64 <exclusions>
65 <exclusion>
66 <groupId>org.slf4j</groupId>
67 <artifactId>slf4j-log4j12</artifactId>
68 </exclusion>
69 </exclusions>
70 </dependency>
Also, thank you for creating STORM-2978. I'm following the issue now.
Thank you and best regards,
Shinhyung Yang
-
On Tue, Feb 27, 2018 at 8:14 AM, Stig Rohde Døssing <sr...@apache.org> wrote:
> Hi Shinhyung,
>
> I looked into this, it's a bug. Zookeeper 3.5.3-beta is ending up on the
> classpath, which was not the intent. I've raised
> https://issues.apache.org/jira/browse/STORM-2978 to track fixing the issue.
> For now, you can work around it by adding an explicit dependency on
> Zookeeper 3.4.6 to your pom.
>
> 2018-02-26 17:35 GMT+01:00 Shinhyung Yang <sh...@yonsei.ac.kr>:
>>
>> Dear Storm users and developers,
>>
>> I have been running the Yahoo streaming benchmarks on Storm 0.9.7. The
>> entire Cloud infrastructure includes Zookeeper 3.4.6 on 3 nodes, 5 Kafka
>> 2.10-0.8.2.1 nodes, 1 Storm 0.9.7 Nimbus node, 10 Storm 0.9.7 supervisor
>> nodes with 4 worker processes on each node, 10 kafka producer nodes, and 1
>> redis 3.0.5 node.
>>
>> With the introduction of Storm 1.2.0, I decided to upgrade from 0.9.7 to
>> 1.2.0. Currently I'm testing Yahoo streaming benchmark's topology on the new
>> setup and I end up getting the following exceptions:
>>
>> #
>> streaming-benchmarks/apache-storm-1.2.0/logs/workers-artifacts/test-topo-1-1519628010/6700/worker.log
>>
>> 148 java.lang.RuntimeException:
>> org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode
>> = Unimplemented for
>> /ad-events/7183b5b2-4971-41a1-b86d-0788f646bc64/partition_0
>> 149 at org.apache.storm.kafka.ZkState.writeBytes(ZkState.java:84)
>> ~[stormjar.jar:?]
>> 150 at org.apache.storm.kafka.ZkState.writeJSON(ZkState.java:70)
>> ~[stormjar.jar:?]
>> 151 at
>> org.apache.storm.kafka.PartitionManager.commit(PartitionManager.java:361)
>> ~[stormjar.jar:?]
>> 152 at
>> org.apache.storm.kafka.KafkaSpout.commit(KafkaSpout.java:257)
>> ~[stormjar.jar:?]
>> 153 at
>> org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:156)
>> ~[stormjar.jar:?]
>> 154 at
>> storm.benchmark.AdvertisingTopology$MyKafkaSpout.nextTuple(AdvertisingTopology.java:181)
>> ~[stormjar.jar:?]
>> 155 at
>> org.apache.storm.daemon.executor$fn__9610$fn__9625$fn__9656.invoke(executor.clj:654)
>> ~[storm-core-1.2.0.jar:1.2.0]
>> 156 at
>> org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
>> [storm-core-1.2.0.jar:1.2.0]
>> 157 at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>> 158 at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
>> 159 Caused by:
>> org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode
>> = Unimplemented for
>> /ad-events/7183b5b2-4971-41a1-b86d-0788f646bc64/partition_0
>> 160 at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:103)
>> ~[stormjar.jar:?]
>> 161 at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> ~[stormjar.jar:?]
>> 162 at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1525)
>> ~[stormjar.jar:?]
>> 163 at
>> org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1189)
>> ~[stormjar.jar:?]
>> 164 at
>> org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1166)
>> ~[stormjar.jar:?]
>> 165 at
>> org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64)
>> ~[stormjar.jar:?]
>> 166 at
>> org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100)
>> ~[stormjar.jar:?]
>> 167 at
>> org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:1163)
>> ~[stormjar.jar:?]
>> 168 at
>> org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:603)
>> ~[stormjar.jar:?]
>> 169 at
>> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:593)
>> ~[stormjar.jar:?]
>> 170 at
>> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:47)
>> ~[stormjar.jar:?]
>> 171 at org.apache.storm.kafka.ZkState.writeBytes(ZkState.java:79)
>> ~[stormjar.jar:?]
>> 172 ... 9 more
>>
>> For each instance of benchmark execution, more than one of the spout
>> instances will eventually generate exception messages in their log files
>> like above.
>>
>> Some people suggest that it's probably the compatibility issue between
>> Zookeeper and Curator:
>>
>> https://stackoverflow.com/a/35753999
>> > Apparently this error can occur if you're using the wrong combination of
>> > Curator in combination with Zookeeper. From curator.apache.org :
>> >> Curator 2.x.x - compatible with both ZooKeeper 3.4.x and ZooKeeper
>> >> 3.5.x
>> >> Curator 3.x.x - compatible only with ZooKeeper 3.5.x and includes
>> >> support for new features such as dynamic reconfiguration, etc.
>>
>> http://curator.apache.org
>> > Apache Curator is meant to be used with ZooKeeper 3.5+. However, it
>> > is also compatible with ZooKeeper 3.4.x. See Compatibility for details.
>>
>> http://curator.apache.org/zk-compatibility.html
>> > While ZooKeeper 3.5.x is still considered "beta" by the ZooKeeper
>> > development team, the reality is that it is used in production by many
>> > users.
>> > However, ZooKeeper 3.4.x is also used in production. Prior to Apache
>> > Curator 4.0,
>> > both versions of ZooKeeper were supported via two versions of Apache
>> > Curator.
>> > Starting with Curator 4.0 both versions of ZooKeeper are supported via
>> > the same Curator libraries.
>> >
>> > ZooKeeper 3.5.x
>> > Curator 4.0 has a hard dependency on ZooKeeper 3.5.x
>> > If you are using ZooKeeper 3.5.x there's nothing additional to do -
>> > just use Curator 4.0
>> >
>> > ZooKeeper 3.4.x
>> > Curator 4.0 supports ZooKeeper 3.4.x ensembles in a soft-compatibility
>> > mode. To use this mode you must exclude ZooKeeper when adding
>> > Curator to your dependency management tool.
>>
>>
>> Could it be possible that something went wrong when Curator 4.0.0 is
>> working against Zookeeper 3.4.6? (I'm really wondering about this
>> soft-compatibility mode introduced in Curator 4.0)
>>
>> Thank you for reading.
>>
>> Best regards,
>> Shinhyung Yang
>>
>>
>
Re: Upgrading from 0.9.7 to 1.2.0
Posted by Stig Rohde Døssing <sr...@apache.org>.
Hi Shinhyung,
I looked into this, it's a bug. Zookeeper 3.5.3-beta is ending up on the
classpath, which was not the intent. I've raised
https://issues.apache.org/jira/browse/STORM-2978 to track fixing the issue.
For now, you can work around it by adding an explicit dependency on
Zookeeper 3.4.6 to your pom.
2018-02-26 17:35 GMT+01:00 Shinhyung Yang <sh...@yonsei.ac.kr>:
> Dear Storm users and developers,
>
> I have been running the Yahoo streaming benchmarks on Storm 0.9.7. The
> entire Cloud infrastructure includes Zookeeper 3.4.6 on 3 nodes, 5 Kafka
> 2.10-0.8.2.1 nodes, 1 Storm 0.9.7 Nimbus node, 10 Storm 0.9.7 supervisor
> nodes with 4 worker processes on each node, 10 kafka producer nodes, and 1
> redis 3.0.5 node.
>
> With the introduction of Storm 1.2.0, I decided to upgrade from 0.9.7 to
> 1.2.0. Currently I'm testing Yahoo streaming benchmark's topology on the
> new setup and I end up getting the following exceptions:
>
> # streaming-benchmarks/apache-storm-1.2.0/logs/workers-
> artifacts/test-topo-1-1519628010/6700/worker.log
>
> 148 java.lang.RuntimeException: org.apache.zookeeper.KeeperException$UnimplementedException:
> KeeperErrorCode = Unimplemented for /ad-events/7183b5b2-4971-41a1-
> b86d-0788f646bc64/partition_0
> 149 at org.apache.storm.kafka.ZkState.writeBytes(ZkState.java:84)
> ~[stormjar.jar:?]
> 150 at org.apache.storm.kafka.ZkState.writeJSON(ZkState.java:70)
> ~[stormjar.jar:?]
> 151 at org.apache.storm.kafka.PartitionManager.commit(PartitionManager.java:361)
> ~[stormjar.jar:?]
> 152 at org.apache.storm.kafka.KafkaSpout.commit(KafkaSpout.java:257)
> ~[stormjar.jar:?]
> 153 at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:156)
> ~[stormjar.jar:?]
> 154 at storm.benchmark.AdvertisingTopology$MyKafkaSpout.nextTuple(AdvertisingTopology.java:181)
> ~[stormjar.jar:?]
> 155 at org.apache.storm.daemon.executor$fn__9610$fn__9625$fn__9656.invoke(executor.clj:654)
> ~[storm-core-1.2.0.jar:1.2.0]
> 156 at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
> [storm-core-1.2.0.jar:1.2.0]
> 157 at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> 158 at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
> 159 Caused by: org.apache.zookeeper.KeeperException$UnimplementedException:
> KeeperErrorCode = Unimplemented for /ad-events/7183b5b2-4971-41a1-
> b86d-0788f646bc64/partition_0
> 160 at org.apache.zookeeper.KeeperException.create(KeeperException.java:103)
> ~[stormjar.jar:?]
> 161 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> ~[stormjar.jar:?]
> 162 at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1525)
> ~[stormjar.jar:?]
> 163 at org.apache.curator.framework.imps.CreateBuilderImpl$17.
> call(CreateBuilderImpl.java:1189) ~[stormjar.jar:?]
> 164 at org.apache.curator.framework.imps.CreateBuilderImpl$17.
> call(CreateBuilderImpl.java:1166) ~[stormjar.jar:?]
> 165 at org.apache.curator.connection.StandardConnectionHandlingPoli
> cy.callWithRetry(StandardConnectionHandlingPolicy.java:64)
> ~[stormjar.jar:?]
> 166 at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100)
> ~[stormjar.jar:?]
> 167 at org.apache.curator.framework.imps.CreateBuilderImpl.
> pathInForeground(CreateBuilderImpl.java:1163) ~[stormjar.jar:?]
> 168 at org.apache.curator.framework.imps.CreateBuilderImpl.
> protectedPathInForeground(CreateBuilderImpl.java:603) ~[stormjar.jar:?]
> 169 at org.apache.curator.framework.imps.CreateBuilderImpl.
> forPath(CreateBuilderImpl.java:593) ~[stormjar.jar:?]
> 170 at org.apache.curator.framework.imps.CreateBuilderImpl.
> forPath(CreateBuilderImpl.java:47) ~[stormjar.jar:?]
> 171 at org.apache.storm.kafka.ZkState.writeBytes(ZkState.java:79)
> ~[stormjar.jar:?]
> 172 ... 9 more
>
> For each instance of benchmark execution, more than one of the spout
> instances will eventually generate exception messages in their log files
> like above.
>
> Some people suggest that it's probably the compatibility issue between
> Zookeeper and Curator:
>
> https://stackoverflow.com/a/35753999
> > Apparently this error can occur if you're using the wrong combination of
> > Curator in combination with Zookeeper. From curator.apache.org :
> >> Curator 2.x.x - compatible with both ZooKeeper 3.4.x and ZooKeeper 3.5.x
> >> Curator 3.x.x - compatible only with ZooKeeper 3.5.x and includes
> >> support for new features such as dynamic reconfiguration, etc.
>
> http://curator.apache.org
> > Apache Curator is meant to be used with ZooKeeper 3.5+. However, it
> > is also compatible with ZooKeeper 3.4.x. See Compatibility for details.
>
> http://curator.apache.org/zk-compatibility.html
> > While ZooKeeper 3.5.x is still considered "beta" by the ZooKeeper
> > development team, the reality is that it is used in production by many
> users.
> > However, ZooKeeper 3.4.x is also used in production. Prior to Apache
> Curator 4.0,
> > both versions of ZooKeeper were supported via two versions of Apache
> Curator.
> > Starting with Curator 4.0 both versions of ZooKeeper are supported via
> > the same Curator libraries.
> >
> > ZooKeeper 3.5.x
> > Curator 4.0 has a hard dependency on ZooKeeper 3.5.x
> > If you are using ZooKeeper 3.5.x there's nothing additional to do -
> > just use Curator 4.0
> >
> > ZooKeeper 3.4.x
> > Curator 4.0 supports ZooKeeper 3.4.x ensembles in a soft-compatibility
> > mode. To use this mode you must exclude ZooKeeper when adding
> > Curator to your dependency management tool.
>
>
> Could it be possible that something went wrong when Curator 4.0.0 is
> working against Zookeeper 3.4.6? (I'm really wondering about this
> soft-compatibility mode introduced in Curator 4.0)
>
> Thank you for reading.
>
> Best regards,
> Shinhyung Yang
>
>
>