You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Abhishek Raj <ab...@saavn.com> on 2016/10/06 07:03:07 UTC
Error when running pacemaker.
Hello, I am using storm 1.0.2. When I start the pacemaker daemon and then
start nimbus, I see the following error in pacemaker.log repetitively and
can't submit topologies.
WARNING: An exception was thrown by a user handler while handling an
>> exception event ([id: 0xa4126833, /127.0.0.1:43693 => /127.0.0.1:6699]
>> EXCEPTION: java.lang.NoSuchMethodError:
>> java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;)
>
> java.lang.NoSuchMethodError:
>> java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
>
> at
>> org.apache.storm.pacemaker.pacemaker$get_all_nodes_for_path.invoke(pacemaker.clj:133)
>
> at
>> org.apache.storm.pacemaker.pacemaker$mk_handler$reify__4023.handleMessage(pacemaker.clj:212)
>
> at
>> org.apache.storm.pacemaker.PacemakerServer.received(PacemakerServer.java:140)
>
> at
>> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
>
> at
>> org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>
> at
>> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>
> at
>> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>
>
>> o.a.s.m.n.StormServerHandler [ERROR] server errors in handling the request
>
> java.lang.NoSuchMethodError:
>> java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
>
> at
>> org.apache.storm.pacemaker.pacemaker$get_all_nodes_for_path.invoke(pacemaker.clj:133)
>> ~[storm-core-1.0.2.jar:1.0.2]
>
> at
>> org.apache.storm.pacemaker.pacemaker$mk_handler$reify__4023.handleMessage(pacemaker.clj:212)
>> ~[storm-core-1.0.2.jar:1.0.2]
>
> at
>> org.apache.storm.pacemaker.PacemakerServer.received(PacemakerServer.java:140)
>> ~[storm-core-1.0.2.jar:1.0.2]
>
> at
>> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
>> ~[storm-core-1.0.2.jar:1.0.2]
>
> at
>> org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>> ~[storm-core-1.0.2.jar:1.0.2]
>
> at
>> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> [storm-core-1.0.2.jar:1.0.2]
>
> at
>> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>> [storm-core-1.0.2.jar:1.0.2]
>
> at
>> org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>> [storm-core-1.0.2.jar:1.0.2]
>
>
I investigated a bit and this
<https://gist.github.com/AlainODea/1375759b8720a3f9f094> seems to be the
cause. Can anyone please help?
Thanks
--
Abhishek
Re: Error when running pacemaker.
Posted by Jungtaek Lim <ka...@gmail.com>.
Sorry to respond a bit late. I just found this.
I don't know pacemaker leaves log for receiving/sending, but since
heartbeat was written to ZK (unless you enable Pacemaker), you can check ZK
to not updating heartbeat. If it is, and your UI shows updated metrics,
then Pacemaker is working well. (since for now we include metrics into
heartbeat)
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 10월 7일 (금) 오후 10:09, Abhishek Raj <ab...@saavn.com>님이 작성:
> Hi Jungtaek, thanks for the response.
>
> 1. My JDK version is 1.7.0_65.
> 2. The error went away on building with JDK 7 and using the custom jar
> but
> 3. The error also went away when I built with JDK 8 and used the
> custom jar.
>
> I used JDK 7 in both cases for all the daemons. I am not really sure
> what's going on. Also, when the pacemaker successfully started in both
> cases I could see
>
> o.a.s.p.pacemaker [INFO] Starting pacemaker server for storm version
> '1.0.2'
>
> o.a.s.p.PacemakerServer [INFO] Bound server to port: 6699
>
>
> But no other info was logged when I submitted a topology. So I am not sure
> if pacemaker is actually processing worker heartbeats. Is there a way to
> confirm that pacemaker is actually working?
>
> Thanks
>
> On Thu, Oct 6, 2016 at 3:20 PM, Jungtaek Lim <ka...@gmail.com> wrote:
>
> Hi Abhishek,
>
> I haven't see this behavior before.
>
> 1. What's your JDK version?
>
> 2. If you don't mind, could you confirm that your link is the cause?
> - build Storm 1.0.2 source code by `mvn clean install -DskipTests` with
> JDK 7
> - replace storm-core jar in lib directory with custom built jar
> - run pacemaker and nimbus with JDK 7
> - see it resolves the issue
>
> 3. If you also don't mind, could you reproduce this behavior from the
> beginning?
> - build Storm 1.0.2 source code by `mvn clean install -DskipTests` with
> JDK 8
> - replace storm-core jar in lib directory with custom built jar
> - run pacemaker and nimbus with JDK 7
> - see it reappears
>
> If it works perfectly, I guess that's the cause. If then please file an
> issue so that we can track it.
>
> Please let me know you'd like me to test this.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> 2016년 10월 6일 (목) 오후 4:03, Abhishek Raj <ab...@saavn.com>님이 작성:
>
> Hello, I am using storm 1.0.2. When I start the pacemaker daemon and then
> start nimbus, I see the following error in pacemaker.log repetitively and
> can't submit topologies.
>
> WARNING: An exception was thrown by a user handler while handling an
> exception event ([id: 0xa4126833, /127.0.0.1:43693 => /127.0.0.1:6699]
> EXCEPTION: java.lang.NoSuchMethodError:
> java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;)
>
> java.lang.NoSuchMethodError:
> java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
>
> at
> org.apache.storm.pacemaker.pacemaker$get_all_nodes_for_path.invoke(pacemaker.clj:133)
>
> at
> org.apache.storm.pacemaker.pacemaker$mk_handler$reify__4023.handleMessage(pacemaker.clj:212)
>
> at
> org.apache.storm.pacemaker.PacemakerServer.received(PacemakerServer.java:140)
>
> at
> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
>
> at org.apache.storm.shade.org
> .jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>
> at org.apache.storm.shade.org
> .jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>
> at org.apache.storm.shade.org
> .jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>
>
> o.a.s.m.n.StormServerHandler [ERROR] server errors in handling the request
>
> java.lang.NoSuchMethodError:
> java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
>
> at
> org.apache.storm.pacemaker.pacemaker$get_all_nodes_for_path.invoke(pacemaker.clj:133)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at
> org.apache.storm.pacemaker.pacemaker$mk_handler$reify__4023.handleMessage(pacemaker.clj:212)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at
> org.apache.storm.pacemaker.PacemakerServer.received(PacemakerServer.java:140)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at
> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> [storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> [storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> [storm-core-1.0.2.jar:1.0.2]
>
>
> I investigated a bit and this
> <https://gist.github.com/AlainODea/1375759b8720a3f9f094> seems to be the
> cause. Can anyone please help?
>
> Thanks
>
> --
> Abhishek
>
>
>
>
> --
> Abhishek
>
Re: Error when running pacemaker.
Posted by Abhishek Raj <ab...@saavn.com>.
Hi Jungtaek, thanks for the response.
1. My JDK version is 1.7.0_65.
2. The error went away on building with JDK 7 and using the custom jar
but
3. The error also went away when I built with JDK 8 and used the custom
jar.
I used JDK 7 in both cases for all the daemons. I am not really sure what's
going on. Also, when the pacemaker successfully started in both cases I
could see
> o.a.s.p.pacemaker [INFO] Starting pacemaker server for storm version
> '1.0.2'
>
o.a.s.p.PacemakerServer [INFO] Bound server to port: 6699
But no other info was logged when I submitted a topology. So I am not sure
if pacemaker is actually processing worker heartbeats. Is there a way to
confirm that pacemaker is actually working?
Thanks
On Thu, Oct 6, 2016 at 3:20 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> Hi Abhishek,
>
> I haven't see this behavior before.
>
> 1. What's your JDK version?
>
> 2. If you don't mind, could you confirm that your link is the cause?
> - build Storm 1.0.2 source code by `mvn clean install -DskipTests` with
> JDK 7
> - replace storm-core jar in lib directory with custom built jar
> - run pacemaker and nimbus with JDK 7
> - see it resolves the issue
>
> 3. If you also don't mind, could you reproduce this behavior from the
> beginning?
> - build Storm 1.0.2 source code by `mvn clean install -DskipTests` with
> JDK 8
> - replace storm-core jar in lib directory with custom built jar
> - run pacemaker and nimbus with JDK 7
> - see it reappears
>
> If it works perfectly, I guess that's the cause. If then please file an
> issue so that we can track it.
>
> Please let me know you'd like me to test this.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> 2016년 10월 6일 (목) 오후 4:03, Abhishek Raj <ab...@saavn.com>님이 작성:
>
>> Hello, I am using storm 1.0.2. When I start the pacemaker daemon and then
>> start nimbus, I see the following error in pacemaker.log repetitively and
>> can't submit topologies.
>>
>> WARNING: An exception was thrown by a user handler while handling an
>>>> exception event ([id: 0xa4126833, /127.0.0.1:43693 => /127.0.0.1:6699]
>>>> EXCEPTION: java.lang.NoSuchMethodError: java.util.concurrent.
>>>> ConcurrentHashMap.keySet()Ljava/util/concurrent/
>>>> ConcurrentHashMap$KeySetView;)
>>>
>>> java.lang.NoSuchMethodError: java.util.concurrent.
>>>> ConcurrentHashMap.keySet()Ljava/util/concurrent/
>>>> ConcurrentHashMap$KeySetView;
>>>
>>> at org.apache.storm.pacemaker.pacemaker$get_all_nodes_for_
>>>> path.invoke(pacemaker.clj:133)
>>>
>>> at org.apache.storm.pacemaker.pacemaker$mk_handler$reify__
>>>> 4023.handleMessage(pacemaker.clj:212)
>>>
>>> at org.apache.storm.pacemaker.PacemakerServer.received(
>>>> PacemakerServer.java:140)
>>>
>>> at org.apache.storm.messaging.netty.StormServerHandler.
>>>> messageReceived(StormServerHandler.java:61)
>>>
>>> at org.apache.storm.shade.org.jboss.netty.channel.
>>>> SimpleChannelUpstreamHandler.handleUpstream(
>>>> SimpleChannelUpstreamHandler.java:70)
>>>
>>> at org.apache.storm.shade.org.jboss.netty.channel.
>>>> DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>
>>> at org.apache.storm.shade.org.jboss.netty.channel.
>>>> DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(
>>>> DefaultChannelPipeline.java:791)
>>>
>>>
>>>> o.a.s.m.n.StormServerHandler [ERROR] server errors in handling the
>>>> request
>>>
>>> java.lang.NoSuchMethodError: java.util.concurrent.
>>>> ConcurrentHashMap.keySet()Ljava/util/concurrent/
>>>> ConcurrentHashMap$KeySetView;
>>>
>>> at org.apache.storm.pacemaker.pacemaker$get_all_nodes_for_path.invoke(pacemaker.clj:133)
>>>> ~[storm-core-1.0.2.jar:1.0.2]
>>>
>>> at org.apache.storm.pacemaker.pacemaker$mk_handler$reify__
>>>> 4023.handleMessage(pacemaker.clj:212) ~[storm-core-1.0.2.jar:1.0.2]
>>>
>>> at org.apache.storm.pacemaker.PacemakerServer.received(PacemakerServer.java:140)
>>>> ~[storm-core-1.0.2.jar:1.0.2]
>>>
>>> at org.apache.storm.messaging.netty.StormServerHandler.
>>>> messageReceived(StormServerHandler.java:61)
>>>> ~[storm-core-1.0.2.jar:1.0.2]
>>>
>>> at org.apache.storm.shade.org.jboss.netty.channel.
>>>> SimpleChannelUpstreamHandler.handleUpstream(
>>>> SimpleChannelUpstreamHandler.java:70) ~[storm-core-1.0.2.jar:1.0.2]
>>>
>>> at org.apache.storm.shade.org.jboss.netty.channel.
>>>> DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>> [storm-core-1.0.2.jar:1.0.2]
>>>
>>> at org.apache.storm.shade.org.jboss.netty.channel.
>>>> DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(
>>>> DefaultChannelPipeline.java:791) [storm-core-1.0.2.jar:1.0.2]
>>>
>>> at org.apache.storm.shade.org.jboss.netty.channel.Channels.
>>>> fireMessageReceived(Channels.java:296) [storm-core-1.0.2.jar:1.0.2]
>>>
>>>
>> I investigated a bit and this
>> <https://gist.github.com/AlainODea/1375759b8720a3f9f094> seems to be the
>> cause. Can anyone please help?
>>
>> Thanks
>>
>> --
>> Abhishek
>>
>
--
Abhishek
Re: Error when running pacemaker.
Posted by Jungtaek Lim <ka...@gmail.com>.
Hi Abhishek,
I haven't see this behavior before.
1. What's your JDK version?
2. If you don't mind, could you confirm that your link is the cause?
- build Storm 1.0.2 source code by `mvn clean install -DskipTests` with JDK
7
- replace storm-core jar in lib directory with custom built jar
- run pacemaker and nimbus with JDK 7
- see it resolves the issue
3. If you also don't mind, could you reproduce this behavior from the
beginning?
- build Storm 1.0.2 source code by `mvn clean install -DskipTests` with JDK
8
- replace storm-core jar in lib directory with custom built jar
- run pacemaker and nimbus with JDK 7
- see it reappears
If it works perfectly, I guess that's the cause. If then please file an
issue so that we can track it.
Please let me know you'd like me to test this.
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 10월 6일 (목) 오후 4:03, Abhishek Raj <ab...@saavn.com>님이 작성:
> Hello, I am using storm 1.0.2. When I start the pacemaker daemon and then
> start nimbus, I see the following error in pacemaker.log repetitively and
> can't submit topologies.
>
> WARNING: An exception was thrown by a user handler while handling an
> exception event ([id: 0xa4126833, /127.0.0.1:43693 => /127.0.0.1:6699]
> EXCEPTION: java.lang.NoSuchMethodError:
> java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;)
>
> java.lang.NoSuchMethodError:
> java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
>
> at
> org.apache.storm.pacemaker.pacemaker$get_all_nodes_for_path.invoke(pacemaker.clj:133)
>
> at
> org.apache.storm.pacemaker.pacemaker$mk_handler$reify__4023.handleMessage(pacemaker.clj:212)
>
> at
> org.apache.storm.pacemaker.PacemakerServer.received(PacemakerServer.java:140)
>
> at
> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
>
> at
> org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>
> at
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>
> at
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>
>
> o.a.s.m.n.StormServerHandler [ERROR] server errors in handling the request
>
> java.lang.NoSuchMethodError:
> java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
>
> at
> org.apache.storm.pacemaker.pacemaker$get_all_nodes_for_path.invoke(pacemaker.clj:133)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at
> org.apache.storm.pacemaker.pacemaker$mk_handler$reify__4023.handleMessage(pacemaker.clj:212)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at
> org.apache.storm.pacemaker.PacemakerServer.received(PacemakerServer.java:140)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at
> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at
> org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> [storm-core-1.0.2.jar:1.0.2]
>
> at
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> [storm-core-1.0.2.jar:1.0.2]
>
> at
> org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> [storm-core-1.0.2.jar:1.0.2]
>
>
> I investigated a bit and this
> <https://gist.github.com/AlainODea/1375759b8720a3f9f094> seems to be the
> cause. Can anyone please help?
>
> Thanks
>
> --
> Abhishek
>