You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Nick Beenham <ni...@gmail.com> on 2014/10/09 20:58:24 UTC
What appears to be a Kryo Conflict
Hi all,
I have a peculiar problem...
I'm using a kafka storm combo to feed updates to my Titan Graph db. The
topology runs quite happily when local but as soon as I try to run it on a
cluster I run into Kryo errors.
2014-10-09 18:41:47 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException:
Encountered unregistered class ID: 10
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: com.esotericsoftware.kryo.KryoException: Encountered
unregistered class ID: 10
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)
~[kryo-2.21.jar:na]
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610)
~[kryo-2.21.jar:na]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721)
~[kryo-2.21.jar:na]
at
com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer.readClassAndObject(KryoSerializer.java:77)
~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.readClassAndObject(StandardSerializer.java:85)
~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(EdgeSerializer.java:206)
~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(EdgeSerializer.java:201)
~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.database.EdgeSerializer.parseRelation(EdgeSerializer.java:134)
~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readRelation(EdgeSerializer.java:70)
~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.transaction.RelationConstructor.readRelation(RelationConstructor.java:61)
~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(RelationConstructor.java:46)
~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(RelationConstructor.java:34)
~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.types.vertices.TitanSchemaVertex.getDefinition(TitanSchemaVertex.java:73)
~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.types.vertices.PropertyKeyVertex.getDataType(PropertyKeyVertex.java:17)
~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.query.QueryUtil.constraints2QNF(QueryUtil.java:148)
~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.query.graph.GraphCentricQueryBuilder.constructQuery(GraphCentricQueryBuilder.java:208)
~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.query.graph.GraphCentricQueryBuilder.vertices(GraphCentricQueryBuilder.java:155)
~[stormjar.jar:na]
at
com.tinkerpop.pipes.transform.GraphQueryPipe.processNextStart(GraphQueryPipe.java:49)
~[stormjar.jar:na]
at
com.tinkerpop.pipes.transform.GraphQueryPipe.processNextStart(GraphQueryPipe.java:17)
~[stormjar.jar:na]
at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
~[stormjar.jar:na]
at com.tinkerpop.pipes.IdentityPipe.processNextStart(IdentityPipe.java:19)
~[stormjar.jar:na]
at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
~[stormjar.jar:na]
at com.tinkerpop.pipes.util.Pipeline.next(Pipeline.java:115)
~[stormjar.jar:na]
at com.tinkerpop.pipes.util.PipeHelper.fillCollection(PipeHelper.java:52)
~[stormjar.jar:na]
at
com.tinkerpop.gremlin.java.GremlinPipeline.toList(GremlinPipeline.java:1564)
~[stormjar.jar:na]
at
com.comcast.graph.event.bolt.GraphSearchMetricsBolt.execute(GraphSearchMetricsBolt.java:83)
~[stormjar.jar:na]
at
backtype.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 6 common frames omitted
Looking at my dependency tree, titan has a dependency on kryo 2.22, how
does this conflict with Storm and 2.21?
titan:
*[INFO] | \- com.esotericsoftware.kryo:kryo:jar:2.22:compile*
Regards,
Nick
Re: What appears to be a Kryo Conflict
Posted by Curtis Allen <cu...@gmail.com>.
Nope, everything is working fine for us.
On Thu, Oct 9, 2014 at 2:58 PM, Nick Beenham <ni...@gmail.com> wrote:
> I saw that on SO and wondered how folks had solved the issue, did you run
> into any issues with 2.22 on the classpath?
>
> Nick
>
> On Thu, Oct 9, 2014 at 4:10 PM, Curtis Allen <cu...@gmail.com>
> wrote:
>
>> Your problem was answered in Stack Overflow
>> http://stackoverflow.com/questions/24244689/kryo-compatibility-in-storm-0-9-1and-titan-graph-database-0-4-2-with-apache-cass/24246845#24246845
>>
>> Titan 0.4.2 + Storm 0.9.2 are compatible.
>> Storm 0.9.2 is *not* compatible with Titan 0.5.0
>>
>> To fix this problem I had to make my own storm distribution with
>> kryo-2.22.jar on the classpath.
>>
>> Hope this helps.
>>
>> On Thu, Oct 9, 2014 at 1:54 PM, Brian O'Neill <bo...@alumni.brown.edu>
>> wrote:
>>
>>>
>>> We just started using 0.5, so…
>>> It was a non-prod environment, and we reloaded the data.
>>> (I’m sure that is no help at all)
>>>
>>> Definitely check your classpath for the different kryo versions though.
>>> Titan has a kryo as a dependency and it conflicts with Storm’s.
>>> The classpath may be different when running in the cluster.
>>>
>>> (I believe it is possible to capture the line from the log file that is
>>> spawning the worker to look at the classpath)
>>>
>>> -brian
>>>
>>> ---
>>>
>>> Brian O'Neill
>>>
>>> Chief Technology Officer
>>>
>>>
>>> *Health Market Science*
>>>
>>> *The Science of Better Results*
>>>
>>> 2700 Horizon Drive • King of Prussia, PA • 19406
>>>
>>> M: 215.588.6024 • @boneill42 <http://www.twitter.com/boneill42> •
>>>
>>> healthmarketscience.com
>>>
>>>
>>> This information transmitted in this email message is for the intended
>>> recipient only and may contain confidential and/or privileged material. If
>>> you received this email in error and are not the intended recipient, or the
>>> person responsible to deliver it to the intended recipient, please contact
>>> the sender at the email above and delete this email and any attachments and
>>> destroy any copies thereof. Any review, retransmission, dissemination,
>>> copying or other use of, or taking any action in reliance upon, this
>>> information by persons or entities other than the intended recipient is
>>> strictly prohibited.
>>>
>>>
>>>
>>>
>>> From: Nick Beenham <ni...@gmail.com>
>>> Reply-To: <us...@storm.apache.org>
>>> Date: Thursday, October 9, 2014 at 3:45 PM
>>> To: <us...@storm.apache.org>
>>> Subject: Re: What appears to be a Kryo Conflict
>>>
>>> It was stored with 0.5 and is being read with the same version. what was
>>> your solution?
>>>
>>> thanks,
>>>
>>> Nick
>>>
>>> On Thu, Oct 9, 2014 at 3:36 PM, Brian O'Neill <bo...@alumni.brown.edu>
>>> wrote:
>>>
>>>>
>>>> We ran into the same problem.
>>>>
>>>> What version of Titan was used to load the graph database?
>>>>
>>>> If you’ve stored data with 0.4, and you are trying to read it with 0.5
>>>> (from Storm), you may run into Kryo version issues.
>>>>
>>>> -brian
>>>>
>>>> ---
>>>>
>>>> Brian O'Neill
>>>>
>>>> Chief Technology Officer
>>>>
>>>>
>>>> *Health Market Science*
>>>>
>>>> *The Science of Better Results*
>>>>
>>>> 2700 Horizon Drive • King of Prussia, PA • 19406
>>>>
>>>> M: 215.588.6024 • @boneill42 <http://www.twitter.com/boneill42> •
>>>>
>>>> healthmarketscience.com
>>>>
>>>>
>>>> This information transmitted in this email message is for the intended
>>>> recipient only and may contain confidential and/or privileged material. If
>>>> you received this email in error and are not the intended recipient, or the
>>>> person responsible to deliver it to the intended recipient, please contact
>>>> the sender at the email above and delete this email and any attachments and
>>>> destroy any copies thereof. Any review, retransmission, dissemination,
>>>> copying or other use of, or taking any action in reliance upon, this
>>>> information by persons or entities other than the intended recipient is
>>>> strictly prohibited.
>>>>
>>>>
>>>>
>>>>
>>>> From: Nick Beenham <ni...@gmail.com>
>>>> Reply-To: <us...@storm.apache.org>
>>>> Date: Thursday, October 9, 2014 at 2:58 PM
>>>> To: <us...@storm.incubator.apache.org>
>>>> Subject: What appears to be a Kryo Conflict
>>>>
>>>> Hi all,
>>>>
>>>> I have a peculiar problem...
>>>>
>>>> I'm using a kafka storm combo to feed updates to my Titan Graph db. The
>>>> topology runs quite happily when local but as soon as I try to run it on a
>>>> cluster I run into Kryo errors.
>>>>
>>>> 2014-10-09 18:41:47 b.s.util [ERROR] Async loop died!
>>>> java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException:
>>>> Encountered unregistered class ID: 10
>>>> at
>>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
>>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at
>>>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
>>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at
>>>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
>>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at
>>>> backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
>>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
>>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
>>>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>>>> Caused by: com.esotericsoftware.kryo.KryoException: Encountered
>>>> unregistered class ID: 10
>>>> at
>>>> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)
>>>> ~[kryo-2.21.jar:na]
>>>> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610)
>>>> ~[kryo-2.21.jar:na]
>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721)
>>>> ~[kryo-2.21.jar:na]
>>>> at
>>>> com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer.readClassAndObject(KryoSerializer.java:77)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.readClassAndObject(StandardSerializer.java:85)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(EdgeSerializer.java:206)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(EdgeSerializer.java:201)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.parseRelation(EdgeSerializer.java:134)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readRelation(EdgeSerializer.java:70)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor.readRelation(RelationConstructor.java:61)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(RelationConstructor.java:46)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(RelationConstructor.java:34)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.thinkaurelius.titan.graphdb.types.vertices.TitanSchemaVertex.getDefinition(TitanSchemaVertex.java:73)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.thinkaurelius.titan.graphdb.types.vertices.PropertyKeyVertex.getDataType(PropertyKeyVertex.java:17)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.thinkaurelius.titan.graphdb.query.QueryUtil.constraints2QNF(QueryUtil.java:148)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.thinkaurelius.titan.graphdb.query.graph.GraphCentricQueryBuilder.constructQuery(GraphCentricQueryBuilder.java:208)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.thinkaurelius.titan.graphdb.query.graph.GraphCentricQueryBuilder.vertices(GraphCentricQueryBuilder.java:155)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.tinkerpop.pipes.transform.GraphQueryPipe.processNextStart(GraphQueryPipe.java:49)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.tinkerpop.pipes.transform.GraphQueryPipe.processNextStart(GraphQueryPipe.java:17)
>>>> ~[stormjar.jar:na]
>>>> at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.tinkerpop.pipes.IdentityPipe.processNextStart(IdentityPipe.java:19)
>>>> ~[stormjar.jar:na]
>>>> at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
>>>> ~[stormjar.jar:na]
>>>> at com.tinkerpop.pipes.util.Pipeline.next(Pipeline.java:115)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.tinkerpop.pipes.util.PipeHelper.fillCollection(PipeHelper.java:52)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.tinkerpop.gremlin.java.GremlinPipeline.toList(GremlinPipeline.java:1564)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> com.comcast.graph.event.bolt.GraphSearchMetricsBolt.execute(GraphSearchMetricsBolt.java:83)
>>>> ~[stormjar.jar:na]
>>>> at
>>>> backtype.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50)
>>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at
>>>> backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
>>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at
>>>> backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
>>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at
>>>> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
>>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at
>>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120)
>>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> ... 6 common frames omitted
>>>>
>>>> Looking at my dependency tree, titan has a dependency on kryo 2.22, how
>>>> does this conflict with Storm and 2.21?
>>>>
>>>> titan:
>>>> *[INFO] | \- com.esotericsoftware.kryo:kryo:jar:2.22:compile*
>>>>
>>>> Regards,
>>>>
>>>> Nick
>>>>
>>>>
>>>
>>
>
Re: What appears to be a Kryo Conflict
Posted by Nick Beenham <ni...@gmail.com>.
I saw that on SO and wondered how folks had solved the issue, did you run
into any issues with 2.22 on the classpath?
Nick
On Thu, Oct 9, 2014 at 4:10 PM, Curtis Allen <cu...@gmail.com>
wrote:
> Your problem was answered in Stack Overflow
> http://stackoverflow.com/questions/24244689/kryo-compatibility-in-storm-0-9-1and-titan-graph-database-0-4-2-with-apache-cass/24246845#24246845
>
> Titan 0.4.2 + Storm 0.9.2 are compatible.
> Storm 0.9.2 is *not* compatible with Titan 0.5.0
>
> To fix this problem I had to make my own storm distribution with
> kryo-2.22.jar on the classpath.
>
> Hope this helps.
>
> On Thu, Oct 9, 2014 at 1:54 PM, Brian O'Neill <bo...@alumni.brown.edu>
> wrote:
>
>>
>> We just started using 0.5, so…
>> It was a non-prod environment, and we reloaded the data.
>> (I’m sure that is no help at all)
>>
>> Definitely check your classpath for the different kryo versions though.
>> Titan has a kryo as a dependency and it conflicts with Storm’s.
>> The classpath may be different when running in the cluster.
>>
>> (I believe it is possible to capture the line from the log file that is
>> spawning the worker to look at the classpath)
>>
>> -brian
>>
>> ---
>>
>> Brian O'Neill
>>
>> Chief Technology Officer
>>
>>
>> *Health Market Science*
>>
>> *The Science of Better Results*
>>
>> 2700 Horizon Drive • King of Prussia, PA • 19406
>>
>> M: 215.588.6024 • @boneill42 <http://www.twitter.com/boneill42> •
>>
>> healthmarketscience.com
>>
>>
>> This information transmitted in this email message is for the intended
>> recipient only and may contain confidential and/or privileged material. If
>> you received this email in error and are not the intended recipient, or the
>> person responsible to deliver it to the intended recipient, please contact
>> the sender at the email above and delete this email and any attachments and
>> destroy any copies thereof. Any review, retransmission, dissemination,
>> copying or other use of, or taking any action in reliance upon, this
>> information by persons or entities other than the intended recipient is
>> strictly prohibited.
>>
>>
>>
>>
>> From: Nick Beenham <ni...@gmail.com>
>> Reply-To: <us...@storm.apache.org>
>> Date: Thursday, October 9, 2014 at 3:45 PM
>> To: <us...@storm.apache.org>
>> Subject: Re: What appears to be a Kryo Conflict
>>
>> It was stored with 0.5 and is being read with the same version. what was
>> your solution?
>>
>> thanks,
>>
>> Nick
>>
>> On Thu, Oct 9, 2014 at 3:36 PM, Brian O'Neill <bo...@alumni.brown.edu>
>> wrote:
>>
>>>
>>> We ran into the same problem.
>>>
>>> What version of Titan was used to load the graph database?
>>>
>>> If you’ve stored data with 0.4, and you are trying to read it with 0.5
>>> (from Storm), you may run into Kryo version issues.
>>>
>>> -brian
>>>
>>> ---
>>>
>>> Brian O'Neill
>>>
>>> Chief Technology Officer
>>>
>>>
>>> *Health Market Science*
>>>
>>> *The Science of Better Results*
>>>
>>> 2700 Horizon Drive • King of Prussia, PA • 19406
>>>
>>> M: 215.588.6024 • @boneill42 <http://www.twitter.com/boneill42> •
>>>
>>> healthmarketscience.com
>>>
>>>
>>> This information transmitted in this email message is for the intended
>>> recipient only and may contain confidential and/or privileged material. If
>>> you received this email in error and are not the intended recipient, or the
>>> person responsible to deliver it to the intended recipient, please contact
>>> the sender at the email above and delete this email and any attachments and
>>> destroy any copies thereof. Any review, retransmission, dissemination,
>>> copying or other use of, or taking any action in reliance upon, this
>>> information by persons or entities other than the intended recipient is
>>> strictly prohibited.
>>>
>>>
>>>
>>>
>>> From: Nick Beenham <ni...@gmail.com>
>>> Reply-To: <us...@storm.apache.org>
>>> Date: Thursday, October 9, 2014 at 2:58 PM
>>> To: <us...@storm.incubator.apache.org>
>>> Subject: What appears to be a Kryo Conflict
>>>
>>> Hi all,
>>>
>>> I have a peculiar problem...
>>>
>>> I'm using a kafka storm combo to feed updates to my Titan Graph db. The
>>> topology runs quite happily when local but as soon as I try to run it on a
>>> cluster I run into Kryo errors.
>>>
>>> 2014-10-09 18:41:47 b.s.util [ERROR] Async loop died!
>>> java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException:
>>> Encountered unregistered class ID: 10
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at
>>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at
>>> backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
>>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>>> Caused by: com.esotericsoftware.kryo.KryoException: Encountered
>>> unregistered class ID: 10
>>> at
>>> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)
>>> ~[kryo-2.21.jar:na]
>>> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610)
>>> ~[kryo-2.21.jar:na]
>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721)
>>> ~[kryo-2.21.jar:na]
>>> at
>>> com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer.readClassAndObject(KryoSerializer.java:77)
>>> ~[stormjar.jar:na]
>>> at
>>> com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.readClassAndObject(StandardSerializer.java:85)
>>> ~[stormjar.jar:na]
>>> at
>>> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(EdgeSerializer.java:206)
>>> ~[stormjar.jar:na]
>>> at
>>> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(EdgeSerializer.java:201)
>>> ~[stormjar.jar:na]
>>> at
>>> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.parseRelation(EdgeSerializer.java:134)
>>> ~[stormjar.jar:na]
>>> at
>>> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readRelation(EdgeSerializer.java:70)
>>> ~[stormjar.jar:na]
>>> at
>>> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor.readRelation(RelationConstructor.java:61)
>>> ~[stormjar.jar:na]
>>> at
>>> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(RelationConstructor.java:46)
>>> ~[stormjar.jar:na]
>>> at
>>> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(RelationConstructor.java:34)
>>> ~[stormjar.jar:na]
>>> at
>>> com.thinkaurelius.titan.graphdb.types.vertices.TitanSchemaVertex.getDefinition(TitanSchemaVertex.java:73)
>>> ~[stormjar.jar:na]
>>> at
>>> com.thinkaurelius.titan.graphdb.types.vertices.PropertyKeyVertex.getDataType(PropertyKeyVertex.java:17)
>>> ~[stormjar.jar:na]
>>> at
>>> com.thinkaurelius.titan.graphdb.query.QueryUtil.constraints2QNF(QueryUtil.java:148)
>>> ~[stormjar.jar:na]
>>> at
>>> com.thinkaurelius.titan.graphdb.query.graph.GraphCentricQueryBuilder.constructQuery(GraphCentricQueryBuilder.java:208)
>>> ~[stormjar.jar:na]
>>> at
>>> com.thinkaurelius.titan.graphdb.query.graph.GraphCentricQueryBuilder.vertices(GraphCentricQueryBuilder.java:155)
>>> ~[stormjar.jar:na]
>>> at
>>> com.tinkerpop.pipes.transform.GraphQueryPipe.processNextStart(GraphQueryPipe.java:49)
>>> ~[stormjar.jar:na]
>>> at
>>> com.tinkerpop.pipes.transform.GraphQueryPipe.processNextStart(GraphQueryPipe.java:17)
>>> ~[stormjar.jar:na]
>>> at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
>>> ~[stormjar.jar:na]
>>> at
>>> com.tinkerpop.pipes.IdentityPipe.processNextStart(IdentityPipe.java:19)
>>> ~[stormjar.jar:na]
>>> at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
>>> ~[stormjar.jar:na]
>>> at com.tinkerpop.pipes.util.Pipeline.next(Pipeline.java:115)
>>> ~[stormjar.jar:na]
>>> at
>>> com.tinkerpop.pipes.util.PipeHelper.fillCollection(PipeHelper.java:52)
>>> ~[stormjar.jar:na]
>>> at
>>> com.tinkerpop.gremlin.java.GremlinPipeline.toList(GremlinPipeline.java:1564)
>>> ~[stormjar.jar:na]
>>> at
>>> com.comcast.graph.event.bolt.GraphSearchMetricsBolt.execute(GraphSearchMetricsBolt.java:83)
>>> ~[stormjar.jar:na]
>>> at
>>> backtype.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50)
>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at
>>> backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at
>>> backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at
>>> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120)
>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> ... 6 common frames omitted
>>>
>>> Looking at my dependency tree, titan has a dependency on kryo 2.22, how
>>> does this conflict with Storm and 2.21?
>>>
>>> titan:
>>> *[INFO] | \- com.esotericsoftware.kryo:kryo:jar:2.22:compile*
>>>
>>> Regards,
>>>
>>> Nick
>>>
>>>
>>
>
Re: What appears to be a Kryo Conflict
Posted by Curtis Allen <cu...@gmail.com>.
Your problem was answered in Stack Overflow
http://stackoverflow.com/questions/24244689/kryo-compatibility-in-storm-0-9-1and-titan-graph-database-0-4-2-with-apache-cass/24246845#24246845
Titan 0.4.2 + Storm 0.9.2 are compatible.
Storm 0.9.2 is *not* compatible with Titan 0.5.0
To fix this problem I had to make my own storm distribution with
kryo-2.22.jar on the classpath.
Hope this helps.
On Thu, Oct 9, 2014 at 1:54 PM, Brian O'Neill <bo...@alumni.brown.edu> wrote:
>
> We just started using 0.5, so…
> It was a non-prod environment, and we reloaded the data.
> (I’m sure that is no help at all)
>
> Definitely check your classpath for the different kryo versions though.
> Titan has a kryo as a dependency and it conflicts with Storm’s.
> The classpath may be different when running in the cluster.
>
> (I believe it is possible to capture the line from the log file that is
> spawning the worker to look at the classpath)
>
> -brian
>
> ---
>
> Brian O'Neill
>
> Chief Technology Officer
>
>
> *Health Market Science*
>
> *The Science of Better Results*
>
> 2700 Horizon Drive • King of Prussia, PA • 19406
>
> M: 215.588.6024 • @boneill42 <http://www.twitter.com/boneill42> •
>
> healthmarketscience.com
>
>
> This information transmitted in this email message is for the intended
> recipient only and may contain confidential and/or privileged material. If
> you received this email in error and are not the intended recipient, or the
> person responsible to deliver it to the intended recipient, please contact
> the sender at the email above and delete this email and any attachments and
> destroy any copies thereof. Any review, retransmission, dissemination,
> copying or other use of, or taking any action in reliance upon, this
> information by persons or entities other than the intended recipient is
> strictly prohibited.
>
>
>
>
> From: Nick Beenham <ni...@gmail.com>
> Reply-To: <us...@storm.apache.org>
> Date: Thursday, October 9, 2014 at 3:45 PM
> To: <us...@storm.apache.org>
> Subject: Re: What appears to be a Kryo Conflict
>
> It was stored with 0.5 and is being read with the same version. what was
> your solution?
>
> thanks,
>
> Nick
>
> On Thu, Oct 9, 2014 at 3:36 PM, Brian O'Neill <bo...@alumni.brown.edu>
> wrote:
>
>>
>> We ran into the same problem.
>>
>> What version of Titan was used to load the graph database?
>>
>> If you’ve stored data with 0.4, and you are trying to read it with 0.5
>> (from Storm), you may run into Kryo version issues.
>>
>> -brian
>>
>> ---
>>
>> Brian O'Neill
>>
>> Chief Technology Officer
>>
>>
>> *Health Market Science*
>>
>> *The Science of Better Results*
>>
>> 2700 Horizon Drive • King of Prussia, PA • 19406
>>
>> M: 215.588.6024 • @boneill42 <http://www.twitter.com/boneill42> •
>>
>> healthmarketscience.com
>>
>>
>> This information transmitted in this email message is for the intended
>> recipient only and may contain confidential and/or privileged material. If
>> you received this email in error and are not the intended recipient, or the
>> person responsible to deliver it to the intended recipient, please contact
>> the sender at the email above and delete this email and any attachments and
>> destroy any copies thereof. Any review, retransmission, dissemination,
>> copying or other use of, or taking any action in reliance upon, this
>> information by persons or entities other than the intended recipient is
>> strictly prohibited.
>>
>>
>>
>>
>> From: Nick Beenham <ni...@gmail.com>
>> Reply-To: <us...@storm.apache.org>
>> Date: Thursday, October 9, 2014 at 2:58 PM
>> To: <us...@storm.incubator.apache.org>
>> Subject: What appears to be a Kryo Conflict
>>
>> Hi all,
>>
>> I have a peculiar problem...
>>
>> I'm using a kafka storm combo to feed updates to my Titan Graph db. The
>> topology runs quite happily when local but as soon as I try to run it on a
>> cluster I run into Kryo errors.
>>
>> 2014-10-09 18:41:47 b.s.util [ERROR] Async loop died!
>> java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException:
>> Encountered unregistered class ID: 10
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>> Caused by: com.esotericsoftware.kryo.KryoException: Encountered
>> unregistered class ID: 10
>> at
>> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)
>> ~[kryo-2.21.jar:na]
>> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610)
>> ~[kryo-2.21.jar:na]
>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721)
>> ~[kryo-2.21.jar:na]
>> at
>> com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer.readClassAndObject(KryoSerializer.java:77)
>> ~[stormjar.jar:na]
>> at
>> com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.readClassAndObject(StandardSerializer.java:85)
>> ~[stormjar.jar:na]
>> at
>> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(EdgeSerializer.java:206)
>> ~[stormjar.jar:na]
>> at
>> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(EdgeSerializer.java:201)
>> ~[stormjar.jar:na]
>> at
>> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.parseRelation(EdgeSerializer.java:134)
>> ~[stormjar.jar:na]
>> at
>> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readRelation(EdgeSerializer.java:70)
>> ~[stormjar.jar:na]
>> at
>> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor.readRelation(RelationConstructor.java:61)
>> ~[stormjar.jar:na]
>> at
>> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(RelationConstructor.java:46)
>> ~[stormjar.jar:na]
>> at
>> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(RelationConstructor.java:34)
>> ~[stormjar.jar:na]
>> at
>> com.thinkaurelius.titan.graphdb.types.vertices.TitanSchemaVertex.getDefinition(TitanSchemaVertex.java:73)
>> ~[stormjar.jar:na]
>> at
>> com.thinkaurelius.titan.graphdb.types.vertices.PropertyKeyVertex.getDataType(PropertyKeyVertex.java:17)
>> ~[stormjar.jar:na]
>> at
>> com.thinkaurelius.titan.graphdb.query.QueryUtil.constraints2QNF(QueryUtil.java:148)
>> ~[stormjar.jar:na]
>> at
>> com.thinkaurelius.titan.graphdb.query.graph.GraphCentricQueryBuilder.constructQuery(GraphCentricQueryBuilder.java:208)
>> ~[stormjar.jar:na]
>> at
>> com.thinkaurelius.titan.graphdb.query.graph.GraphCentricQueryBuilder.vertices(GraphCentricQueryBuilder.java:155)
>> ~[stormjar.jar:na]
>> at
>> com.tinkerpop.pipes.transform.GraphQueryPipe.processNextStart(GraphQueryPipe.java:49)
>> ~[stormjar.jar:na]
>> at
>> com.tinkerpop.pipes.transform.GraphQueryPipe.processNextStart(GraphQueryPipe.java:17)
>> ~[stormjar.jar:na]
>> at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
>> ~[stormjar.jar:na]
>> at
>> com.tinkerpop.pipes.IdentityPipe.processNextStart(IdentityPipe.java:19)
>> ~[stormjar.jar:na]
>> at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
>> ~[stormjar.jar:na]
>> at com.tinkerpop.pipes.util.Pipeline.next(Pipeline.java:115)
>> ~[stormjar.jar:na]
>> at com.tinkerpop.pipes.util.PipeHelper.fillCollection(PipeHelper.java:52)
>> ~[stormjar.jar:na]
>> at
>> com.tinkerpop.gremlin.java.GremlinPipeline.toList(GremlinPipeline.java:1564)
>> ~[stormjar.jar:na]
>> at
>> com.comcast.graph.event.bolt.GraphSearchMetricsBolt.execute(GraphSearchMetricsBolt.java:83)
>> ~[stormjar.jar:na]
>> at
>> backtype.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> ... 6 common frames omitted
>>
>> Looking at my dependency tree, titan has a dependency on kryo 2.22, how
>> does this conflict with Storm and 2.21?
>>
>> titan:
>> *[INFO] | \- com.esotericsoftware.kryo:kryo:jar:2.22:compile*
>>
>> Regards,
>>
>> Nick
>>
>>
>
Re: What appears to be a Kryo Conflict
Posted by Brian O'Neill <bo...@alumni.brown.edu>.
We just started using 0.5, so
It was a non-prod environment, and we reloaded the data.
(I¹m sure that is no help at all)
Definitely check your classpath for the different kryo versions though.
Titan has a kryo as a dependency and it conflicts with Storm¹s.
The classpath may be different when running in the cluster.
(I believe it is possible to capture the line from the log file that is
spawning the worker to look at the classpath)
-brian
---
Brian O'Neill
Chief Technology Officer
Health Market Science
The Science of Better Results
2700 Horizon Drive King of Prussia, PA 19406
M: 215.588.6024 @boneill42 <http://www.twitter.com/boneill42>
healthmarketscience.com
This information transmitted in this email message is for the intended
recipient only and may contain confidential and/or privileged material. If
you received this email in error and are not the intended recipient, or the
person responsible to deliver it to the intended recipient, please contact
the sender at the email above and delete this email and any attachments and
destroy any copies thereof. Any review, retransmission, dissemination,
copying or other use of, or taking any action in reliance upon, this
information by persons or entities other than the intended recipient is
strictly prohibited.
From: Nick Beenham <ni...@gmail.com>
Reply-To: <us...@storm.apache.org>
Date: Thursday, October 9, 2014 at 3:45 PM
To: <us...@storm.apache.org>
Subject: Re: What appears to be a Kryo Conflict
It was stored with 0.5 and is being read with the same version. what was
your solution?
thanks,
Nick
On Thu, Oct 9, 2014 at 3:36 PM, Brian O'Neill <bo...@alumni.brown.edu> wrote:
>
> We ran into the same problem.
>
> What version of Titan was used to load the graph database?
>
> If you¹ve stored data with 0.4, and you are trying to read it with 0.5 (from
> Storm), you may run into Kryo version issues.
>
> -brian
>
> ---
> Brian O'Neill
> Chief Technology Officer
>
>
> Health Market Science
> The Science of Better Results
> 2700 Horizon Drive King of Prussia, PA 19406
> M: 215.588.6024 <tel:215.588.6024> @boneill42
> <http://www.twitter.com/boneill42>
> healthmarketscience.com
>
>
> This information transmitted in this email message is for the intended
> recipient only and may contain confidential and/or privileged material. If you
> received this email in error and are not the intended recipient, or the person
> responsible to deliver it to the intended recipient, please contact the sender
> at the email above and delete this email and any attachments and destroy any
> copies thereof. Any review, retransmission, dissemination, copying or other
> use of, or taking any action in reliance upon, this information by persons or
> entities other than the intended recipient is strictly prohibited.
>
>
>
> From: Nick Beenham <ni...@gmail.com>
> Reply-To: <us...@storm.apache.org>
> Date: Thursday, October 9, 2014 at 2:58 PM
> To: <us...@storm.incubator.apache.org>
> Subject: What appears to be a Kryo Conflict
>
> Hi all,
>
> I have a peculiar problem...
>
> I'm using a kafka storm combo to feed updates to my Titan Graph db. The
> topology runs quite happily when local but as soon as I try to run it on a
> cluster I run into Kryo errors.
>
> 2014-10-09 18:41:47 b.s.util [ERROR] Async loop died!
> java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException:
> Encountered unregistered class ID: 10
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:1
> 28) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.j
> ava:99) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:
> 746) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> Caused by: com.esotericsoftware.kryo.KryoException: Encountered unregistered
> class ID: 10
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassReso
> lver.java:119) ~[kryo-2.21.jar:na]
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610) ~[kryo-2.21.jar:na]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721)
> ~[kryo-2.21.jar:na]
> at
> com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer.readCla
> ssAndObject(KryoSerializer.java:77) ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.readClas
> sAndObject(StandardSerializer.java:85) ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(Edge
> Serializer.java:206) ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(Edge
> Serializer.java:201) ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.parseRelation(EdgeSeri
> alizer.java:134) ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readRelation(EdgeSeria
> lizer.java:70) ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor.readRelation(R
> elationConstructor.java:61) ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(Relat
> ionConstructor.java:46) ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(Relat
> ionConstructor.java:34) ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.types.vertices.TitanSchemaVertex.getDefinition
> (TitanSchemaVertex.java:73) ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.types.vertices.PropertyKeyVertex.getDataType(P
> ropertyKeyVertex.java:17) ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.query.QueryUtil.constraints2QNF(QueryUtil.java
> :148) ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.query.graph.GraphCentricQueryBuilder.construct
> Query(GraphCentricQueryBuilder.java:208) ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.query.graph.GraphCentricQueryBuilder.vertices(
> GraphCentricQueryBuilder.java:155) ~[stormjar.jar:na]
> at
> com.tinkerpop.pipes.transform.GraphQueryPipe.processNextStart(GraphQueryPipe.j
> ava:49) ~[stormjar.jar:na]
> at
> com.tinkerpop.pipes.transform.GraphQueryPipe.processNextStart(GraphQueryPipe.j
> ava:17) ~[stormjar.jar:na]
> at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
> ~[stormjar.jar:na]
> at com.tinkerpop.pipes.IdentityPipe.processNextStart(IdentityPipe.java:19)
> ~[stormjar.jar:na]
> at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
> ~[stormjar.jar:na]
> at com.tinkerpop.pipes.util.Pipeline.next(Pipeline.java:115)
> ~[stormjar.jar:na]
> at com.tinkerpop.pipes.util.PipeHelper.fillCollection(PipeHelper.java:52)
> ~[stormjar.jar:na]
> at
> com.tinkerpop.gremlin.java.GremlinPipeline.toList(GremlinPipeline.java:1564)
> ~[stormjar.jar:na]
> at
> com.comcast.graph.event.bolt.GraphSearchMetricsBolt.execute(GraphSearchMetrics
> Bolt.java:83) ~[stormjar.jar:na]
> at
> backtype.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.
> clj:631) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:3
> 99) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:1
> 20) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 6 common frames omitted
>
> Looking at my dependency tree, titan has a dependency on kryo 2.22, how does
> this conflict with Storm and 2.21?
>
> titan:
> [INFO] | \- com.esotericsoftware.kryo:kryo:jar:2.22:compile
>
> Regards,
>
> Nick
>
Re: What appears to be a Kryo Conflict
Posted by Nick Beenham <ni...@gmail.com>.
It was stored with 0.5 and is being read with the same version. what was
your solution?
thanks,
Nick
On Thu, Oct 9, 2014 at 3:36 PM, Brian O'Neill <bo...@alumni.brown.edu> wrote:
>
> We ran into the same problem.
>
> What version of Titan was used to load the graph database?
>
> If you’ve stored data with 0.4, and you are trying to read it with 0.5
> (from Storm), you may run into Kryo version issues.
>
> -brian
>
> ---
>
> Brian O'Neill
>
> Chief Technology Officer
>
>
> *Health Market Science*
>
> *The Science of Better Results*
>
> 2700 Horizon Drive • King of Prussia, PA • 19406
>
> M: 215.588.6024 • @boneill42 <http://www.twitter.com/boneill42> •
>
> healthmarketscience.com
>
>
> This information transmitted in this email message is for the intended
> recipient only and may contain confidential and/or privileged material. If
> you received this email in error and are not the intended recipient, or the
> person responsible to deliver it to the intended recipient, please contact
> the sender at the email above and delete this email and any attachments and
> destroy any copies thereof. Any review, retransmission, dissemination,
> copying or other use of, or taking any action in reliance upon, this
> information by persons or entities other than the intended recipient is
> strictly prohibited.
>
>
>
>
> From: Nick Beenham <ni...@gmail.com>
> Reply-To: <us...@storm.apache.org>
> Date: Thursday, October 9, 2014 at 2:58 PM
> To: <us...@storm.incubator.apache.org>
> Subject: What appears to be a Kryo Conflict
>
> Hi all,
>
> I have a peculiar problem...
>
> I'm using a kafka storm combo to feed updates to my Titan Graph db. The
> topology runs quite happily when local but as soon as I try to run it on a
> cluster I run into Kryo errors.
>
> 2014-10-09 18:41:47 b.s.util [ERROR] Async loop died!
> java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException:
> Encountered unregistered class ID: 10
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> Caused by: com.esotericsoftware.kryo.KryoException: Encountered
> unregistered class ID: 10
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)
> ~[kryo-2.21.jar:na]
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610)
> ~[kryo-2.21.jar:na]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721)
> ~[kryo-2.21.jar:na]
> at
> com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer.readClassAndObject(KryoSerializer.java:77)
> ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.readClassAndObject(StandardSerializer.java:85)
> ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(EdgeSerializer.java:206)
> ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(EdgeSerializer.java:201)
> ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.parseRelation(EdgeSerializer.java:134)
> ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readRelation(EdgeSerializer.java:70)
> ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor.readRelation(RelationConstructor.java:61)
> ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(RelationConstructor.java:46)
> ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(RelationConstructor.java:34)
> ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.types.vertices.TitanSchemaVertex.getDefinition(TitanSchemaVertex.java:73)
> ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.types.vertices.PropertyKeyVertex.getDataType(PropertyKeyVertex.java:17)
> ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.query.QueryUtil.constraints2QNF(QueryUtil.java:148)
> ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.query.graph.GraphCentricQueryBuilder.constructQuery(GraphCentricQueryBuilder.java:208)
> ~[stormjar.jar:na]
> at
> com.thinkaurelius.titan.graphdb.query.graph.GraphCentricQueryBuilder.vertices(GraphCentricQueryBuilder.java:155)
> ~[stormjar.jar:na]
> at
> com.tinkerpop.pipes.transform.GraphQueryPipe.processNextStart(GraphQueryPipe.java:49)
> ~[stormjar.jar:na]
> at
> com.tinkerpop.pipes.transform.GraphQueryPipe.processNextStart(GraphQueryPipe.java:17)
> ~[stormjar.jar:na]
> at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
> ~[stormjar.jar:na]
> at com.tinkerpop.pipes.IdentityPipe.processNextStart(IdentityPipe.java:19)
> ~[stormjar.jar:na]
> at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
> ~[stormjar.jar:na]
> at com.tinkerpop.pipes.util.Pipeline.next(Pipeline.java:115)
> ~[stormjar.jar:na]
> at com.tinkerpop.pipes.util.PipeHelper.fillCollection(PipeHelper.java:52)
> ~[stormjar.jar:na]
> at
> com.tinkerpop.gremlin.java.GremlinPipeline.toList(GremlinPipeline.java:1564)
> ~[stormjar.jar:na]
> at
> com.comcast.graph.event.bolt.GraphSearchMetricsBolt.execute(GraphSearchMetricsBolt.java:83)
> ~[stormjar.jar:na]
> at
> backtype.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 6 common frames omitted
>
> Looking at my dependency tree, titan has a dependency on kryo 2.22, how
> does this conflict with Storm and 2.21?
>
> titan:
> *[INFO] | \- com.esotericsoftware.kryo:kryo:jar:2.22:compile*
>
> Regards,
>
> Nick
>
>
Re: What appears to be a Kryo Conflict
Posted by Brian O'Neill <bo...@alumni.brown.edu>.
We ran into the same problem.
What version of Titan was used to load the graph database?
If you¹ve stored data with 0.4, and you are trying to read it with 0.5 (from
Storm), you may run into Kryo version issues.
-brian
---
Brian O'Neill
Chief Technology Officer
Health Market Science
The Science of Better Results
2700 Horizon Drive King of Prussia, PA 19406
M: 215.588.6024 @boneill42 <http://www.twitter.com/boneill42>
healthmarketscience.com
This information transmitted in this email message is for the intended
recipient only and may contain confidential and/or privileged material. If
you received this email in error and are not the intended recipient, or the
person responsible to deliver it to the intended recipient, please contact
the sender at the email above and delete this email and any attachments and
destroy any copies thereof. Any review, retransmission, dissemination,
copying or other use of, or taking any action in reliance upon, this
information by persons or entities other than the intended recipient is
strictly prohibited.
From: Nick Beenham <ni...@gmail.com>
Reply-To: <us...@storm.apache.org>
Date: Thursday, October 9, 2014 at 2:58 PM
To: <us...@storm.incubator.apache.org>
Subject: What appears to be a Kryo Conflict
Hi all,
I have a peculiar problem...
I'm using a kafka storm combo to feed updates to my Titan Graph db. The
topology runs quite happily when local but as soon as I try to run it on a
cluster I run into Kryo errors.
2014-10-09 18:41:47 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException:
Encountered unregistered class ID: 10
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java
:128) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue
.java:99) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:8
0) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.cl
j:746) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: com.esotericsoftware.kryo.KryoException: Encountered unregistered
class ID: 10
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassRe
solver.java:119) ~[kryo-2.21.jar:na]
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610)
~[kryo-2.21.jar:na]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721)
~[kryo-2.21.jar:na]
at
com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer.readC
lassAndObject(KryoSerializer.java:77) ~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.readCl
assAndObject(StandardSerializer.java:85) ~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(Ed
geSerializer.java:206) ~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(Ed
geSerializer.java:201) ~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.database.EdgeSerializer.parseRelation(EdgeSe
rializer.java:134) ~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readRelation(EdgeSer
ializer.java:70) ~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.transaction.RelationConstructor.readRelation
(RelationConstructor.java:61) ~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(Rel
ationConstructor.java:46) ~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(Rel
ationConstructor.java:34) ~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.types.vertices.TitanSchemaVertex.getDefiniti
on(TitanSchemaVertex.java:73) ~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.types.vertices.PropertyKeyVertex.getDataType
(PropertyKeyVertex.java:17) ~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.query.QueryUtil.constraints2QNF(QueryUtil.ja
va:148) ~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.query.graph.GraphCentricQueryBuilder.constru
ctQuery(GraphCentricQueryBuilder.java:208) ~[stormjar.jar:na]
at
com.thinkaurelius.titan.graphdb.query.graph.GraphCentricQueryBuilder.vertice
s(GraphCentricQueryBuilder.java:155) ~[stormjar.jar:na]
at
com.tinkerpop.pipes.transform.GraphQueryPipe.processNextStart(GraphQueryPipe
.java:49) ~[stormjar.jar:na]
at
com.tinkerpop.pipes.transform.GraphQueryPipe.processNextStart(GraphQueryPipe
.java:17) ~[stormjar.jar:na]
at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
~[stormjar.jar:na]
at com.tinkerpop.pipes.IdentityPipe.processNextStart(IdentityPipe.java:19)
~[stormjar.jar:na]
at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
~[stormjar.jar:na]
at com.tinkerpop.pipes.util.Pipeline.next(Pipeline.java:115)
~[stormjar.jar:na]
at com.tinkerpop.pipes.util.PipeHelper.fillCollection(PipeHelper.java:52)
~[stormjar.jar:na]
at
com.tinkerpop.gremlin.java.GremlinPipeline.toList(GremlinPipeline.java:1564)
~[stormjar.jar:na]
at
com.comcast.graph.event.bolt.GraphSearchMetricsBolt.execute(GraphSearchMetri
csBolt.java:83) ~[stormjar.jar:na]
at
backtype.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executo
r.clj:631) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj
:399) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58
) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java
:120) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 6 common frames omitted
Looking at my dependency tree, titan has a dependency on kryo 2.22, how does
this conflict with Storm and 2.21?
titan:
[INFO] | \- com.esotericsoftware.kryo:kryo:jar:2.22:compile
Regards,
Nick