You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by Sebastian Schelter <ss...@apache.org> on 2011/12/23 10:49:41 UTC
Unable to load vertices
Hi,
I'm currently implementing an algorithm for diameter and radius
estimation. It already works when I run it on toy data via
InternalVertexRunner in a unit test.
Unfortunately, in my tests with a single node hadoop instance and real
cluster, I always run into the attached exception during startup. Does
anybody have an idea what might cause this?
--sebastian
2011-12-23 10:43:09,769 INFO org.apache.hadoop.mapred.TaskInProgress:
Error from attempt_201112230924_0006_m_000001_0:
java.lang.IllegalStateException: run: Caught an unrecoverable exception
setup: Offlining servers due to exception...
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:641)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Caused by: java.lang.RuntimeException: setup: Offlining servers due to
exception...
at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:466)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:630)
... 7 more
Caused by: java.lang.IllegalStateException: setup: loadVertices failed
at
org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:582)
at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:458)
... 8 more
Caused by: java.lang.RuntimeException: java.io.IOException: Call to
poodle-6/127.0.1.1:30002 failed on local exception: java.io.EOFException
at
org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:768)
at
org.apache.giraph.graph.BspServiceWorker.loadVertices(BspServiceWorker.java:304)
at
org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:575)
... 9 more
Caused by: java.io.IOException: Call to poodle-6/127.0.1.1:30002 failed
on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065)
at org.apache.hadoop.ipc.Client.call(Client.java:1033)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
at $Proxy3.putVertexList(Unknown Source)
at
org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:765)
... 11 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712)
Re: Unable to load vertices
Posted by Avery Ching <ac...@apache.org>.
Glad you found the issue Sebastian! =)
Avery
On 12/27/11 12:47 PM, Sebastian Schelter wrote:
> You were write it was an issue with writing/reading the vertex value.
> Only took me three days of searching to find out that I simply forgot to
> call setVertexValue() ... :)
>
> --sebastian
>
>
>
> On 23.12.2011 18:28, Avery Ching wrote:
>> Without looking at your code, maybe your I, V, E, or M types might have
>> Writable issues? In the single worker case, does checkpointing work?
>> That would verify the writing part of Writable is okay, but not the
>> reading part...(well you can do a manual checkpoint restart I guess to
>> verify that).
>>
>> Avery
>>
>> On 12/23/11 9:23 AM, Sebastian Schelter wrote:
>>> I'm extending org.apache.giraph.graph.Vertex directly. I also created
>>> unit tests for the serialization of the Writables (writing them to a
>>> byte array and reading them back) without finding something. Thank you
>>> for the advice however, I'll continue searching :)
>>>
>>> --sebastian
>>>
>>>
>>> On 23.12.2011 18:14, Avery Ching wrote:
>>>> What MutableVertex implementation are you using? Sounds like the issue
>>>> only happens during the RPC to send the vertex to another worker. Maybe
>>>> a bug in the Writable implementation?
>>>>
>>>> Avery
>>>>
>>>> On 12/23/11 3:14 AM, Sebastian Schelter wrote:
>>>>> Hmm, the job works if I use a single worker only locally, strange...
>>>>>
>>>>> On 23.12.2011 11:07, Claudio Martella wrote:
>>>>>> With a super quick look, so i might be completely wrong, this looks
>>>>>> like you're running a different hadoop locally and on your test. Is
>>>>>> there any chance you're not using hadoop non_secure locally but you're
>>>>>> in your distributed mode?
>>>>>>
>>>>>> On Fri, Dec 23, 2011 at 10:49 AM, Sebastian Schelter<ss...@apache.org>
>>>>>> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm currently implementing an algorithm for diameter and radius
>>>>>>> estimation. It already works when I run it on toy data via
>>>>>>> InternalVertexRunner in a unit test.
>>>>>>>
>>>>>>> Unfortunately, in my tests with a single node hadoop instance and
>>>>>>> real
>>>>>>> cluster, I always run into the attached exception during startup.
>>>>>>> Does
>>>>>>> anybody have an idea what might cause this?
>>>>>>>
>>>>>>> --sebastian
>>>>>>>
>>>>>>>
>>>>>>> 2011-12-23 10:43:09,769 INFO org.apache.hadoop.mapred.TaskInProgress:
>>>>>>> Error from attempt_201112230924_0006_m_000001_0:
>>>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable
>>>>>>> exception
>>>>>>> setup: Offlining servers due to exception...
>>>>>>> at
>>>>>>> org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:641)
>>>>>>> at
>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
>>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>>>>>>> at java.security.AccessController.doPrivileged(Native
>>>>>>> Method)
>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>>> at
>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>>>>>>>
>>>>>>>
>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>>>>>>> Caused by: java.lang.RuntimeException: setup: Offlining servers
>>>>>>> due to
>>>>>>> exception...
>>>>>>> at
>>>>>>> org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:466)
>>>>>>> at
>>>>>>> org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:630)
>>>>>>> ... 7 more
>>>>>>> Caused by: java.lang.IllegalStateException: setup: loadVertices
>>>>>>> failed
>>>>>>> at
>>>>>>> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:582)
>>>>>>>
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:458)
>>>>>>> ... 8 more
>>>>>>> Caused by: java.lang.RuntimeException: java.io.IOException: Call to
>>>>>>> poodle-6/127.0.1.1:30002 failed on local exception:
>>>>>>> java.io.EOFException
>>>>>>> at
>>>>>>> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:768)
>>>>>>>
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.giraph.graph.BspServiceWorker.loadVertices(BspServiceWorker.java:304)
>>>>>>>
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:575)
>>>>>>>
>>>>>>>
>>>>>>> ... 9 more
>>>>>>> Caused by: java.io.IOException: Call to poodle-6/127.0.1.1:30002
>>>>>>> failed
>>>>>>> on local exception: java.io.EOFException
>>>>>>> at
>>>>>>> org.apache.hadoop.ipc.Client.wrapException(Client.java:1065)
>>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1033)
>>>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
>>>>>>> at $Proxy3.putVertexList(Unknown Source)
>>>>>>> at
>>>>>>> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:765)
>>>>>>>
>>>>>>>
>>>>>>> ... 11 more
>>>>>>> Caused by: java.io.EOFException
>>>>>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>>>>>> at
>>>>>>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767)
>>>>>>>
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.ipc.Client$Connection.run(Client.java:712)
Re: Unable to load vertices
Posted by Sebastian Schelter <ss...@apache.org>.
You were write it was an issue with writing/reading the vertex value.
Only took me three days of searching to find out that I simply forgot to
call setVertexValue() ... :)
--sebastian
On 23.12.2011 18:28, Avery Ching wrote:
> Without looking at your code, maybe your I, V, E, or M types might have
> Writable issues? In the single worker case, does checkpointing work?
> That would verify the writing part of Writable is okay, but not the
> reading part...(well you can do a manual checkpoint restart I guess to
> verify that).
>
> Avery
>
> On 12/23/11 9:23 AM, Sebastian Schelter wrote:
>> I'm extending org.apache.giraph.graph.Vertex directly. I also created
>> unit tests for the serialization of the Writables (writing them to a
>> byte array and reading them back) without finding something. Thank you
>> for the advice however, I'll continue searching :)
>>
>> --sebastian
>>
>>
>> On 23.12.2011 18:14, Avery Ching wrote:
>>> What MutableVertex implementation are you using? Sounds like the issue
>>> only happens during the RPC to send the vertex to another worker. Maybe
>>> a bug in the Writable implementation?
>>>
>>> Avery
>>>
>>> On 12/23/11 3:14 AM, Sebastian Schelter wrote:
>>>> Hmm, the job works if I use a single worker only locally, strange...
>>>>
>>>> On 23.12.2011 11:07, Claudio Martella wrote:
>>>>> With a super quick look, so i might be completely wrong, this looks
>>>>> like you're running a different hadoop locally and on your test. Is
>>>>> there any chance you're not using hadoop non_secure locally but you're
>>>>> in your distributed mode?
>>>>>
>>>>> On Fri, Dec 23, 2011 at 10:49 AM, Sebastian Schelter<ss...@apache.org>
>>>>> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'm currently implementing an algorithm for diameter and radius
>>>>>> estimation. It already works when I run it on toy data via
>>>>>> InternalVertexRunner in a unit test.
>>>>>>
>>>>>> Unfortunately, in my tests with a single node hadoop instance and
>>>>>> real
>>>>>> cluster, I always run into the attached exception during startup.
>>>>>> Does
>>>>>> anybody have an idea what might cause this?
>>>>>>
>>>>>> --sebastian
>>>>>>
>>>>>>
>>>>>> 2011-12-23 10:43:09,769 INFO org.apache.hadoop.mapred.TaskInProgress:
>>>>>> Error from attempt_201112230924_0006_m_000001_0:
>>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable
>>>>>> exception
>>>>>> setup: Offlining servers due to exception...
>>>>>> at
>>>>>> org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:641)
>>>>>> at
>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>>>>>> at java.security.AccessController.doPrivileged(Native
>>>>>> Method)
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>> at
>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>>>>>>
>>>>>>
>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>>>>>> Caused by: java.lang.RuntimeException: setup: Offlining servers
>>>>>> due to
>>>>>> exception...
>>>>>> at
>>>>>> org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:466)
>>>>>> at
>>>>>> org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:630)
>>>>>> ... 7 more
>>>>>> Caused by: java.lang.IllegalStateException: setup: loadVertices
>>>>>> failed
>>>>>> at
>>>>>> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:582)
>>>>>>
>>>>>>
>>>>>> at
>>>>>> org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:458)
>>>>>> ... 8 more
>>>>>> Caused by: java.lang.RuntimeException: java.io.IOException: Call to
>>>>>> poodle-6/127.0.1.1:30002 failed on local exception:
>>>>>> java.io.EOFException
>>>>>> at
>>>>>> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:768)
>>>>>>
>>>>>>
>>>>>> at
>>>>>> org.apache.giraph.graph.BspServiceWorker.loadVertices(BspServiceWorker.java:304)
>>>>>>
>>>>>>
>>>>>> at
>>>>>> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:575)
>>>>>>
>>>>>>
>>>>>> ... 9 more
>>>>>> Caused by: java.io.IOException: Call to poodle-6/127.0.1.1:30002
>>>>>> failed
>>>>>> on local exception: java.io.EOFException
>>>>>> at
>>>>>> org.apache.hadoop.ipc.Client.wrapException(Client.java:1065)
>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1033)
>>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
>>>>>> at $Proxy3.putVertexList(Unknown Source)
>>>>>> at
>>>>>> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:765)
>>>>>>
>>>>>>
>>>>>> ... 11 more
>>>>>> Caused by: java.io.EOFException
>>>>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>>>>> at
>>>>>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767)
>>>>>>
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.ipc.Client$Connection.run(Client.java:712)
>>>>>
>
Re: Unable to load vertices
Posted by Avery Ching <ac...@apache.org>.
Without looking at your code, maybe your I, V, E, or M types might have
Writable issues? In the single worker case, does checkpointing work?
That would verify the writing part of Writable is okay, but not the
reading part...(well you can do a manual checkpoint restart I guess to
verify that).
Avery
On 12/23/11 9:23 AM, Sebastian Schelter wrote:
> I'm extending org.apache.giraph.graph.Vertex directly. I also created
> unit tests for the serialization of the Writables (writing them to a
> byte array and reading them back) without finding something. Thank you
> for the advice however, I'll continue searching :)
>
> --sebastian
>
>
> On 23.12.2011 18:14, Avery Ching wrote:
>> What MutableVertex implementation are you using? Sounds like the issue
>> only happens during the RPC to send the vertex to another worker. Maybe
>> a bug in the Writable implementation?
>>
>> Avery
>>
>> On 12/23/11 3:14 AM, Sebastian Schelter wrote:
>>> Hmm, the job works if I use a single worker only locally, strange...
>>>
>>> On 23.12.2011 11:07, Claudio Martella wrote:
>>>> With a super quick look, so i might be completely wrong, this looks
>>>> like you're running a different hadoop locally and on your test. Is
>>>> there any chance you're not using hadoop non_secure locally but you're
>>>> in your distributed mode?
>>>>
>>>> On Fri, Dec 23, 2011 at 10:49 AM, Sebastian Schelter<ss...@apache.org>
>>>> wrote:
>>>>> Hi,
>>>>>
>>>>> I'm currently implementing an algorithm for diameter and radius
>>>>> estimation. It already works when I run it on toy data via
>>>>> InternalVertexRunner in a unit test.
>>>>>
>>>>> Unfortunately, in my tests with a single node hadoop instance and real
>>>>> cluster, I always run into the attached exception during startup. Does
>>>>> anybody have an idea what might cause this?
>>>>>
>>>>> --sebastian
>>>>>
>>>>>
>>>>> 2011-12-23 10:43:09,769 INFO org.apache.hadoop.mapred.TaskInProgress:
>>>>> Error from attempt_201112230924_0006_m_000001_0:
>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable exception
>>>>> setup: Offlining servers due to exception...
>>>>> at
>>>>> org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:641)
>>>>> at
>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>> at
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>>>>>
>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>>>>> Caused by: java.lang.RuntimeException: setup: Offlining servers due to
>>>>> exception...
>>>>> at
>>>>> org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:466)
>>>>> at
>>>>> org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:630)
>>>>> ... 7 more
>>>>> Caused by: java.lang.IllegalStateException: setup: loadVertices failed
>>>>> at
>>>>> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:582)
>>>>>
>>>>> at
>>>>> org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:458)
>>>>> ... 8 more
>>>>> Caused by: java.lang.RuntimeException: java.io.IOException: Call to
>>>>> poodle-6/127.0.1.1:30002 failed on local exception:
>>>>> java.io.EOFException
>>>>> at
>>>>> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:768)
>>>>>
>>>>> at
>>>>> org.apache.giraph.graph.BspServiceWorker.loadVertices(BspServiceWorker.java:304)
>>>>>
>>>>> at
>>>>> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:575)
>>>>>
>>>>> ... 9 more
>>>>> Caused by: java.io.IOException: Call to poodle-6/127.0.1.1:30002 failed
>>>>> on local exception: java.io.EOFException
>>>>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065)
>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1033)
>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
>>>>> at $Proxy3.putVertexList(Unknown Source)
>>>>> at
>>>>> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:765)
>>>>>
>>>>> ... 11 more
>>>>> Caused by: java.io.EOFException
>>>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>>>> at
>>>>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767)
>>>>>
>>>>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712)
>>>>
Re: Unable to load vertices
Posted by Sebastian Schelter <ss...@apache.org>.
I'm extending org.apache.giraph.graph.Vertex directly. I also created
unit tests for the serialization of the Writables (writing them to a
byte array and reading them back) without finding something. Thank you
for the advice however, I'll continue searching :)
--sebastian
On 23.12.2011 18:14, Avery Ching wrote:
> What MutableVertex implementation are you using? Sounds like the issue
> only happens during the RPC to send the vertex to another worker. Maybe
> a bug in the Writable implementation?
>
> Avery
>
> On 12/23/11 3:14 AM, Sebastian Schelter wrote:
>> Hmm, the job works if I use a single worker only locally, strange...
>>
>> On 23.12.2011 11:07, Claudio Martella wrote:
>>> With a super quick look, so i might be completely wrong, this looks
>>> like you're running a different hadoop locally and on your test. Is
>>> there any chance you're not using hadoop non_secure locally but you're
>>> in your distributed mode?
>>>
>>> On Fri, Dec 23, 2011 at 10:49 AM, Sebastian Schelter<ss...@apache.org>
>>> wrote:
>>>> Hi,
>>>>
>>>> I'm currently implementing an algorithm for diameter and radius
>>>> estimation. It already works when I run it on toy data via
>>>> InternalVertexRunner in a unit test.
>>>>
>>>> Unfortunately, in my tests with a single node hadoop instance and real
>>>> cluster, I always run into the attached exception during startup. Does
>>>> anybody have an idea what might cause this?
>>>>
>>>> --sebastian
>>>>
>>>>
>>>> 2011-12-23 10:43:09,769 INFO org.apache.hadoop.mapred.TaskInProgress:
>>>> Error from attempt_201112230924_0006_m_000001_0:
>>>> java.lang.IllegalStateException: run: Caught an unrecoverable exception
>>>> setup: Offlining servers due to exception...
>>>> at
>>>> org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:641)
>>>> at
>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>>>>
>>>> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>>>> Caused by: java.lang.RuntimeException: setup: Offlining servers due to
>>>> exception...
>>>> at
>>>> org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:466)
>>>> at
>>>> org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:630)
>>>> ... 7 more
>>>> Caused by: java.lang.IllegalStateException: setup: loadVertices failed
>>>> at
>>>> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:582)
>>>>
>>>> at
>>>> org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:458)
>>>> ... 8 more
>>>> Caused by: java.lang.RuntimeException: java.io.IOException: Call to
>>>> poodle-6/127.0.1.1:30002 failed on local exception:
>>>> java.io.EOFException
>>>> at
>>>> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:768)
>>>>
>>>> at
>>>> org.apache.giraph.graph.BspServiceWorker.loadVertices(BspServiceWorker.java:304)
>>>>
>>>> at
>>>> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:575)
>>>>
>>>> ... 9 more
>>>> Caused by: java.io.IOException: Call to poodle-6/127.0.1.1:30002 failed
>>>> on local exception: java.io.EOFException
>>>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065)
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1033)
>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
>>>> at $Proxy3.putVertexList(Unknown Source)
>>>> at
>>>> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:765)
>>>>
>>>> ... 11 more
>>>> Caused by: java.io.EOFException
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>>> at
>>>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767)
>>>>
>>>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712)
>>>
>>>
>
Re: Unable to load vertices
Posted by Avery Ching <ac...@apache.org>.
What MutableVertex implementation are you using? Sounds like the issue
only happens during the RPC to send the vertex to another worker. Maybe
a bug in the Writable implementation?
Avery
On 12/23/11 3:14 AM, Sebastian Schelter wrote:
> Hmm, the job works if I use a single worker only locally, strange...
>
> On 23.12.2011 11:07, Claudio Martella wrote:
>> With a super quick look, so i might be completely wrong, this looks
>> like you're running a different hadoop locally and on your test. Is
>> there any chance you're not using hadoop non_secure locally but you're
>> in your distributed mode?
>>
>> On Fri, Dec 23, 2011 at 10:49 AM, Sebastian Schelter<ss...@apache.org> wrote:
>>> Hi,
>>>
>>> I'm currently implementing an algorithm for diameter and radius
>>> estimation. It already works when I run it on toy data via
>>> InternalVertexRunner in a unit test.
>>>
>>> Unfortunately, in my tests with a single node hadoop instance and real
>>> cluster, I always run into the attached exception during startup. Does
>>> anybody have an idea what might cause this?
>>>
>>> --sebastian
>>>
>>>
>>> 2011-12-23 10:43:09,769 INFO org.apache.hadoop.mapred.TaskInProgress:
>>> Error from attempt_201112230924_0006_m_000001_0:
>>> java.lang.IllegalStateException: run: Caught an unrecoverable exception
>>> setup: Offlining servers due to exception...
>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:641)
>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>>> Caused by: java.lang.RuntimeException: setup: Offlining servers due to
>>> exception...
>>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:466)
>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:630)
>>> ... 7 more
>>> Caused by: java.lang.IllegalStateException: setup: loadVertices failed
>>> at
>>> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:582)
>>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:458)
>>> ... 8 more
>>> Caused by: java.lang.RuntimeException: java.io.IOException: Call to
>>> poodle-6/127.0.1.1:30002 failed on local exception: java.io.EOFException
>>> at
>>> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:768)
>>> at
>>> org.apache.giraph.graph.BspServiceWorker.loadVertices(BspServiceWorker.java:304)
>>> at
>>> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:575)
>>> ... 9 more
>>> Caused by: java.io.IOException: Call to poodle-6/127.0.1.1:30002 failed
>>> on local exception: java.io.EOFException
>>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065)
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1033)
>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
>>> at $Proxy3.putVertexList(Unknown Source)
>>> at
>>> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:765)
>>> ... 11 more
>>> Caused by: java.io.EOFException
>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767)
>>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712)
>>
>>
Re: Unable to load vertices
Posted by Sebastian Schelter <ss...@apache.org>.
Hmm, the job works if I use a single worker only locally, strange...
On 23.12.2011 11:07, Claudio Martella wrote:
> With a super quick look, so i might be completely wrong, this looks
> like you're running a different hadoop locally and on your test. Is
> there any chance you're not using hadoop non_secure locally but you're
> in your distributed mode?
>
> On Fri, Dec 23, 2011 at 10:49 AM, Sebastian Schelter <ss...@apache.org> wrote:
>> Hi,
>>
>> I'm currently implementing an algorithm for diameter and radius
>> estimation. It already works when I run it on toy data via
>> InternalVertexRunner in a unit test.
>>
>> Unfortunately, in my tests with a single node hadoop instance and real
>> cluster, I always run into the attached exception during startup. Does
>> anybody have an idea what might cause this?
>>
>> --sebastian
>>
>>
>> 2011-12-23 10:43:09,769 INFO org.apache.hadoop.mapred.TaskInProgress:
>> Error from attempt_201112230924_0006_m_000001_0:
>> java.lang.IllegalStateException: run: Caught an unrecoverable exception
>> setup: Offlining servers due to exception...
>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:641)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>> Caused by: java.lang.RuntimeException: setup: Offlining servers due to
>> exception...
>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:466)
>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:630)
>> ... 7 more
>> Caused by: java.lang.IllegalStateException: setup: loadVertices failed
>> at
>> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:582)
>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:458)
>> ... 8 more
>> Caused by: java.lang.RuntimeException: java.io.IOException: Call to
>> poodle-6/127.0.1.1:30002 failed on local exception: java.io.EOFException
>> at
>> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:768)
>> at
>> org.apache.giraph.graph.BspServiceWorker.loadVertices(BspServiceWorker.java:304)
>> at
>> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:575)
>> ... 9 more
>> Caused by: java.io.IOException: Call to poodle-6/127.0.1.1:30002 failed
>> on local exception: java.io.EOFException
>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1033)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
>> at $Proxy3.putVertexList(Unknown Source)
>> at
>> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:765)
>> ... 11 more
>> Caused by: java.io.EOFException
>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767)
>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712)
>
>
>
Re: Unable to load vertices
Posted by Sebastian Schelter <ss...@apache.org>.
Shouldn't be the case, I always use 0.20.203... I also implemented a
very similar algorithm to find all connected components in a graph and
that one worked fine.
--sebastian
On 23.12.2011 11:07, Claudio Martella wrote:
> With a super quick look, so i might be completely wrong, this looks
> like you're running a different hadoop locally and on your test. Is
> there any chance you're not using hadoop non_secure locally but you're
> in your distributed mode?
>
> On Fri, Dec 23, 2011 at 10:49 AM, Sebastian Schelter <ss...@apache.org> wrote:
>> Hi,
>>
>> I'm currently implementing an algorithm for diameter and radius
>> estimation. It already works when I run it on toy data via
>> InternalVertexRunner in a unit test.
>>
>> Unfortunately, in my tests with a single node hadoop instance and real
>> cluster, I always run into the attached exception during startup. Does
>> anybody have an idea what might cause this?
>>
>> --sebastian
>>
>>
>> 2011-12-23 10:43:09,769 INFO org.apache.hadoop.mapred.TaskInProgress:
>> Error from attempt_201112230924_0006_m_000001_0:
>> java.lang.IllegalStateException: run: Caught an unrecoverable exception
>> setup: Offlining servers due to exception...
>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:641)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>> Caused by: java.lang.RuntimeException: setup: Offlining servers due to
>> exception...
>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:466)
>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:630)
>> ... 7 more
>> Caused by: java.lang.IllegalStateException: setup: loadVertices failed
>> at
>> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:582)
>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:458)
>> ... 8 more
>> Caused by: java.lang.RuntimeException: java.io.IOException: Call to
>> poodle-6/127.0.1.1:30002 failed on local exception: java.io.EOFException
>> at
>> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:768)
>> at
>> org.apache.giraph.graph.BspServiceWorker.loadVertices(BspServiceWorker.java:304)
>> at
>> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:575)
>> ... 9 more
>> Caused by: java.io.IOException: Call to poodle-6/127.0.1.1:30002 failed
>> on local exception: java.io.EOFException
>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1033)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
>> at $Proxy3.putVertexList(Unknown Source)
>> at
>> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:765)
>> ... 11 more
>> Caused by: java.io.EOFException
>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767)
>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712)
>
>
>
Re: Unable to load vertices
Posted by Claudio Martella <cl...@gmail.com>.
With a super quick look, so i might be completely wrong, this looks
like you're running a different hadoop locally and on your test. Is
there any chance you're not using hadoop non_secure locally but you're
in your distributed mode?
On Fri, Dec 23, 2011 at 10:49 AM, Sebastian Schelter <ss...@apache.org> wrote:
> Hi,
>
> I'm currently implementing an algorithm for diameter and radius
> estimation. It already works when I run it on toy data via
> InternalVertexRunner in a unit test.
>
> Unfortunately, in my tests with a single node hadoop instance and real
> cluster, I always run into the attached exception during startup. Does
> anybody have an idea what might cause this?
>
> --sebastian
>
>
> 2011-12-23 10:43:09,769 INFO org.apache.hadoop.mapred.TaskInProgress:
> Error from attempt_201112230924_0006_m_000001_0:
> java.lang.IllegalStateException: run: Caught an unrecoverable exception
> setup: Offlining servers due to exception...
> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:641)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
> Caused by: java.lang.RuntimeException: setup: Offlining servers due to
> exception...
> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:466)
> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:630)
> ... 7 more
> Caused by: java.lang.IllegalStateException: setup: loadVertices failed
> at
> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:582)
> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:458)
> ... 8 more
> Caused by: java.lang.RuntimeException: java.io.IOException: Call to
> poodle-6/127.0.1.1:30002 failed on local exception: java.io.EOFException
> at
> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:768)
> at
> org.apache.giraph.graph.BspServiceWorker.loadVertices(BspServiceWorker.java:304)
> at
> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:575)
> ... 9 more
> Caused by: java.io.IOException: Call to poodle-6/127.0.1.1:30002 failed
> on local exception: java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065)
> at org.apache.hadoop.ipc.Client.call(Client.java:1033)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
> at $Proxy3.putVertexList(Unknown Source)
> at
> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:765)
> ... 11 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712)
--
Claudio Martella
claudio.martella@gmail.com