You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Roman Shaposhnik <ro...@shaposhnik.org> on 2012/05/25 18:24:00 UTC
Giraph on Hadoop 2.0.0-alpha
Hi!
I'm trying to run Giraph trunk on top of Hadoop 2.0.0 and I'm getting
the following error while submitting an example job:
$ hadoop jar /usr/lib/giraph/giraph-jar-with-dependencies.jar
org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -V 100000 -w 3
Now, if I look at the state of HDFS right after the job fails I see that
the job has created file structure all the way up to
_bsp/_defaultZkManagerDir/job_1337959594450_0002/
I even see
_bsp/_defaultZkManagerDir/job_1337959594450_0002/zkServerList_ahmed-laptop 0
so it is unlikely to be file permission problems or anything like that.
Could you, please, suggest a way to debug it from here?
Oh, and here's the exception I'm getting:
2012-05-25 08:31:34,335 INFO [IPC Server handler 16 on 33249]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report
from attempt_1337959594450_0002_m_000001_3: Error:
java.lang.RuntimeException: java.io.FileNotFoundException: File
_bsp/_defaultZkManagerDir/job_1337959594450_0002/_zkServer does not
exist.
at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:748)
at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:424)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:645)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
Caused by: java.io.FileNotFoundException: File
_bsp/_defaultZkManagerDir/job_1337959594450_0002/_zkServer does not
exist.
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:365)
at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:707)
... 9 more
Thanks,
Roman.
Re: Giraph on Hadoop 2.0.0-alpha
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Tue, May 29, 2012 at 1:25 PM, Avery Ching <ac...@apache.org> wrote:
> Did you compile with the appropriate flags?
>
> From the README:
>
> - Apache Hadoop 0.23.1
>
> You may tell maven to use this version with "mvn -Phadoop_0.23 <goals>".
Did that, but given that I was trying to run against 2.0.0-alpha I also had to
manually adjust the version. Perhaps now that 2.0.0-alpha is out it would
be useful to adjust the giraph profile to bind to that version instead of 0.23.
Anyway, I can now successfully submit the giraph job, but the job can't
complete. In fact I see that 3 maps failed with the following. And the remaining
4 maps just refuse to exit (hence the job hangs).
Does this look like a known issue or should I file a JIRA?
2012-05-31 10:31:28,518 WARN [main]
org.apache.hadoop.mapred.YarnChild: Exception running child :
java.lang.RuntimeException: java.lang.NoSuchFieldException: versionID
at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:170)
at org.apache.hadoop.ipc.WritableRpcEngine$Invocation.(WritableRpcEngine.java:110)
at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:224)
at $Proxy11.putVertexList(Unknown Source)
at org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:921)
at org.apache.giraph.graph.BspServiceWorker.loadVertices(BspServiceWorker.java:322)
at org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:594)
at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:473)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:645)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
Caused by: java.lang.NoSuchFieldException: versionID
at java.lang.Class.getField(Class.java:1520)
at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:166)
... 15 more
Thanks,
Roman.
Re: Giraph on Hadoop 2.0.0-alpha
Posted by Avery Ching <ac...@apache.org>.
Did you compile with the appropriate flags?
From the README:
- Apache Hadoop 0.23.1
You may tell maven to use this version with "mvn -Phadoop_0.23 <goals>".
On 5/25/12 9:24 AM, Roman Shaposhnik wrote:
> Hi!
>
> I'm trying to run Giraph trunk on top of Hadoop 2.0.0 and I'm getting
> the following error while submitting an example job:
> $ hadoop jar /usr/lib/giraph/giraph-jar-with-dependencies.jar
> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -V 100000 -w 3
>
> Now, if I look at the state of HDFS right after the job fails I see that
> the job has created file structure all the way up to
> _bsp/_defaultZkManagerDir/job_1337959594450_0002/
> I even see
> _bsp/_defaultZkManagerDir/job_1337959594450_0002/zkServerList_ahmed-laptop 0
> so it is unlikely to be file permission problems or anything like that.
>
> Could you, please, suggest a way to debug it from here?
>
> Oh, and here's the exception I'm getting:
>
> 2012-05-25 08:31:34,335 INFO [IPC Server handler 16 on 33249]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report
> from attempt_1337959594450_0002_m_000001_3: Error:
> java.lang.RuntimeException: java.io.FileNotFoundException: File
> _bsp/_defaultZkManagerDir/job_1337959594450_0002/_zkServer does not
> exist.
> at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:748)
> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:424)
> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:645)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
> Caused by: java.io.FileNotFoundException: File
> _bsp/_defaultZkManagerDir/job_1337959594450_0002/_zkServer does not
> exist.
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:365)
> at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:707)
> ... 9 more
>
> Thanks,
> Roman.