You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by Arghya Kusum Das <ar...@gmail.com> on 2014/09/24 06:07:40 UTC
Problem in running SSSP on hadoop-0.23.10 (insisting use LocalJob)
Hi,
I was trying to run the giraph simple shortest path example (that comes
with giraph package) on 3 nodes (1 namenode and 2 datanodes) hadoop-0.23.10
cluster. I used 7 mappers and 7 reducers run simultaneously at each node in
mapred-site.xml.
The command I typed:
$HADOOP_HOME/bin/hadoop jar
/usr/local/giraph-1.1.0/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-0.23.1-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/hduser/input/tiny_graph.txt -of
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/output/shortestpaths *-w 12*
When I ran with 12 workers (-w 12) the error was :
'Exception in thread "main" java.lang.IllegalArgumentException:
checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have only
one worker since only 1 task at a time!'
When I change it to 1 (-w 1) the error was:
'Exception in thread "main" java.lang.IllegalArgumentException:
checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run
in split master / worker mode since there is only 1 task at a time!'
What am I missing?
--
Thanks and regards,
Arghya Kusum Das
(225-362-4031)
Re: Problem in running SSSP on hadoop-0.23.10 (insisting use LocalJob)
Posted by Arghya Kusum Das <ar...@gmail.com>.
Hi,
I figured out that we need to use mapreduce.jobtracker.address instead of
mapred.job.tracker in order to get rid of this error (with
giraph1.1.0/hadoop-0.23.10). So the job is starting now.
But, Now I am getting another error right after starting the job (Something
related to zookeeper).... I tried with my own code also (which worked in
previous hadoop platform like giraph1.1.0/hadoop-0.20.203.0). But got the
same error.
2014-09-24 10:41:21,943 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.cluster.local.dir; Ignoring.
2014-09-24 10:41:21,944 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2014-09-24 10:41:21,945 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2014-09-24 10:41:21,946 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: mapreduce.cluster.temp.dir;
Ignoring.
2014-09-24 10:41:21,947 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: Log level remains at info
2014-09-24 10:41:21,981 INFO [main]
org.apache.giraph.graph.GraphTaskManager: Distributed cache is empty.
Assuming fatjar.
2014-09-24 10:41:21,981 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: classpath @
/tmp/hadoop-yarn/staging/hadoop/.staging/job_1411572537241_0002/job.jar for
job MySimpleShortestPathsExample
2014-09-24 10:41:21,983 WARN [main] org.apache.hadoop.conf.Configuration:
mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
2014-09-24 10:41:21,983 WARN [main] org.apache.hadoop.conf.Configuration:
mapred.job.id is deprecated. Instead, use mapreduce.job.id
2014-09-24 10:41:21,983 WARN [main] org.apache.hadoop.conf.Configuration:
job.local.dir is deprecated. Instead, use mapreduce.job.local.dir
2014-09-24 10:41:21,986 INFO [main] org.apache.giraph.zk.ZooKeeperManager:
createCandidateStamp: Made the directory
_bsp/_defaultZkManagerDir/job_1411572537241_0002
2014-09-24 10:41:21,992 INFO [main] org.apache.giraph.zk.ZooKeeperManager:
createCandidateStamp: Creating my filestamp
_bsp/_defaultZkManagerDir/job_1411572537241_0002/_task/
bicstorage2.cct.lsu.edu 3
2014-09-24 10:41:22,023 INFO [main] org.apache.giraph.zk.ZooKeeperManager:
getZooKeeperServerList: For task 3, got file '
zkServerList_bicstorage5.cct.lsu.edu 0 ' (polling period is 3000)
2014-09-24 10:41:22,023 INFO [main] org.apache.giraph.zk.ZooKeeperManager:
getZooKeeperServerList: Found [bicstorage5.cct.lsu.edu, 0] 2 hosts in
filename 'zkServerList_bicstorage5.cct.lsu.edu 0 '
2014-09-24 10:41:22,032 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.IllegalStateException: run: Caught an
unrecoverable exception java.io.FileNotFoundException: File
_bsp/_defaultZkManagerDir/job_1411572537241_0002/_zkServer does not exist.
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:726)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1264)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File
_bsp/_defaultZkManagerDir/job_1411572537241_0002/_zkServer does not exist.
at
org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:792)
at
org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:371)
at
org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:204)
at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89)
... 7 more
Caused by: java.io.FileNotFoundException: File
_bsp/_defaultZkManagerDir/job_1411572537241_0002/_zkServer does not exist.
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:362)
at
org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:751)
... 11 more
2014-09-24 10:41:22,037 INFO [main] org.apache.hadoop.mapred.Task: Runnning
cleanup for the task
2014-09-24 10:41:22,063 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.cluster.local.dir; Ignoring.
2014-09-24 10:41:22,064 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.cluster.local.dir; Ignoring.
2014-09-24 10:41:22,064 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2014-09-24 10:41:22,064 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2014-09-24 10:41:22,065 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: mapreduce.cluster.temp.dir;
Ignoring.
2014-09-24 10:41:22,068 WARN [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not
delete hdfs://
bicstorage1.cct.lsu.edu:9010/output/_temporary/1/_temporary/attempt_1411572537241_0002_m_000003_0
2014-09-24 10:41:22,071 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics
system...
2014-09-24 10:41:22,071 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
stopped.
2014-09-24 10:41:22,071 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
shutdown complete.
On Wed, Sep 24, 2014 at 10:24 AM, Arghya Kusum Das <
arghyakusumdas2266@gmail.com> wrote:
> Yes...it is set properly.... Still this error....
>
> On Wed, Sep 24, 2014 at 9:33 AM, xuhong zhang <xu...@gmail.com>
> wrote:
>
>> Make sure set this value in your mapred-site.xml
>>
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>***********************</value>
>> <description>jobtracker's address</description>
>> </property>
>>
>> On Wed, Sep 24, 2014 at 12:07 AM, Arghya Kusum Das <
>> arghyakusumdas2266@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I was trying to run the giraph simple shortest path example (that comes
>> > with giraph package) on 3 nodes (1 namenode and 2 datanodes)
>> hadoop-0.23.10
>> > cluster. I used 7 mappers and 7 reducers run simultaneously at each
>> node in
>> > mapred-site.xml.
>> >
>> > The command I typed:
>> > $HADOOP_HOME/bin/hadoop jar
>> >
>> >
>> /usr/local/giraph-1.1.0/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-0.23.1-jar-with-dependencies.jar
>> > org.apache.giraph.GiraphRunner
>> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> > -vip /user/hduser/input/tiny_graph.txt -of
>> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> > /output/shortestpaths *-w 12*
>> >
>> > When I ran with 12 workers (-w 12) the error was :
>> > 'Exception in thread "main" java.lang.IllegalArgumentException:
>> > checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have
>> only
>> > one worker since only 1 task at a time!'
>> >
>> > When I change it to 1 (-w 1) the error was:
>> > 'Exception in thread "main" java.lang.IllegalArgumentException:
>> > checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot
>> run
>> > in split master / worker mode since there is only 1 task at a time!'
>> >
>> > What am I missing?
>> >
>> > --
>> > Thanks and regards,
>> > Arghya Kusum Das
>> > (225-362-4031)
>> >
>>
>>
>>
>> --
>> Xuhong Zhang
>>
>
>
>
> --
> Thanks and regards,
> Arghya Kusum Das
> (225-362-4031)
>
--
Thanks and regards,
Arghya Kusum Das
(225-362-4031)
Re: Problem in running SSSP on hadoop-0.23.10 (insisting use LocalJob)
Posted by Arghya Kusum Das <ar...@gmail.com>.
Yes...it is set properly.... Still this error....
On Wed, Sep 24, 2014 at 9:33 AM, xuhong zhang <xu...@gmail.com> wrote:
> Make sure set this value in your mapred-site.xml
>
> <property>
> <name>mapred.job.tracker</name>
> <value>***********************</value>
> <description>jobtracker's address</description>
> </property>
>
> On Wed, Sep 24, 2014 at 12:07 AM, Arghya Kusum Das <
> arghyakusumdas2266@gmail.com> wrote:
>
> > Hi,
> >
> > I was trying to run the giraph simple shortest path example (that comes
> > with giraph package) on 3 nodes (1 namenode and 2 datanodes)
> hadoop-0.23.10
> > cluster. I used 7 mappers and 7 reducers run simultaneously at each node
> in
> > mapred-site.xml.
> >
> > The command I typed:
> > $HADOOP_HOME/bin/hadoop jar
> >
> >
> /usr/local/giraph-1.1.0/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-0.23.1-jar-with-dependencies.jar
> > org.apache.giraph.GiraphRunner
> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> > -vip /user/hduser/input/tiny_graph.txt -of
> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> > /output/shortestpaths *-w 12*
> >
> > When I ran with 12 workers (-w 12) the error was :
> > 'Exception in thread "main" java.lang.IllegalArgumentException:
> > checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have
> only
> > one worker since only 1 task at a time!'
> >
> > When I change it to 1 (-w 1) the error was:
> > 'Exception in thread "main" java.lang.IllegalArgumentException:
> > checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot
> run
> > in split master / worker mode since there is only 1 task at a time!'
> >
> > What am I missing?
> >
> > --
> > Thanks and regards,
> > Arghya Kusum Das
> > (225-362-4031)
> >
>
>
>
> --
> Xuhong Zhang
>
--
Thanks and regards,
Arghya Kusum Das
(225-362-4031)
Re: Problem in running SSSP on hadoop-0.23.10 (insisting use LocalJob)
Posted by xuhong zhang <xu...@gmail.com>.
Make sure set this value in your mapred-site.xml
<property>
<name>mapred.job.tracker</name>
<value>***********************</value>
<description>jobtracker's address</description>
</property>
On Wed, Sep 24, 2014 at 12:07 AM, Arghya Kusum Das <
arghyakusumdas2266@gmail.com> wrote:
> Hi,
>
> I was trying to run the giraph simple shortest path example (that comes
> with giraph package) on 3 nodes (1 namenode and 2 datanodes) hadoop-0.23.10
> cluster. I used 7 mappers and 7 reducers run simultaneously at each node in
> mapred-site.xml.
>
> The command I typed:
> $HADOOP_HOME/bin/hadoop jar
>
> /usr/local/giraph-1.1.0/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-0.23.1-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /user/hduser/input/tiny_graph.txt -of
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /output/shortestpaths *-w 12*
>
> When I ran with 12 workers (-w 12) the error was :
> 'Exception in thread "main" java.lang.IllegalArgumentException:
> checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have only
> one worker since only 1 task at a time!'
>
> When I change it to 1 (-w 1) the error was:
> 'Exception in thread "main" java.lang.IllegalArgumentException:
> checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run
> in split master / worker mode since there is only 1 task at a time!'
>
> What am I missing?
>
> --
> Thanks and regards,
> Arghya Kusum Das
> (225-362-4031)
>
--
Xuhong Zhang