You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Alex Waagen <aw...@gmail.com> on 2013/07/30 17:02:07 UTC
Shortest paths example in giraf 1.1.0.
I am having some trouble getting these examples running. I’m using giraph
version 1.1.0 and hadoop 0.20.2. I am using the following json file as
input:
[0,0,[[1,1],[3,3]]]
[1,0,[[0,1],[2,2],[3,1]]]
[2,0,[[1,2],[4,4]]]
[3,0,[[0,3],[1,1],[4,4]]]
[4,0,[[3,4],[2,4]]]
The command I use is:
hadoop jar
/path-to-giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /path-to-input/input_file.json -of
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/path-to-output/outShortest -w 1
I see the following output.
13/07/29 14:36:06 INFO utils.ConfigurationUtils: No edge input format
specified. Ensure your InputFormat does not require one.
13/07/29 14:36:06 INFO job.GiraphJob: run: Since checkpointing is disabled
(default), do not allow any task retries (setting mapred.map.max.attempts =
0, old value = 4)
13/07/29 14:36:20 INFO mapred.JobClient: Running job: job_201307232135_0588
13/07/29 14:36:21 INFO mapred.JobClient: map 0% reduce 0%
13/07/29 14:36:52 INFO mapred.JobClient: map 50% reduce 0%
13/07/29 14:47:24 INFO mapred.JobClient: map 0% reduce 0%
13/07/29 14:47:39 INFO mapred.JobClient: Job complete: job_201307232135_0588
13/07/29 14:47:39 INFO mapred.JobClient: Counters: 6
13/07/29 14:47:39 INFO mapred.JobClient: Job Counters
13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=670508
13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all reduces
waiting after reserving slots (ms)=0
13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all maps
waiting after reserving slots (ms)=0
13/07/29 14:47:39 INFO mapred.JobClient: Launched map tasks=2
13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
13/07/29 14:47:39 INFO mapred.JobClient: Failed map tasks=1
When I check the job tracker, I see that two map jobs were killed, with the
following errors:
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
java.lang.IllegalStateException: run: Caught an unrecoverable exception
exists: Failed to check
/_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
after 3 tries!
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.IllegalStateException: exists: Failed to check
/_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
after 3 tries!
at org.apache.giraph.zk.ZooKeeperExt.exists(ZooKeeperExt.java:369)
at org.apache.giraph.worker.BspServiceWorker.s
Task attempt_201307232135_0588_m_000001_0 failed to report status for 600
seconds. Killing!
Any idea what the problem is?
Thanks in advance.
Re: Shortest paths example in giraf 1.1.0.
Posted by Alex Waagen <aw...@gmail.com>.
I think the problem may have something to do with the fact that we are
using cloudera. Is there anyone who has experience with cloudera who may
have some idea how to fix this?
On Wed, Jul 31, 2013 at 6:59 AM, Alex Waagen <aw...@gmail.com> wrote:
> I tried running both the shortest paths and pagerank examples, but I am
> still getting the same errors.
>
>
> On Tue, Jul 30, 2013 at 12:49 PM, Ryan Compton <co...@gmail.com>wrote:
>
>> Hey Alex,
>>
>> Our cluster uses Cloudera for Hadoop admin stuff. To the best of my
>> knowledge, we need to do our configs through the Cloudera GUI (which
>> is a royal pain).
>>
>> The "mapred.tasktracker.map.tasks.maximum" should be at 16 everywhere
>> (save for a few of the higher nodes which already had it at 32).
>> However, I can't figure how to get at "mapred.map.tasks" from the
>> Cloudera GUI. Can you give it another try?
>>
>> -Ryan
>>
>> On Tue, Jul 30, 2013 at 8:58 AM, Kyle Orlando <ky...@gmail.com>
>> wrote:
>> > Ah, that might be your problem.
>> >
>> > Try adding this between the <configuration> and </configuration>:
>> >
>> > <property>
>> > <name>mapred.tasktracker.map.tasks.maximum</name>
>> > <value>4</value>
>> > </property>
>> >
>> > <property>
>> > <name>mapred.map.tasks</name>
>> > <value>4</value>
>> > </property>
>> >
>> > See if it works, that's really the only thing I can think of.
>> >
>> > By default, the max number of map tasks and reduce tasksfor Hadoop is
>> > 2. This changes the max number of map tasks to 4, and "hints" to
>> > Hadoop (whatever that means) that it should utilize 4 map tasks. I
>> > believe that Giraph workers hijack map tasks, so reduce tasks are
>> > unneeded, but someone who is more familiar with Giraph will have to
>> > tell you more.
>> >
>> > On Tue, Jul 30, 2013 at 11:38 AM, Alex Waagen <aw...@gmail.com>
>> wrote:
>> >> Here is the file. It is almost empty.
>> >>
>> >> <?xml version="1.0"?>
>> >> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> >>
>> >> <!-- Put site-specific property overrides in this file. -->
>> >>
>> >> <configuration>
>> >>
>> >> </configuration>
>> >>
>> >>
>> >>
>> >> On Tue, Jul 30, 2013 at 8:33 AM, Kyle Orlando <
>> kyle.r.orlando@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hmmm, could you post the contents of your mapred-site.xml in
>> >>> $HADOOP_HOME/conf? You may need to increase the number of map tasks.
>> >>>
>> >>> On Tue, Jul 30, 2013 at 11:02 AM, Alex Waagen <aw...@gmail.com>
>> wrote:
>> >>> > I am having some trouble getting these examples running. I’m using
>> >>> > giraph
>> >>> > version 1.1.0 and hadoop 0.20.2. I am using the following json file
>> as
>> >>> > input:
>> >>> >
>> >>> > [0,0,[[1,1],[3,3]]]
>> >>> > [1,0,[[0,1],[2,2],[3,1]]]
>> >>> > [2,0,[[1,2],[4,4]]]
>> >>> > [3,0,[[0,3],[1,1],[4,4]]]
>> >>> > [4,0,[[3,4],[2,4]]]
>> >>> >
>> >>> > The command I use is:
>> >>> >
>> >>> > hadoop jar
>> >>> >
>> >>> >
>> /path-to-giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>> >>> > org.apache.giraph.GiraphRunner
>> >>> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> >>> >
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> >>> > -vip
>> >>> > /path-to-input/input_file.json -of
>> >>> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> >>> > /path-to-output/outShortest -w 1
>> >>> >
>> >>> >
>> >>> > I see the following output.
>> >>> >
>> >>> > 13/07/29 14:36:06 INFO utils.ConfigurationUtils: No edge input
>> format
>> >>> > specified. Ensure your InputFormat does not require one.
>> >>> > 13/07/29 14:36:06 INFO job.GiraphJob: run: Since checkpointing is
>> >>> > disabled
>> >>> > (default), do not allow any task retries (setting
>> >>> > mapred.map.max.attempts =
>> >>> > 0, old value = 4)
>> >>> > 13/07/29 14:36:20 INFO mapred.JobClient: Running job:
>> >>> > job_201307232135_0588
>> >>> > 13/07/29 14:36:21 INFO mapred.JobClient: map 0% reduce 0%
>> >>> > 13/07/29 14:36:52 INFO mapred.JobClient: map 50% reduce 0%
>> >>> > 13/07/29 14:47:24 INFO mapred.JobClient: map 0% reduce 0%
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Job complete:
>> >>> > job_201307232135_0588
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Counters: 6
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Job Counters
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=670508
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all
>> reduces
>> >>> > waiting after reserving slots (ms)=0
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all
>> maps
>> >>> > waiting after reserving slots (ms)=0
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Launched map tasks=2
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Failed map tasks=1
>> >>> >
>> >>> >
>> >>> > When I check the job tracker, I see that two map jobs were killed,
>> with
>> >>> > the
>> >>> > following errors:
>> >>> >
>> >>> > java.lang.Throwable: Child Error
>> >>> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
>> >>> > Caused by: java.io.IOException: Task process exit with nonzero
>> status of
>> >>> > 1.
>> >>> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>> >>> >
>> >>> > java.lang.IllegalStateException: run: Caught an unrecoverable
>> exception
>> >>> > exists: Failed to check
>> >>> >
>> >>> >
>> /_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
>> >>> > after 3 tries!
>> >>> > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
>> >>> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >>> > at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >>> > at java.security.AccessController.doPrivileged(Native Method)
>> >>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> >>> > at
>> >>> >
>> >>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
>> >>> > at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >>> > Caused by: java.lang.IllegalStateException: exists: Failed to check
>> >>> >
>> >>> >
>> /_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
>> >>> > after 3 tries!
>> >>> > at org.apache.giraph.zk.ZooKeeperExt.exists(ZooKeeperExt.java:369)
>> >>> > at org.apache.giraph.worker.BspServiceWorker.s
>> >>> > Task attempt_201307232135_0588_m_000001_0 failed to report status
>> for
>> >>> > 600
>> >>> > seconds. Killing!
>> >>> >
>> >>> > Any idea what the problem is?
>> >>> > Thanks in advance.
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Kyle Orlando
>> >>> Computer Engineering Major
>> >>> University of Maryland
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Kyle Orlando
>> > Computer Engineering Major
>> > University of Maryland
>>
>
>
Re: Shortest paths example in giraf 1.1.0.
Posted by Alex Waagen <aw...@gmail.com>.
I tried running both the shortest paths and pagerank examples, but I am
still getting the same errors.
On Tue, Jul 30, 2013 at 12:49 PM, Ryan Compton <co...@gmail.com>wrote:
> Hey Alex,
>
> Our cluster uses Cloudera for Hadoop admin stuff. To the best of my
> knowledge, we need to do our configs through the Cloudera GUI (which
> is a royal pain).
>
> The "mapred.tasktracker.map.tasks.maximum" should be at 16 everywhere
> (save for a few of the higher nodes which already had it at 32).
> However, I can't figure how to get at "mapred.map.tasks" from the
> Cloudera GUI. Can you give it another try?
>
> -Ryan
>
> On Tue, Jul 30, 2013 at 8:58 AM, Kyle Orlando <ky...@gmail.com>
> wrote:
> > Ah, that might be your problem.
> >
> > Try adding this between the <configuration> and </configuration>:
> >
> > <property>
> > <name>mapred.tasktracker.map.tasks.maximum</name>
> > <value>4</value>
> > </property>
> >
> > <property>
> > <name>mapred.map.tasks</name>
> > <value>4</value>
> > </property>
> >
> > See if it works, that's really the only thing I can think of.
> >
> > By default, the max number of map tasks and reduce tasksfor Hadoop is
> > 2. This changes the max number of map tasks to 4, and "hints" to
> > Hadoop (whatever that means) that it should utilize 4 map tasks. I
> > believe that Giraph workers hijack map tasks, so reduce tasks are
> > unneeded, but someone who is more familiar with Giraph will have to
> > tell you more.
> >
> > On Tue, Jul 30, 2013 at 11:38 AM, Alex Waagen <aw...@gmail.com> wrote:
> >> Here is the file. It is almost empty.
> >>
> >> <?xml version="1.0"?>
> >> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >>
> >> <!-- Put site-specific property overrides in this file. -->
> >>
> >> <configuration>
> >>
> >> </configuration>
> >>
> >>
> >>
> >> On Tue, Jul 30, 2013 at 8:33 AM, Kyle Orlando <kyle.r.orlando@gmail.com
> >
> >> wrote:
> >>>
> >>> Hmmm, could you post the contents of your mapred-site.xml in
> >>> $HADOOP_HOME/conf? You may need to increase the number of map tasks.
> >>>
> >>> On Tue, Jul 30, 2013 at 11:02 AM, Alex Waagen <aw...@gmail.com>
> wrote:
> >>> > I am having some trouble getting these examples running. I’m using
> >>> > giraph
> >>> > version 1.1.0 and hadoop 0.20.2. I am using the following json file
> as
> >>> > input:
> >>> >
> >>> > [0,0,[[1,1],[3,3]]]
> >>> > [1,0,[[0,1],[2,2],[3,1]]]
> >>> > [2,0,[[1,2],[4,4]]]
> >>> > [3,0,[[0,3],[1,1],[4,4]]]
> >>> > [4,0,[[3,4],[2,4]]]
> >>> >
> >>> > The command I use is:
> >>> >
> >>> > hadoop jar
> >>> >
> >>> >
> /path-to-giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
> >>> > org.apache.giraph.GiraphRunner
> >>> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >>> >
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> >>> > -vip
> >>> > /path-to-input/input_file.json -of
> >>> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >>> > /path-to-output/outShortest -w 1
> >>> >
> >>> >
> >>> > I see the following output.
> >>> >
> >>> > 13/07/29 14:36:06 INFO utils.ConfigurationUtils: No edge input format
> >>> > specified. Ensure your InputFormat does not require one.
> >>> > 13/07/29 14:36:06 INFO job.GiraphJob: run: Since checkpointing is
> >>> > disabled
> >>> > (default), do not allow any task retries (setting
> >>> > mapred.map.max.attempts =
> >>> > 0, old value = 4)
> >>> > 13/07/29 14:36:20 INFO mapred.JobClient: Running job:
> >>> > job_201307232135_0588
> >>> > 13/07/29 14:36:21 INFO mapred.JobClient: map 0% reduce 0%
> >>> > 13/07/29 14:36:52 INFO mapred.JobClient: map 50% reduce 0%
> >>> > 13/07/29 14:47:24 INFO mapred.JobClient: map 0% reduce 0%
> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Job complete:
> >>> > job_201307232135_0588
> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Counters: 6
> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Job Counters
> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=670508
> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all
> reduces
> >>> > waiting after reserving slots (ms)=0
> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all maps
> >>> > waiting after reserving slots (ms)=0
> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Launched map tasks=2
> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Failed map tasks=1
> >>> >
> >>> >
> >>> > When I check the job tracker, I see that two map jobs were killed,
> with
> >>> > the
> >>> > following errors:
> >>> >
> >>> > java.lang.Throwable: Child Error
> >>> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
> >>> > Caused by: java.io.IOException: Task process exit with nonzero
> status of
> >>> > 1.
> >>> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
> >>> >
> >>> > java.lang.IllegalStateException: run: Caught an unrecoverable
> exception
> >>> > exists: Failed to check
> >>> >
> >>> >
> /_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
> >>> > after 3 tries!
> >>> > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
> >>> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> >>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> >>> > at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> >>> > at java.security.AccessController.doPrivileged(Native Method)
> >>> > at javax.security.auth.Subject.doAs(Subject.java:396)
> >>> > at
> >>> >
> >>> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> >>> > at org.apache.hadoop.mapred.Child.main(Child.java:264)
> >>> > Caused by: java.lang.IllegalStateException: exists: Failed to check
> >>> >
> >>> >
> /_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
> >>> > after 3 tries!
> >>> > at org.apache.giraph.zk.ZooKeeperExt.exists(ZooKeeperExt.java:369)
> >>> > at org.apache.giraph.worker.BspServiceWorker.s
> >>> > Task attempt_201307232135_0588_m_000001_0 failed to report status for
> >>> > 600
> >>> > seconds. Killing!
> >>> >
> >>> > Any idea what the problem is?
> >>> > Thanks in advance.
> >>>
> >>>
> >>>
> >>> --
> >>> Kyle Orlando
> >>> Computer Engineering Major
> >>> University of Maryland
> >>
> >>
> >
> >
> >
> > --
> > Kyle Orlando
> > Computer Engineering Major
> > University of Maryland
>
Re: Shortest paths example in giraf 1.1.0.
Posted by Ryan Compton <co...@gmail.com>.
Hey Alex,
Our cluster uses Cloudera for Hadoop admin stuff. To the best of my
knowledge, we need to do our configs through the Cloudera GUI (which
is a royal pain).
The "mapred.tasktracker.map.tasks.maximum" should be at 16 everywhere
(save for a few of the higher nodes which already had it at 32).
However, I can't figure how to get at "mapred.map.tasks" from the
Cloudera GUI. Can you give it another try?
-Ryan
On Tue, Jul 30, 2013 at 8:58 AM, Kyle Orlando <ky...@gmail.com> wrote:
> Ah, that might be your problem.
>
> Try adding this between the <configuration> and </configuration>:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <value>4</value>
> </property>
>
> <property>
> <name>mapred.map.tasks</name>
> <value>4</value>
> </property>
>
> See if it works, that's really the only thing I can think of.
>
> By default, the max number of map tasks and reduce tasksfor Hadoop is
> 2. This changes the max number of map tasks to 4, and "hints" to
> Hadoop (whatever that means) that it should utilize 4 map tasks. I
> believe that Giraph workers hijack map tasks, so reduce tasks are
> unneeded, but someone who is more familiar with Giraph will have to
> tell you more.
>
> On Tue, Jul 30, 2013 at 11:38 AM, Alex Waagen <aw...@gmail.com> wrote:
>> Here is the file. It is almost empty.
>>
>> <?xml version="1.0"?>
>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>
>> <!-- Put site-specific property overrides in this file. -->
>>
>> <configuration>
>>
>> </configuration>
>>
>>
>>
>> On Tue, Jul 30, 2013 at 8:33 AM, Kyle Orlando <ky...@gmail.com>
>> wrote:
>>>
>>> Hmmm, could you post the contents of your mapred-site.xml in
>>> $HADOOP_HOME/conf? You may need to increase the number of map tasks.
>>>
>>> On Tue, Jul 30, 2013 at 11:02 AM, Alex Waagen <aw...@gmail.com> wrote:
>>> > I am having some trouble getting these examples running. I’m using
>>> > giraph
>>> > version 1.1.0 and hadoop 0.20.2. I am using the following json file as
>>> > input:
>>> >
>>> > [0,0,[[1,1],[3,3]]]
>>> > [1,0,[[0,1],[2,2],[3,1]]]
>>> > [2,0,[[1,2],[4,4]]]
>>> > [3,0,[[0,3],[1,1],[4,4]]]
>>> > [4,0,[[3,4],[2,4]]]
>>> >
>>> > The command I use is:
>>> >
>>> > hadoop jar
>>> >
>>> > /path-to-giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>> > org.apache.giraph.GiraphRunner
>>> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>> > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>> > -vip
>>> > /path-to-input/input_file.json -of
>>> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>> > /path-to-output/outShortest -w 1
>>> >
>>> >
>>> > I see the following output.
>>> >
>>> > 13/07/29 14:36:06 INFO utils.ConfigurationUtils: No edge input format
>>> > specified. Ensure your InputFormat does not require one.
>>> > 13/07/29 14:36:06 INFO job.GiraphJob: run: Since checkpointing is
>>> > disabled
>>> > (default), do not allow any task retries (setting
>>> > mapred.map.max.attempts =
>>> > 0, old value = 4)
>>> > 13/07/29 14:36:20 INFO mapred.JobClient: Running job:
>>> > job_201307232135_0588
>>> > 13/07/29 14:36:21 INFO mapred.JobClient: map 0% reduce 0%
>>> > 13/07/29 14:36:52 INFO mapred.JobClient: map 50% reduce 0%
>>> > 13/07/29 14:47:24 INFO mapred.JobClient: map 0% reduce 0%
>>> > 13/07/29 14:47:39 INFO mapred.JobClient: Job complete:
>>> > job_201307232135_0588
>>> > 13/07/29 14:47:39 INFO mapred.JobClient: Counters: 6
>>> > 13/07/29 14:47:39 INFO mapred.JobClient: Job Counters
>>> > 13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=670508
>>> > 13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all reduces
>>> > waiting after reserving slots (ms)=0
>>> > 13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all maps
>>> > waiting after reserving slots (ms)=0
>>> > 13/07/29 14:47:39 INFO mapred.JobClient: Launched map tasks=2
>>> > 13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>>> > 13/07/29 14:47:39 INFO mapred.JobClient: Failed map tasks=1
>>> >
>>> >
>>> > When I check the job tracker, I see that two map jobs were killed, with
>>> > the
>>> > following errors:
>>> >
>>> > java.lang.Throwable: Child Error
>>> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
>>> > Caused by: java.io.IOException: Task process exit with nonzero status of
>>> > 1.
>>> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>>> >
>>> > java.lang.IllegalStateException: run: Caught an unrecoverable exception
>>> > exists: Failed to check
>>> >
>>> > /_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
>>> > after 3 tries!
>>> > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
>>> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>>> > at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>> > at java.security.AccessController.doPrivileged(Native Method)
>>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>>> > at
>>> >
>>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
>>> > at org.apache.hadoop.mapred.Child.main(Child.java:264)
>>> > Caused by: java.lang.IllegalStateException: exists: Failed to check
>>> >
>>> > /_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
>>> > after 3 tries!
>>> > at org.apache.giraph.zk.ZooKeeperExt.exists(ZooKeeperExt.java:369)
>>> > at org.apache.giraph.worker.BspServiceWorker.s
>>> > Task attempt_201307232135_0588_m_000001_0 failed to report status for
>>> > 600
>>> > seconds. Killing!
>>> >
>>> > Any idea what the problem is?
>>> > Thanks in advance.
>>>
>>>
>>>
>>> --
>>> Kyle Orlando
>>> Computer Engineering Major
>>> University of Maryland
>>
>>
>
>
>
> --
> Kyle Orlando
> Computer Engineering Major
> University of Maryland
Re: Shortest paths example in giraf 1.1.0.
Posted by Kyle Orlando <ky...@gmail.com>.
Ah, that might be your problem.
Try adding this between the <configuration> and </configuration>:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>4</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>4</value>
</property>
See if it works, that's really the only thing I can think of.
By default, the max number of map tasks and reduce tasksfor Hadoop is
2. This changes the max number of map tasks to 4, and "hints" to
Hadoop (whatever that means) that it should utilize 4 map tasks. I
believe that Giraph workers hijack map tasks, so reduce tasks are
unneeded, but someone who is more familiar with Giraph will have to
tell you more.
On Tue, Jul 30, 2013 at 11:38 AM, Alex Waagen <aw...@gmail.com> wrote:
> Here is the file. It is almost empty.
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
>
> </configuration>
>
>
>
> On Tue, Jul 30, 2013 at 8:33 AM, Kyle Orlando <ky...@gmail.com>
> wrote:
>>
>> Hmmm, could you post the contents of your mapred-site.xml in
>> $HADOOP_HOME/conf? You may need to increase the number of map tasks.
>>
>> On Tue, Jul 30, 2013 at 11:02 AM, Alex Waagen <aw...@gmail.com> wrote:
>> > I am having some trouble getting these examples running. I’m using
>> > giraph
>> > version 1.1.0 and hadoop 0.20.2. I am using the following json file as
>> > input:
>> >
>> > [0,0,[[1,1],[3,3]]]
>> > [1,0,[[0,1],[2,2],[3,1]]]
>> > [2,0,[[1,2],[4,4]]]
>> > [3,0,[[0,3],[1,1],[4,4]]]
>> > [4,0,[[3,4],[2,4]]]
>> >
>> > The command I use is:
>> >
>> > hadoop jar
>> >
>> > /path-to-giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>> > org.apache.giraph.GiraphRunner
>> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> > -vip
>> > /path-to-input/input_file.json -of
>> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> > /path-to-output/outShortest -w 1
>> >
>> >
>> > I see the following output.
>> >
>> > 13/07/29 14:36:06 INFO utils.ConfigurationUtils: No edge input format
>> > specified. Ensure your InputFormat does not require one.
>> > 13/07/29 14:36:06 INFO job.GiraphJob: run: Since checkpointing is
>> > disabled
>> > (default), do not allow any task retries (setting
>> > mapred.map.max.attempts =
>> > 0, old value = 4)
>> > 13/07/29 14:36:20 INFO mapred.JobClient: Running job:
>> > job_201307232135_0588
>> > 13/07/29 14:36:21 INFO mapred.JobClient: map 0% reduce 0%
>> > 13/07/29 14:36:52 INFO mapred.JobClient: map 50% reduce 0%
>> > 13/07/29 14:47:24 INFO mapred.JobClient: map 0% reduce 0%
>> > 13/07/29 14:47:39 INFO mapred.JobClient: Job complete:
>> > job_201307232135_0588
>> > 13/07/29 14:47:39 INFO mapred.JobClient: Counters: 6
>> > 13/07/29 14:47:39 INFO mapred.JobClient: Job Counters
>> > 13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=670508
>> > 13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all reduces
>> > waiting after reserving slots (ms)=0
>> > 13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all maps
>> > waiting after reserving slots (ms)=0
>> > 13/07/29 14:47:39 INFO mapred.JobClient: Launched map tasks=2
>> > 13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>> > 13/07/29 14:47:39 INFO mapred.JobClient: Failed map tasks=1
>> >
>> >
>> > When I check the job tracker, I see that two map jobs were killed, with
>> > the
>> > following errors:
>> >
>> > java.lang.Throwable: Child Error
>> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
>> > Caused by: java.io.IOException: Task process exit with nonzero status of
>> > 1.
>> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>> >
>> > java.lang.IllegalStateException: run: Caught an unrecoverable exception
>> > exists: Failed to check
>> >
>> > /_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
>> > after 3 tries!
>> > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
>> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> > at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
>> > at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> > Caused by: java.lang.IllegalStateException: exists: Failed to check
>> >
>> > /_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
>> > after 3 tries!
>> > at org.apache.giraph.zk.ZooKeeperExt.exists(ZooKeeperExt.java:369)
>> > at org.apache.giraph.worker.BspServiceWorker.s
>> > Task attempt_201307232135_0588_m_000001_0 failed to report status for
>> > 600
>> > seconds. Killing!
>> >
>> > Any idea what the problem is?
>> > Thanks in advance.
>>
>>
>>
>> --
>> Kyle Orlando
>> Computer Engineering Major
>> University of Maryland
>
>
--
Kyle Orlando
Computer Engineering Major
University of Maryland
Re: Shortest paths example in giraf 1.1.0.
Posted by Alex Waagen <aw...@gmail.com>.
Here is the file. It is almost empty.
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
</configuration>
On Tue, Jul 30, 2013 at 8:33 AM, Kyle Orlando <ky...@gmail.com>wrote:
> Hmmm, could you post the contents of your mapred-site.xml in
> $HADOOP_HOME/conf? You may need to increase the number of map tasks.
>
> On Tue, Jul 30, 2013 at 11:02 AM, Alex Waagen <aw...@gmail.com> wrote:
> > I am having some trouble getting these examples running. I’m using giraph
> > version 1.1.0 and hadoop 0.20.2. I am using the following json file as
> > input:
> >
> > [0,0,[[1,1],[3,3]]]
> > [1,0,[[0,1],[2,2],[3,1]]]
> > [2,0,[[1,2],[4,4]]]
> > [3,0,[[0,3],[1,1],[4,4]]]
> > [4,0,[[3,4],[2,4]]]
> >
> > The command I use is:
> >
> > hadoop jar
> >
> /path-to-giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
> > org.apache.giraph.GiraphRunner
> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip
> > /path-to-input/input_file.json -of
> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> > /path-to-output/outShortest -w 1
> >
> >
> > I see the following output.
> >
> > 13/07/29 14:36:06 INFO utils.ConfigurationUtils: No edge input format
> > specified. Ensure your InputFormat does not require one.
> > 13/07/29 14:36:06 INFO job.GiraphJob: run: Since checkpointing is
> disabled
> > (default), do not allow any task retries (setting
> mapred.map.max.attempts =
> > 0, old value = 4)
> > 13/07/29 14:36:20 INFO mapred.JobClient: Running job:
> job_201307232135_0588
> > 13/07/29 14:36:21 INFO mapred.JobClient: map 0% reduce 0%
> > 13/07/29 14:36:52 INFO mapred.JobClient: map 50% reduce 0%
> > 13/07/29 14:47:24 INFO mapred.JobClient: map 0% reduce 0%
> > 13/07/29 14:47:39 INFO mapred.JobClient: Job complete:
> job_201307232135_0588
> > 13/07/29 14:47:39 INFO mapred.JobClient: Counters: 6
> > 13/07/29 14:47:39 INFO mapred.JobClient: Job Counters
> > 13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=670508
> > 13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all reduces
> > waiting after reserving slots (ms)=0
> > 13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all maps
> > waiting after reserving slots (ms)=0
> > 13/07/29 14:47:39 INFO mapred.JobClient: Launched map tasks=2
> > 13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> > 13/07/29 14:47:39 INFO mapred.JobClient: Failed map tasks=1
> >
> >
> > When I check the job tracker, I see that two map jobs were killed, with
> the
> > following errors:
> >
> > java.lang.Throwable: Child Error
> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
> > Caused by: java.io.IOException: Task process exit with nonzero status of
> 1.
> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
> >
> > java.lang.IllegalStateException: run: Caught an unrecoverable exception
> > exists: Failed to check
> >
> /_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
> > after 3 tries!
> > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> > at org.apache.hadoop.mapred.Child.main(Child.java:264)
> > Caused by: java.lang.IllegalStateException: exists: Failed to check
> >
> /_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
> > after 3 tries!
> > at org.apache.giraph.zk.ZooKeeperExt.exists(ZooKeeperExt.java:369)
> > at org.apache.giraph.worker.BspServiceWorker.s
> > Task attempt_201307232135_0588_m_000001_0 failed to report status for 600
> > seconds. Killing!
> >
> > Any idea what the problem is?
> > Thanks in advance.
>
>
>
> --
> Kyle Orlando
> Computer Engineering Major
> University of Maryland
>
Re: Shortest paths example in giraf 1.1.0.
Posted by Kyle Orlando <ky...@gmail.com>.
Hmmm, could you post the contents of your mapred-site.xml in
$HADOOP_HOME/conf? You may need to increase the number of map tasks.
On Tue, Jul 30, 2013 at 11:02 AM, Alex Waagen <aw...@gmail.com> wrote:
> I am having some trouble getting these examples running. I’m using giraph
> version 1.1.0 and hadoop 0.20.2. I am using the following json file as
> input:
>
> [0,0,[[1,1],[3,3]]]
> [1,0,[[0,1],[2,2],[3,1]]]
> [2,0,[[1,2],[4,4]]]
> [3,0,[[0,3],[1,1],[4,4]]]
> [4,0,[[3,4],[2,4]]]
>
> The command I use is:
>
> hadoop jar
> /path-to-giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
> /path-to-input/input_file.json -of
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /path-to-output/outShortest -w 1
>
>
> I see the following output.
>
> 13/07/29 14:36:06 INFO utils.ConfigurationUtils: No edge input format
> specified. Ensure your InputFormat does not require one.
> 13/07/29 14:36:06 INFO job.GiraphJob: run: Since checkpointing is disabled
> (default), do not allow any task retries (setting mapred.map.max.attempts =
> 0, old value = 4)
> 13/07/29 14:36:20 INFO mapred.JobClient: Running job: job_201307232135_0588
> 13/07/29 14:36:21 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/29 14:36:52 INFO mapred.JobClient: map 50% reduce 0%
> 13/07/29 14:47:24 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/29 14:47:39 INFO mapred.JobClient: Job complete: job_201307232135_0588
> 13/07/29 14:47:39 INFO mapred.JobClient: Counters: 6
> 13/07/29 14:47:39 INFO mapred.JobClient: Job Counters
> 13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=670508
> 13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all reduces
> waiting after reserving slots (ms)=0
> 13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/29 14:47:39 INFO mapred.JobClient: Launched map tasks=2
> 13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/29 14:47:39 INFO mapred.JobClient: Failed map tasks=1
>
>
> When I check the job tracker, I see that two map jobs were killed, with the
> following errors:
>
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>
> java.lang.IllegalStateException: run: Caught an unrecoverable exception
> exists: Failed to check
> /_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
> after 3 tries!
> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.IllegalStateException: exists: Failed to check
> /_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
> after 3 tries!
> at org.apache.giraph.zk.ZooKeeperExt.exists(ZooKeeperExt.java:369)
> at org.apache.giraph.worker.BspServiceWorker.s
> Task attempt_201307232135_0588_m_000001_0 failed to report status for 600
> seconds. Killing!
>
> Any idea what the problem is?
> Thanks in advance.
--
Kyle Orlando
Computer Engineering Major
University of Maryland