You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Esteban Molina-Estolano <ee...@soe.ucsc.edu> on 2007/06/01 10:14:50 UTC
Adding new filesystem to Hadoop causing too many Map tasks
I'm adding support in Hadoop for Ceph (http://ceph.sourceforge.net/),
a distributed filesystem developed at UC Santa Cruz (http://
ssrc.cse.ucsc.edu/). Ceph runs entirely in userspace and is written
in C++. My current implementation is a subclass of FileSystem that
uses a bit of JNI glue to invoke the C++ Ceph client code.
I'm having trouble with a small test: RandomWriter, 4 TaskTracker
nodes, 5 maps per node, 10 MB per map, for a total of 200 MB over 20
Map tasks. I tried it on Hadoop with DFS, and it took about 30
seconds. Then, I ran the same test using Ceph. I changed
fs.default.name to "ceph:///"; added fs.ceph.impl as
org.apache.hadoop.fs.ceph.CephFileSystem; and left all other
configuration settings untouched. It ran horrifically slowly.
I ran the JobTracker and each TaskTracker in a separate terminal to
watch the output. One of the TaskTracker nodes gave me this:
07/06/01 00:16:49 INFO mapred.TaskRunner: task_0001_r_000000_0 Need
400 map output(s)
07/06/01 00:16:49 INFO mapred.TaskRunner: task_0001_r_000000_0 Need
400 map output location(s)
Then the JobTracker spawned 400 Map tasks:
07/06/01 00:23:11 INFO mapred.JobTracker: Adding task
'task_0001_m_000397_0' to tip tip_0001_m_000397, for tracker
'tracker_issdm-11.cse.ucsc.edu:50050'
07/06/01 00:23:12 INFO mapred.JobInProgress: Task
'task_0001_m_000396_0' has completed tip_0001_m_000396 successfully.
07/06/01 00:23:12 INFO mapred.TaskInProgress: Task
'task_0001_m_000396_0' has completed.
07/06/01 00:23:12 INFO mapred.JobInProgress: Choosing normal task
tip_0001_m_000398
07/06/01 00:23:12 INFO mapred.JobTracker: Adding task
'task_0001_m_000398_0' to tip tip_0001_m_000398, for tracker
'tracker_issdm-8.cse.ucsc.edu:50050'
07/06/01 00:23:13 INFO mapred.JobInProgress: Task
'task_0001_m_000397_0' has completed tip_0001_m_000397 successfully.
07/06/01 00:23:13 INFO mapred.TaskInProgress: Task
'task_0001_m_000397_0' has completed.
07/06/01 00:23:13 INFO mapred.JobInProgress: Choosing normal task
tip_0001_m_000399
07/06/01 00:23:13 INFO mapred.JobTracker: Adding task
'task_0001_m_000399_0' to tip tip_0001_m_000399, for tracker
'tracker_issdm-11.cse.ucsc.edu:50050'
I'm ending up with way too many Map tasks, and as a result the job
takes way too long to run.
I strongly suspect this is a problem with my implementation, but I'm
not sure where to start looking. What sort of problem on the
FileSystem side could cause MapReduce to spawn so many extra tasks?
How can I pin down the cause?
Thanks,
~ Esteban
Re: Adding new filesystem to Hadoop causing too many Map tasks
Posted by Doug Cutting <cu...@apache.org>.
Arun C Murthy wrote:
> I'd suggest you checkout the 0.12 branch from here:
> http://svn.apache.org/repos/asf/lucene/hadoop/branches/branch-0.12/
Or, for a specific release, use its tag:
http://svn.apache.org/repos/asf/lucene/hadoop/tags/release-0.12.3/
Doug
Re: Adding new filesystem to Hadoop causing too many Map tasks
Posted by Arun C Murthy <ar...@yahoo-inc.com>.
Esteban Molina-Estolano wrote:
> Thanks for the advice. I am using an old version.
> I'm trying to upgrade to 0.12.3, but when I try to compile (even
> without adding in my own code) I get:
>
> [eestolan@issdm-1 ~/hadoop-0.12.3]$ ant
> Buildfile: build.xml
>
> init:
>
> BUILD FAILED
> /cse/grads/eestolan/hadoop-0.12.3/build.xml:114: Specify at least one
> source--a file or resource collection.
>
I'm guessing the non-existence of the *.template files (e.g.
hadoop-site.xml.template) in the *conf* directory of the releases is
causing this... I'm not sure you can just d/w the release and compile it...
I'd suggest you checkout the 0.12 branch from here:
http://svn.apache.org/repos/asf/lucene/hadoop/branches/branch-0.12/
hth,
Arun
> Total time: 0 seconds
>
> That line in build.xml has the following:
>
> <touch datetime="01/25/1971 2:00 pm">
> <fileset dir="${conf.dir}" includes="**/*.template"/>
> <fileset dir="${contrib.dir}" includes="**/*.template"/>
> </touch>
>
> What might be causing the error?
>
> Thanks,
> ~ Esteban
>
>
> On Jun 1, 2007, at 9:26 AM, Owen O'Malley wrote:
>
>>
>> On Jun 1, 2007, at 1:14 AM, Esteban Molina-Estolano wrote:
>>
>>> I'm having trouble with a small test: RandomWriter, 4 TaskTracker
>>> nodes, 5 maps per node, 10 MB per map, for a total of 200 MB over 20
>>> Map tasks. I tried it on Hadoop with DFS, and it took about 30
>>> seconds. Then, I ran the same test using Ceph. I changed
>>> fs.default.name to "ceph:///"; added fs.ceph.impl as
>>> org.apache.hadoop.fs.ceph.CephFileSystem; and left all other
>>> configuration settings untouched. It ran horrifically slowly.
>>>
>>> Then the JobTracker spawned 400 Map tasks:
>>>
>>> I'm ending up with way too many Map tasks, and as a result the job
>>> takes way too long to run.
>>
>>
>> That is really strange, especially because RandomWriter isn't looking
>> at any real inputs. (Unless you are using version 0.11 or earlier of
>> Hadoop...) Are you using an old version of Hadoop? If so, I'd
>> suspect it has something to do with the blocksize for the input files
>> being too small (likely 1 byte or so). You need to return much bigger
>> numbers for FileSystem.getBlockSize(Path) or map/ reduce will default
>> to making very small input splits.
>>
>> -- Owen
>
>
Re: Adding new filesystem to Hadoop causing too many Map tasks
Posted by Konstantin Shvachko <sh...@yahoo-inc.com>.
Could you pls check your ant version.
Should be at least 1.6.3 I believe.
--Konst
Esteban Molina-Estolano wrote:
> Thanks for the advice. I am using an old version.
> I'm trying to upgrade to 0.12.3, but when I try to compile (even
> without adding in my own code) I get:
>
> [eestolan@issdm-1 ~/hadoop-0.12.3]$ ant
> Buildfile: build.xml
>
> init:
>
> BUILD FAILED
> /cse/grads/eestolan/hadoop-0.12.3/build.xml:114: Specify at least one
> source--a file or resource collection.
>
> Total time: 0 seconds
>
> That line in build.xml has the following:
>
> <touch datetime="01/25/1971 2:00 pm">
> <fileset dir="${conf.dir}" includes="**/*.template"/>
> <fileset dir="${contrib.dir}" includes="**/*.template"/>
> </touch>
>
> What might be causing the error?
>
> Thanks,
> ~ Esteban
>
>
> On Jun 1, 2007, at 9:26 AM, Owen O'Malley wrote:
>
>>
>> On Jun 1, 2007, at 1:14 AM, Esteban Molina-Estolano wrote:
>>
>>> I'm having trouble with a small test: RandomWriter, 4 TaskTracker
>>> nodes, 5 maps per node, 10 MB per map, for a total of 200 MB over
>>> 20 Map tasks. I tried it on Hadoop with DFS, and it took about 30
>>> seconds. Then, I ran the same test using Ceph. I changed
>>> fs.default.name to "ceph:///"; added fs.ceph.impl as
>>> org.apache.hadoop.fs.ceph.CephFileSystem; and left all other
>>> configuration settings untouched. It ran horrifically slowly.
>>>
>>> Then the JobTracker spawned 400 Map tasks:
>>>
>>> I'm ending up with way too many Map tasks, and as a result the job
>>> takes way too long to run.
>>
>>
>> That is really strange, especially because RandomWriter isn't
>> looking at any real inputs. (Unless you are using version 0.11 or
>> earlier of Hadoop...) Are you using an old version of Hadoop? If
>> so, I'd suspect it has something to do with the blocksize for the
>> input files being too small (likely 1 byte or so). You need to
>> return much bigger numbers for FileSystem.getBlockSize(Path) or map/
>> reduce will default to making very small input splits.
>>
>> -- Owen
>
>
>
Re: Adding new filesystem to Hadoop causing too many Map tasks
Posted by Esteban Molina-Estolano <ee...@soe.ucsc.edu>.
Thanks for the advice. I am using an old version.
I'm trying to upgrade to 0.12.3, but when I try to compile (even
without adding in my own code) I get:
[eestolan@issdm-1 ~/hadoop-0.12.3]$ ant
Buildfile: build.xml
init:
BUILD FAILED
/cse/grads/eestolan/hadoop-0.12.3/build.xml:114: Specify at least one
source--a file or resource collection.
Total time: 0 seconds
That line in build.xml has the following:
<touch datetime="01/25/1971 2:00 pm">
<fileset dir="${conf.dir}" includes="**/*.template"/>
<fileset dir="${contrib.dir}" includes="**/*.template"/>
</touch>
What might be causing the error?
Thanks,
~ Esteban
On Jun 1, 2007, at 9:26 AM, Owen O'Malley wrote:
>
> On Jun 1, 2007, at 1:14 AM, Esteban Molina-Estolano wrote:
>
>> I'm having trouble with a small test: RandomWriter, 4 TaskTracker
>> nodes, 5 maps per node, 10 MB per map, for a total of 200 MB over
>> 20 Map tasks. I tried it on Hadoop with DFS, and it took about 30
>> seconds. Then, I ran the same test using Ceph. I changed
>> fs.default.name to "ceph:///"; added fs.ceph.impl as
>> org.apache.hadoop.fs.ceph.CephFileSystem; and left all other
>> configuration settings untouched. It ran horrifically slowly.
>>
>> Then the JobTracker spawned 400 Map tasks:
>>
>> I'm ending up with way too many Map tasks, and as a result the job
>> takes way too long to run.
>
> That is really strange, especially because RandomWriter isn't
> looking at any real inputs. (Unless you are using version 0.11 or
> earlier of Hadoop...) Are you using an old version of Hadoop? If
> so, I'd suspect it has something to do with the blocksize for the
> input files being too small (likely 1 byte or so). You need to
> return much bigger numbers for FileSystem.getBlockSize(Path) or map/
> reduce will default to making very small input splits.
>
> -- Owen
Re: Adding new filesystem to Hadoop causing too many Map tasks
Posted by Owen O'Malley <oo...@yahoo-inc.com>.
On Jun 1, 2007, at 1:14 AM, Esteban Molina-Estolano wrote:
> I'm having trouble with a small test: RandomWriter, 4 TaskTracker
> nodes, 5 maps per node, 10 MB per map, for a total of 200 MB over
> 20 Map tasks. I tried it on Hadoop with DFS, and it took about 30
> seconds. Then, I ran the same test using Ceph. I changed
> fs.default.name to "ceph:///"; added fs.ceph.impl as
> org.apache.hadoop.fs.ceph.CephFileSystem; and left all other
> configuration settings untouched. It ran horrifically slowly.
>
> Then the JobTracker spawned 400 Map tasks:
>
> I'm ending up with way too many Map tasks, and as a result the job
> takes way too long to run.
That is really strange, especially because RandomWriter isn't looking
at any real inputs. (Unless you are using version 0.11 or earlier of
Hadoop...) Are you using an old version of Hadoop? If so, I'd
suspect it has something to do with the blocksize for the input files
being too small (likely 1 byte or so). You need to return much bigger
numbers for FileSystem.getBlockSize(Path) or map/reduce will default
to making very small input splits.
-- Owen
RE: Adding new filesystem to Hadoop causing too many Map tasks
Posted by Devaraj Das <dd...@yahoo-inc.com>.
Moving this to hadoop-user.
Just to clarify, did you set test.randomwrite.maps_per_host to 5 in the run
with ceph?
-----Original Message-----
From: Esteban Molina-Estolano [mailto:eestolan@soe.ucsc.edu]
Sent: Friday, June 01, 2007 1:45 PM
To: hadoop-dev@lucene.apache.org
Subject: Adding new filesystem to Hadoop causing too many Map tasks
I'm adding support in Hadoop for Ceph (http://ceph.sourceforge.net/), a
distributed filesystem developed at UC Santa Cruz (http://
ssrc.cse.ucsc.edu/). Ceph runs entirely in userspace and is written in C++.
My current implementation is a subclass of FileSystem that uses a bit of JNI
glue to invoke the C++ Ceph client code.
I'm having trouble with a small test: RandomWriter, 4 TaskTracker nodes, 5
maps per node, 10 MB per map, for a total of 200 MB over 20 Map tasks. I
tried it on Hadoop with DFS, and it took about 30 seconds. Then, I ran the
same test using Ceph. I changed fs.default.name to "ceph:///"; added
fs.ceph.impl as org.apache.hadoop.fs.ceph.CephFileSystem; and left all other
configuration settings untouched. It ran horrifically slowly.
I ran the JobTracker and each TaskTracker in a separate terminal to watch
the output. One of the TaskTracker nodes gave me this:
07/06/01 00:16:49 INFO mapred.TaskRunner: task_0001_r_000000_0 Need 400 map
output(s)
07/06/01 00:16:49 INFO mapred.TaskRunner: task_0001_r_000000_0 Need 400 map
output location(s)
Then the JobTracker spawned 400 Map tasks:
07/06/01 00:23:11 INFO mapred.JobTracker: Adding task 'task_0001_m_000397_0'
to tip tip_0001_m_000397, for tracker 'tracker_issdm-11.cse.ucsc.edu:50050'
07/06/01 00:23:12 INFO mapred.JobInProgress: Task 'task_0001_m_000396_0' has
completed tip_0001_m_000396 successfully.
07/06/01 00:23:12 INFO mapred.TaskInProgress: Task 'task_0001_m_000396_0'
has completed.
07/06/01 00:23:12 INFO mapred.JobInProgress: Choosing normal task
tip_0001_m_000398
07/06/01 00:23:12 INFO mapred.JobTracker: Adding task 'task_0001_m_000398_0'
to tip tip_0001_m_000398, for tracker 'tracker_issdm-8.cse.ucsc.edu:50050'
07/06/01 00:23:13 INFO mapred.JobInProgress: Task 'task_0001_m_000397_0' has
completed tip_0001_m_000397 successfully.
07/06/01 00:23:13 INFO mapred.TaskInProgress: Task 'task_0001_m_000397_0'
has completed.
07/06/01 00:23:13 INFO mapred.JobInProgress: Choosing normal task
tip_0001_m_000399
07/06/01 00:23:13 INFO mapred.JobTracker: Adding task 'task_0001_m_000399_0'
to tip tip_0001_m_000399, for tracker 'tracker_issdm-11.cse.ucsc.edu:50050'
I'm ending up with way too many Map tasks, and as a result the job takes way
too long to run.
I strongly suspect this is a problem with my implementation, but I'm not
sure where to start looking. What sort of problem on the FileSystem side
could cause MapReduce to spawn so many extra tasks?
How can I pin down the cause?
Thanks,
~ Esteban