You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Travis Crawford <tr...@gmail.com> on 2011/03/31 20:45:28 UTC
distributed cache exceeding local.cache.size
Hi mapreduce gurus -
Today while looking into a few TaskTracker's with full disks I came
across the following directory using 207GB:
/data/disk3/mapred/local/taskTracker/archive/namenode.foo.com/tmp/temp-2024081/tmp942425908/tmp942425908
Digging a bit further a job did indeed reference a 200+GB directory in
HDFS via the distributed cache, and it appeared to copy the whole
thing locally to the point where the disk filled up. Looking at the
cluster config we don't explicitly set local.cache.size, so the 10GB
default limit should take effect.
Is anyone familiar with how the distributed cache deals when datasets
larger than the total cache size are referenced? I've disabled the job
that caused this situation but am wondering if I can configure things
more defensively.
Thanks!
Travis
Re: distributed cache exceeding local.cache.size
Posted by Allen Wittenauer <aw...@apache.org>.
On Apr 1, 2011, at 12:05 PM, Travis Crawford wrote:
> On Thu, Mar 31, 2011 at 3:25 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>
>> On Mar 31, 2011, at 11:45 AM, Travis Crawford wrote:
>>
>>> Is anyone familiar with how the distributed cache deals when datasets
>>> larger than the total cache size are referenced? I've disabled the job
>>> that caused this situation but am wondering if I can configure things
>>> more defensively.
>>
>> I've started building specific file systems on drives to store the map reduce spill space. It seems to be the only reliable way to prevent MR from going nuts. Sure, some jobs may fail, but that seems to be a better strategy than the alternative.
>>
>
> Interesting. So for example, say you have 2 disks in a
> DataNode+TaskTracker machine. You'd make two partitions on each disk,
> and expose 4 partitions to the system, then give two partitions (one
> from each disk) to each app?
Yes, exactly.
> Is the idea here to prevent runaway jobs from causing DataNode disks
> from filling up, which causes write failures?
You got it. :) Plus some side benefits:
- HDFS now has a fixed quantity, making capacity planning less of a guessing game.
- Any per-fs kernel buffer caches, structures, etc are now doubled and specific to each use case. This opens the door for lots of interesting things...
The big con is that you'll need to spend some time to figure out what size you really need for your MR space. We had the benefit of ZFS allowing us to adjust the value. On non-pooled storage systems, this is obviously harder. FWIW, experience has shown that you'll likely end up somewhere between 100-200G of space per fs, but YMMV.
I've thought about trying fs quotas, but hard partitions just seems like it would be better in the long run.
Re: distributed cache exceeding local.cache.size
Posted by Travis Crawford <tr...@gmail.com>.
On Thu, Mar 31, 2011 at 3:25 PM, Allen Wittenauer <aw...@apache.org> wrote:
>
> On Mar 31, 2011, at 11:45 AM, Travis Crawford wrote:
>
>> Is anyone familiar with how the distributed cache deals when datasets
>> larger than the total cache size are referenced? I've disabled the job
>> that caused this situation but am wondering if I can configure things
>> more defensively.
>
> I've started building specific file systems on drives to store the map reduce spill space. It seems to be the only reliable way to prevent MR from going nuts. Sure, some jobs may fail, but that seems to be a better strategy than the alternative.
>
Interesting. So for example, say you have 2 disks in a
DataNode+TaskTracker machine. You'd make two partitions on each disk,
and expose 4 partitions to the system, then give two partitions (one
from each disk) to each app?
Is the idea here to prevent runaway jobs from causing DataNode disks
from filling up, which causes write failures?
--travis
Re: distributed cache exceeding local.cache.size
Posted by Allen Wittenauer <aw...@apache.org>.
On Mar 31, 2011, at 11:45 AM, Travis Crawford wrote:
> Is anyone familiar with how the distributed cache deals when datasets
> larger than the total cache size are referenced? I've disabled the job
> that caused this situation but am wondering if I can configure things
> more defensively.
I've started building specific file systems on drives to store the map reduce spill space. It seems to be the only reliable way to prevent MR from going nuts. Sure, some jobs may fail, but that seems to be a better strategy than the alternative.
Re: post
Posted by Harsh J <qw...@gmail.com>.
On Fri, Apr 1, 2011 at 2:07 PM, Abdin Tarkmani <a....@chelab.it> wrote:
> Hi hadoop/mapreduce gurus
>
> Is it ok to post a hadoop/mr job here using this forum?
>
If by post you mean posting code of a job or logs of a problem you're
trying to solve with Hadoop MapReduce, then this list is the right
one. Do post your logs/code on public services like
http://gists.github.com, http://paste.pocoo.org, etc. and send us a
link to that with a description of your exact problem or a question.
--
Harsh J
http://harshj.com
Re: post
Posted by Abdin Tarkmani <a....@chelab.it>.
Hi hadoop/mapreduce gurus
Is it ok to post a hadoop/mr job here using this forum?
thanks.
dean