You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ashish Thusoo <at...@facebook.com> on 2008/02/10 00:52:18 UTC
Speculative execution and output directory
Hi Hadoop users,
We have intermittently hit issues with speculative execution and hadoop
streaming where we see a directory of the form
_task_200...._...._m_..._.
formed in the output directory. Has anyone out there hit similar issues
or knows what might be happening here? We did scan the JIRA but could
not find any related issues reported there.
Thanks,
Ashish
RE: Speculative execution and output directory
Posted by Ashish Thusoo <at...@facebook.com>.
Thanks a bunch Arun. I am ignoring these files for now as you suggest.
Thanks for pointing to the JIRA.
Ashish
-----Original Message-----
From: Arun C Murthy [mailto:acm@yahoo-inc.com]
Sent: Saturday, February 09, 2008 10:00 PM
To: core-user@hadoop.apache.org
Subject: Re: Speculative execution and output directory
On Feb 9, 2008, at 3:52 PM, Ashish Thusoo wrote:
> Hi Hadoop users,
>
>
>
> We have intermittently hit issues with speculative execution and
> hadoop
> streaming where we see a directory of the form
>
> _task_200...._...._m_..._.
>
It's an unfortunate side-effect of the current implementation of
speculative execution, we have been discussing ways to fix it here:
http://issues.apache.org/jira/browse/HADOOP-2391
In the meanwhile, it is completely safe to ignore directories
starting with with underscore ("_") and use the rest...
Arun
>
>
> formed in the output directory. Has anyone out there hit similar
> issues
> or knows what might be happening here? We did scan the JIRA but could
> not find any related issues reported there.
>
>
>
> Thanks,
>
> Ashish
>
>
>
Re: Speculative execution and output directory
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On Feb 9, 2008, at 3:52 PM, Ashish Thusoo wrote:
> Hi Hadoop users,
>
>
>
> We have intermittently hit issues with speculative execution and
> hadoop
> streaming where we see a directory of the form
>
> _task_200...._...._m_..._.
>
It's an unfortunate side-effect of the current implementation of
speculative execution, we have been discussing ways to fix it here:
http://issues.apache.org/jira/browse/HADOOP-2391
In the meanwhile, it is completely safe to ignore directories
starting with with underscore ("_") and use the rest...
Arun
>
>
> formed in the output directory. Has anyone out there hit similar
> issues
> or knows what might be happening here? We did scan the JIRA but could
> not find any related issues reported there.
>
>
>
> Thanks,
>
> Ashish
>
>
>