You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Steve Gao <st...@yahoo.com> on 2008/07/19 01:53:33 UTC
[Streaming]What is the difference between streaming options: -file and -CacheFile ?
Hi All,
I am using Hadoop Streaming. I am confused by streaming options: -file and -CacheFile. Seems that they mean the same thing, right?
Another misleading options are : -NumReduceTasks and -jobconf mapred.reduce.tasks. Both are used to control (or give hit to) the number of reducers.
Thanks
Re: [Streaming]What is the difference between streaming options: -file and -CacheFile ?
Posted by Steve Gao <st...@yahoo.com>.
One more little question, why Hadoop streaming is designed in this way to use 2 different options to do the same thing (i.e. control the reduce number)? What's the point here?
Thanks
--- On Fri, 7/18/08, Arun C Murthy <ac...@yahoo-inc.com> wrote:
From: Arun C Murthy <ac...@yahoo-inc.com>
Subject: Re: [Streaming]What is the difference between streaming options: -file and -CacheFile ?
To: core-user@hadoop.apache.org, "Steve Gao" <st...@yahoo.com>
Date: Friday, July 18, 2008, 8:27 PM
On Jul 18, 2008, at 4:53 PM, Steve Gao wrote:
> Hi All,
> I am using Hadoop Streaming. I am confused by streaming
> options: -file and -CacheFile. Seems that they mean the same thing,
> right?
>
The difference is that -file will 'ship' your file (local file) to
the cluster, while -cachefile assumes that it is already present on
HDFS at the given path.
> Another misleading options are : -NumReduceTasks and -jobconf
> mapred.reduce.tasks. Both are used to control (or give hit to) the
> number of reducers.
>
Yes, they are both equivalent.
hth,
Arun
Re: [Streaming]What is the difference between streaming options: -file and -CacheFile ?
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On Jul 18, 2008, at 4:53 PM, Steve Gao wrote:
> Hi All,
> I am using Hadoop Streaming. I am confused by streaming
> options: -file and -CacheFile. Seems that they mean the same thing,
> right?
>
The difference is that -file will 'ship' your file (local file) to
the cluster, while -cachefile assumes that it is already present on
HDFS at the given path.
> Another misleading options are : -NumReduceTasks and -jobconf
> mapred.reduce.tasks. Both are used to control (or give hit to) the
> number of reducers.
>
Yes, they are both equivalent.
hth,
Arun