You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-user@hadoop.apache.org by Mapred Learn <ma...@gmail.com> on 2011/05/18 02:44:27 UTC

Are hadoop fs commands serial or parallel

 Hi,
My question is when I run a command from hdfs client, for eg. hadoop fs
-copyFromLocal or create a sequence file writer in java code and append
key/values to it through Hadoop APIs, does it internally transfer/write data
to HDFS serially or in parallel ?

Thanks in advance,
-JJ

Re: Are hadoop fs commands serial or parallel

Posted by Mapred Learn <ma...@gmail.com>.

Thanks Joey !
I will try to find out abt copyFromLocal. Looks like Hadoop Apis write serially as you pointed out.

Thanks,
-JJ

On May 17, 2011, at 8:32 PM, Joey Echeverria <jo...@cloudera.com> wrote:

> The sequence file writer definitely does it serially as you can only
> ever write to the end of a file in Hadoop.
> 
> Doing copyFromLocal could write multiple files in parallel (I'm not
> sure if it does or not), but a single file would be written serially.
> 
> -Joey
> 
> On Tue, May 17, 2011 at 5:44 PM, Mapred Learn <ma...@gmail.com> wrote:
>>  Hi,
>> My question is when I run a command from hdfs client, for eg. hadoop fs
>> -copyFromLocal or create a sequence file writer in java code and append
>> key/values to it through Hadoop APIs, does it internally transfer/write data
>> to HDFS serially or in parallel ?
>> 
>> Thanks in advance,
>> -JJ
>> 
> 
> 
> 
> -- 
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434

Re: Are hadoop fs commands serial or parallel

Posted by Mapred Learn <ma...@gmail.com>.

Thanks Joey !
I will try to find out abt copyFromLocal. Looks like Hadoop Apis write serially as you pointed out.

Thanks,
-JJ

On May 17, 2011, at 8:32 PM, Joey Echeverria <jo...@cloudera.com> wrote:

> The sequence file writer definitely does it serially as you can only
> ever write to the end of a file in Hadoop.
> 
> Doing copyFromLocal could write multiple files in parallel (I'm not
> sure if it does or not), but a single file would be written serially.
> 
> -Joey
> 
> On Tue, May 17, 2011 at 5:44 PM, Mapred Learn <ma...@gmail.com> wrote:
>>  Hi,
>> My question is when I run a command from hdfs client, for eg. hadoop fs
>> -copyFromLocal or create a sequence file writer in java code and append
>> key/values to it through Hadoop APIs, does it internally transfer/write data
>> to HDFS serially or in parallel ?
>> 
>> Thanks in advance,
>> -JJ
>> 
> 
> 
> 
> -- 
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434

Re: Are hadoop fs commands serial or parallel

Posted by Joey Echeverria <jo...@cloudera.com>.

The sequence file writer definitely does it serially as you can only
ever write to the end of a file in Hadoop.

Doing copyFromLocal could write multiple files in parallel (I'm not
sure if it does or not), but a single file would be written serially.

-Joey

On Tue, May 17, 2011 at 5:44 PM, Mapred Learn <ma...@gmail.com> wrote:
>  Hi,
> My question is when I run a command from hdfs client, for eg. hadoop fs
> -copyFromLocal or create a sequence file writer in java code and append
> key/values to it through Hadoop APIs, does it internally transfer/write data
> to HDFS serially or in parallel ?
>
> Thanks in advance,
> -JJ
>

-- 
Joseph Echeverria
Cloudera, Inc.
443.305.9434