Upload files directly to hdfs from standard out using pipes

[Mapred Learn <ma...@gmail.com>]

Hi,
I found out that :
https://github.com/cloudera/hue/blob/master/desktop/libs/hadoop/src/hadoop/fs/hadoopfs.py

can be used to write data directly to HDFS without writing to a local
filesystem but I am not able to understand how.

Could somebody tell me how can I do this ? Or is there some other best way to do it ?

Basiclly my use case is to pipe standard out from a program directly
to HDFS so that it does not have to go through the disk write.


Thanks in advance !

Re: Upload files directly to hdfs from standard out using pipes

[Koji Noguchi <kn...@yahoo-inc.com>]

1. Run map-only (no reducer) job.  Output goes directly to hdfs.
or
2. echo abc | hadoop dfs ­put - /user/knoguchi/somefile
   would write to hdfs reading from stdin.

Koji

On 1/12/11 2:07 AM, "Mapred Learn" <ma...@gmail.com> wrote:

> Hi,
> I found out that :
> https://github.com/cloudera/hue/blob/master/desktop/libs/hadoop/src/hadoop/fs/
> hadoopfs.py 
> <https://github.com/cloudera/hue/blob/master/desktop/libs/hadoop/src/hadoop/fs
> /hadoopfs.py> 
> 
> can be used to write data directly to HDFS without writing to a local
> filesystem but I am not able to understand how.
> 
> Could somebody tell me how can I do this ? Or is there some other best way to
> do it ?
> 
> Basiclly my use case is to pipe standard out from a program directly
> to HDFS so that it does not have to go through the disk write.
> 
> 
> Thanks in advance !
> 
>