You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Mark Meissonnier <MM...@looksmart.net> on 2007/06/01 02:16:34 UTC

Pipe/redirection to HDFS?

Hi,
is it possible to pipe/redirect something to HDFS?
Say you have a gz file that you want to put in distributed filesystem in
an uncompressed form,
do you have to make a local file first? What if that file is huge?
Thanks
Mark

Re: Pipe/redirection to HDFS?

Posted by Doug Cutting <cu...@apache.org>.

Mark Meissonnier wrote:
> Sweet. It works. Thanks
> Someone should put it on this wiki page
> http://wiki.apache.org/lucene-hadoop/hadoop-0.1-dev/bin/hadoop_dfs
> I don't have editing priviledges. 

Anyone can create themselves a wiki account and edit pages.  Just use 
the Login button at the top of the page.

Doug

RE: Pipe/redirection to HDFS?

Posted by Mark Meissonnier <MM...@looksmart.net>.

Sweet. It works. Thanks
Someone should put it on this wiki page
http://wiki.apache.org/lucene-hadoop/hadoop-0.1-dev/bin/hadoop_dfs
I don't have editing priviledges. 

-----Original Message-----
From: Hairong Kuang [mailto:hairong@yahoo-inc.com] 
Sent: Thursday, May 31, 2007 6:39 PM
To: hadoop-user@lucene.apache.org
Subject: RE: Pipe/redirection to HDFS?

Please try
Zcat <gzipped file> | bin/hadoop dfs -put - <dst file>

Hairong 

-----Original Message-----
From: Mark Meissonnier [mailto:MMeissonnier@looksmart.net]
Sent: Thursday, May 31, 2007 5:17 PM
To: hadoop-user@lucene.apache.org
Subject: Pipe/redirection to HDFS?

Hi,
is it possible to pipe/redirect something to HDFS?
Say you have a gz file that you want to put in distributed filesystem in
an uncompressed form, do you have to make a local file first? What if
that file is huge?
Thanks
Mark

RE: Pipe/redirection to HDFS?

Posted by Hairong Kuang <ha...@yahoo-inc.com>.

Please try
Zcat <gzipped file> | bin/hadoop dfs -put - <dst file>

Hairong 

-----Original Message-----
From: Mark Meissonnier [mailto:MMeissonnier@looksmart.net] 
Sent: Thursday, May 31, 2007 5:17 PM
To: hadoop-user@lucene.apache.org
Subject: Pipe/redirection to HDFS?

Hi,
is it possible to pipe/redirect something to HDFS?
Say you have a gz file that you want to put in distributed filesystem in an
uncompressed form, do you have to make a local file first? What if that file
is huge?
Thanks
Mark