You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by FreeFoodEveryday <yz...@adconion.com> on 2010/10/15 02:09:04 UTC

Can distcp handle the copy from a non-HDFS cluster to a HDFS cluster?

We have a non-HDFS cluster (it's not a S3 cluster either). We would like to
use Hadoop distcp to copy some of the files to a Hadoop cluster. Is it
currently supported?

Thanks.
-- 
View this message in context: http://old.nabble.com/Can-distcp-handle-the-copy-from-a-non-HDFS-cluster-to-a-HDFS-cluster--tp29967710p29967710.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: Can distcp handle the copy from a non-HDFS cluster to a HDFS cluster?

Posted by aniket ray <an...@gmail.com>.

Hi,

Did you get any way to copy from a non-HDFS linux cluster to another HDFS
cluster?

Thanks,
Aniket Ray

On Fri, Oct 15, 2010 at 5:39 AM, FreeFoodEveryday <yz...@adconion.com>wrote:

>
> We have a non-HDFS cluster (it's not a S3 cluster either). We would like to
> use Hadoop distcp to copy some of the files to a Hadoop cluster. Is it
> currently supported?
>
> Thanks.
> --
> View this message in context:
> http://old.nabble.com/Can-distcp-handle-the-copy-from-a-non-HDFS-cluster-to-a-HDFS-cluster--tp29967710p29967710.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>

Re: Can distcp handle the copy from a non-HDFS cluster to a HDFS cluster?

Posted by Harsh J <qw...@gmail.com>.

If Hadoop can work with it, then sure. For example, the following
works since the uris are acceptable by Hadoop:

hadoop distcp hdfs://master:9000/user/hadoop/somefile.txt
file:///home/hadoop/abc.def

However, doing the above won't ensure it comes back to your machine,
instead abc.def would be created on the node the map task ran on,
cause that's how file:// is resolved. Just a heads up if you're gonna
use that on a non NFS thingy.

On Fri, Oct 15, 2010 at 5:39 AM, FreeFoodEveryday <yz...@adconion.com> wrote:
>
> We have a non-HDFS cluster (it's not a S3 cluster either). We would like to
> use Hadoop distcp to copy some of the files to a Hadoop cluster. Is it
> currently supported?
>
> Thanks.
> --
> View this message in context: http://old.nabble.com/Can-distcp-handle-the-copy-from-a-non-HDFS-cluster-to-a-HDFS-cluster--tp29967710p29967710.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>

-- 
Harsh J
www.harshj.com