You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Rutuja Joshi <ru...@flurry.com> on 2008/07/30 23:06:01 UTC

How to move files from one location to another on hadoop

Hi all,

Could anyone suggest any efficient way to move files from one location to
another on Hadoop. Please note that both the locations are on HDFS.
I tried looking for inbuilt file system APIs but couldn't find anything
suitable.

I could use moveToLocalFile and moveFromLocalFile but I do not think that's
the efficient solution here.

Any help is appreciated,
Regards,
Rutuja

Re: How to move files from one location to another on hadoop

Posted by Serge Blazhievsky <ea...@gmail.com>.
You can something like this:

testPath.getFileSystem(conf).rename(new Path(strSrc), new Path(strDst));

On Wed, Jul 30, 2008 at 2:11 PM, Miles Osborne <mi...@inf.ed.ac.uk> wrote:

> bin/hadoop dfs -mv
>
> Miles
>
> 2008/7/30 Rutuja Joshi <ru...@flurry.com>:
>
> > Hi all,
> >
> > Could anyone suggest any efficient way to move files from one location to
> > another on Hadoop. Please note that both the locations are on HDFS.
> > I tried looking for inbuilt file system APIs but couldn't find anything
> > suitable.
> >
> > I could use moveToLocalFile and moveFromLocalFile but I do not think
> that's
> > the efficient solution here.
> >
> > Any help is appreciated,
> > Regards,
> > Rutuja
> >
>
>
>
> --
> The University of Edinburgh is a charitable body, registered in Scotland,
> with registration number SC005336.
>

Re: How to move files from one location to another on hadoop

Posted by Miles Osborne <mi...@inf.ed.ac.uk>.
bin/hadoop dfs -mv

Miles

2008/7/30 Rutuja Joshi <ru...@flurry.com>:

> Hi all,
>
> Could anyone suggest any efficient way to move files from one location to
> another on Hadoop. Please note that both the locations are on HDFS.
> I tried looking for inbuilt file system APIs but couldn't find anything
> suitable.
>
> I could use moveToLocalFile and moveFromLocalFile but I do not think that's
> the efficient solution here.
>
> Any help is appreciated,
> Regards,
> Rutuja
>



-- 
The University of Edinburgh is a charitable body, registered in Scotland,
with registration number SC005336.

Re: How to move files from one location to another on hadoop

Posted by Chris Douglas <ch...@yahoo-inc.com>.
Copying between filesystems, particularly between HDFS filesystems, is  
best done with distcp:

http://hadoop.apache.org/core/docs/r0.18.0/distcp.html

-C

On Sep 12, 2008, at 8:04 AM, James Moore wrote:

> On Wed, Jul 30, 2008 at 2:06 PM, Rutuja Joshi <ru...@flurry.com>  
> wrote:
>> Could anyone suggest any efficient way to move files from one  
>> location to
>> another on Hadoop. Please note that both the locations are on HDFS.
>> I tried looking for inbuilt file system APIs but couldn't find  
>> anything
>> suitable.
>
> The code you want to start with is:
>
> src/core/org/apache/hadoop/fs/FsShell.java
>
> (in 0.18.0, but I think it's been around for a while)
>
> That's where you'll see the implementation of 'hadoop dfs -mv filea
> fileb' - in this case, you're looking for rename().
>
> -- 
> James Moore | james@restphone.com
> Ruby and Ruby on Rails consulting
> blog.restphone.com

Re: How to move files from one location to another on hadoop

Posted by James Moore <ja...@gmail.com>.
On Wed, Jul 30, 2008 at 2:06 PM, Rutuja Joshi <ru...@flurry.com> wrote:
> Could anyone suggest any efficient way to move files from one location to
> another on Hadoop. Please note that both the locations are on HDFS.
> I tried looking for inbuilt file system APIs but couldn't find anything
> suitable.

The code you want to start with is:

src/core/org/apache/hadoop/fs/FsShell.java

(in 0.18.0, but I think it's been around for a while)

That's where you'll see the implementation of 'hadoop dfs -mv filea
fileb' - in this case, you're looking for rename().

-- 
James Moore | james@restphone.com
Ruby and Ruby on Rails consulting
blog.restphone.com