Best way to get data out of HDFS to other platforms

[Mapred Learn <ma...@gmail.com>]

Hi All,

I was trying to find out the best options to get data out of HDFS to other platforms like AIX with parameters like reliability, performance and parallelism and data integrity in mind.


Thanks,
Sent from my iPhone

Re: Best way to get data out of HDFS to other platforms

[Mapred Learn <ma...@gmail.com>]

This is for copying to Linux locally
I m talking about cross platform where you want to get data to systems like AIX where these APIs don't work.

Sent from my iPhone

On Apr 13, 2012, at 1:02 PM, Dave Shine <Da...@channelintelligence.com> wrote:

> fs.copyToLocal(sourcePath, destinationPath);
> 
>  
> 
> Am I missing something???
> 
>  
> 
> Dave
> 
>  
> 
>  
> 
> From: Mac Noland [mailto:mcdonaldnoland@yahoo.com] 
> Sent: Friday, April 13, 2012 2:59 PM
> To: hdfs-user@hadoop.apache.org; CDH Users
> Subject: Re: Best way to get data out of HDFS to other platforms
> 
>  
> 
> There are probably much, much, much better and more standard ways, but in the simple push and pulls I've done, I've just rolled my own.
> 
>  
> 
>     public void copyFileFromHDFS(String src, String dest, boolean overWrite) throws IOException{
> 
>         Path hdfsPath = new Path(src);
>         FSDataInputStream hdfsFileInPut = fs.open(hdfsPath);
>         
>         byte[] data = new byte[1024];
>         File localFile = new File(dest);
>         
>         if (overWrite && localFile.exists()){
>             localFile.delete();
>         }
>         
>         FileOutputStream localFileOutPut = new FileOutputStream(localFile);
>         int off;
>         
>         while ((off = hdfsFileInPut.read(data)) > -1){
>             localFileOutPut.write(data,0,off);
>         }
>         
>         hdfsFileInPut.close();
>         localFileOutPut.close();
> 
>     }
> 
>  
> 
> From: Mapred Learn <ma...@gmail.com>
> To: hdfs-user@hadoop.apache.org; CDH Users <cd...@cloudera.org> 
> Sent: Friday, April 13, 2012 10:29 AM
> Subject: Best way to get data out of HDFS to other platforms
> 
> 
> 
> Hi All,
> 
> I was trying to find out the best options to get data out of HDFS to other platforms like AIX with parameters like reliability, performance and parallelism and data integrity in mind.
> 
> 
> Thanks,
> Sent from my iPhone
> 
> 
> The information contained in this email message is considered confidential and proprietary to the sender and is intended solely for review and use by the named recipient. Any unauthorized review, use or distribution is strictly prohibited. If you have received this message in error, please advise the sender by reply email and delete the message.

Re: Best way to get data out of HDFS to other platforms

[Mac Noland <mc...@yahoo.com>]

There are probably much, much, much better and more standard ways, but in the simple push and pulls I've done, I've just rolled my own.

    public void copyFileFromHDFS(String src, String dest, boolean overWrite) throws IOException{

        Path hdfsPath = new Path(src);
        FSDataInputStream hdfsFileInPut = fs.open(hdfsPath);
        
        byte[] data = new byte[1024];
        File localFile = new File(dest);
        
        if (overWrite && localFile.exists()){
            localFile.delete();
        }
        
        FileOutputStream localFileOutPut = new FileOutputStream(localFile);
        int off;
        
        while ((off = hdfsFileInPut.read(data)) > -1){
            localFileOutPut.write(data,0,off);
        }
        
        hdfsFileInPut.close();
        localFileOutPut.close();

    }



________________________________
 From: Mapred Learn <ma...@gmail.com>
To: hdfs-user@hadoop.apache.org; CDH Users <cd...@cloudera.org> 
Sent: Friday, April 13, 2012 10:29 AM
Subject: Best way to get data out of HDFS to other platforms 
 

Hi All,

I was trying to find out the best options to get data out of HDFS to other platforms like AIX with parameters like reliability, performance and parallelism and data integrity in mind.


Thanks,
Sent from my iPhone