You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Panshul Whisper <ou...@gmail.com> on 2013/01/25 02:09:15 UTC
Copy files from remote folder to HDFS
Hello,
I am trying to copy files, Json files from a remote folder - (a folder on
my local system, Cloudfiles folder or a folder on S3 server) to the HDFS of
a cluster running at a remote location.
The job submitting Application is based on Spring Hadoop.
Can someone please suggest or point me in the right direction for best
option to achieve the above task:
1. Use Spring Integration data pipelines to poll the folders for files and
copy them to the HDFS as they arrive in the source folder. - I have tried
to implement the solution in Spring Data book, but it does not run - no
idea what is wrong as it does not generate logs.
2. Use some other script method to transfer files.
Main requirement, I need to transfer files from a remote folder to HDFS
everyday at a fixed time for processing in the hadoop cluster. These files
are collecting from various sources in the remote folders.
Please suggest an efficient approach. I have been searching and finding a
lot of approaches but unable to decide what will work best. As this
transfer needs to be as fast as possible.
The files to be transferred will be almost 10 GB of Json files not more
than 6kb each file.
Thanking You,
--
Regards,
Ouch Whisper
010101010101
Re: Copy files from remote folder to HDFS
Posted by Nitin Pawar <ni...@gmail.com>.
if this is one time activity
then just download hadoop binaries from apache
replace the hdfs-site.xml and core-site.xml with the one you have on
hadoop cluster
allow this machine to connect with hadoop cluster
then you can just do it with hadoop command line scripts.
On Fri, Jan 25, 2013 at 1:01 PM, Mahesh Balija
<ba...@gmail.com>wrote:
> Hi Panshul,
>
> I am also working on similar requirement, one approach is,
> mount your remote folder on your hadoop master node.
> And simply write a shell script to copy the files to HDFS
> using crontab.
>
> I believe Flume is literally a wrong choice as Flume is a
> data collection and aggregation framework and NOT a file transfer tool and
> may NOT be a good choice when you actually want to copy the files as-is
> onto your cluster (NOT 100% sure as I am also working on that).
>
> Thanks,
> Mahesh Balija,
> CalsoftLabs.
>
> On Fri, Jan 25, 2013 at 6:39 AM, Panshul Whisper <ou...@gmail.com>wrote:
>
>> Hello,
>>
>> I am trying to copy files, Json files from a remote folder - (a folder on
>> my local system, Cloudfiles folder or a folder on S3 server) to the HDFS of
>> a cluster running at a remote location.
>> The job submitting Application is based on Spring Hadoop.
>>
>> Can someone please suggest or point me in the right direction for best
>> option to achieve the above task:
>> 1. Use Spring Integration data pipelines to poll the folders for files
>> and copy them to the HDFS as they arrive in the source folder. - I have
>> tried to implement the solution in Spring Data book, but it does not run -
>> no idea what is wrong as it does not generate logs.
>>
>> 2. Use some other script method to transfer files.
>>
>> Main requirement, I need to transfer files from a remote folder to HDFS
>> everyday at a fixed time for processing in the hadoop cluster. These files
>> are collecting from various sources in the remote folders.
>>
>> Please suggest an efficient approach. I have been searching and finding a
>> lot of approaches but unable to decide what will work best. As this
>> transfer needs to be as fast as possible.
>> The files to be transferred will be almost 10 GB of Json files not more
>> than 6kb each file.
>>
>> Thanking You,
>>
>>
>> --
>> Regards,
>> Ouch Whisper
>> 010101010101
>>
>
>
--
Nitin Pawar
Re: Copy files from remote folder to HDFS
Posted by Nitin Pawar <ni...@gmail.com>.
if this is one time activity
then just download hadoop binaries from apache
replace the hdfs-site.xml and core-site.xml with the one you have on
hadoop cluster
allow this machine to connect with hadoop cluster
then you can just do it with hadoop command line scripts.
On Fri, Jan 25, 2013 at 1:01 PM, Mahesh Balija
<ba...@gmail.com>wrote:
> Hi Panshul,
>
> I am also working on similar requirement, one approach is,
> mount your remote folder on your hadoop master node.
> And simply write a shell script to copy the files to HDFS
> using crontab.
>
> I believe Flume is literally a wrong choice as Flume is a
> data collection and aggregation framework and NOT a file transfer tool and
> may NOT be a good choice when you actually want to copy the files as-is
> onto your cluster (NOT 100% sure as I am also working on that).
>
> Thanks,
> Mahesh Balija,
> CalsoftLabs.
>
> On Fri, Jan 25, 2013 at 6:39 AM, Panshul Whisper <ou...@gmail.com>wrote:
>
>> Hello,
>>
>> I am trying to copy files, Json files from a remote folder - (a folder on
>> my local system, Cloudfiles folder or a folder on S3 server) to the HDFS of
>> a cluster running at a remote location.
>> The job submitting Application is based on Spring Hadoop.
>>
>> Can someone please suggest or point me in the right direction for best
>> option to achieve the above task:
>> 1. Use Spring Integration data pipelines to poll the folders for files
>> and copy them to the HDFS as they arrive in the source folder. - I have
>> tried to implement the solution in Spring Data book, but it does not run -
>> no idea what is wrong as it does not generate logs.
>>
>> 2. Use some other script method to transfer files.
>>
>> Main requirement, I need to transfer files from a remote folder to HDFS
>> everyday at a fixed time for processing in the hadoop cluster. These files
>> are collecting from various sources in the remote folders.
>>
>> Please suggest an efficient approach. I have been searching and finding a
>> lot of approaches but unable to decide what will work best. As this
>> transfer needs to be as fast as possible.
>> The files to be transferred will be almost 10 GB of Json files not more
>> than 6kb each file.
>>
>> Thanking You,
>>
>>
>> --
>> Regards,
>> Ouch Whisper
>> 010101010101
>>
>
>
--
Nitin Pawar
Re: Copy files from remote folder to HDFS
Posted by Nitin Pawar <ni...@gmail.com>.
if this is one time activity
then just download hadoop binaries from apache
replace the hdfs-site.xml and core-site.xml with the one you have on
hadoop cluster
allow this machine to connect with hadoop cluster
then you can just do it with hadoop command line scripts.
On Fri, Jan 25, 2013 at 1:01 PM, Mahesh Balija
<ba...@gmail.com>wrote:
> Hi Panshul,
>
> I am also working on similar requirement, one approach is,
> mount your remote folder on your hadoop master node.
> And simply write a shell script to copy the files to HDFS
> using crontab.
>
> I believe Flume is literally a wrong choice as Flume is a
> data collection and aggregation framework and NOT a file transfer tool and
> may NOT be a good choice when you actually want to copy the files as-is
> onto your cluster (NOT 100% sure as I am also working on that).
>
> Thanks,
> Mahesh Balija,
> CalsoftLabs.
>
> On Fri, Jan 25, 2013 at 6:39 AM, Panshul Whisper <ou...@gmail.com>wrote:
>
>> Hello,
>>
>> I am trying to copy files, Json files from a remote folder - (a folder on
>> my local system, Cloudfiles folder or a folder on S3 server) to the HDFS of
>> a cluster running at a remote location.
>> The job submitting Application is based on Spring Hadoop.
>>
>> Can someone please suggest or point me in the right direction for best
>> option to achieve the above task:
>> 1. Use Spring Integration data pipelines to poll the folders for files
>> and copy them to the HDFS as they arrive in the source folder. - I have
>> tried to implement the solution in Spring Data book, but it does not run -
>> no idea what is wrong as it does not generate logs.
>>
>> 2. Use some other script method to transfer files.
>>
>> Main requirement, I need to transfer files from a remote folder to HDFS
>> everyday at a fixed time for processing in the hadoop cluster. These files
>> are collecting from various sources in the remote folders.
>>
>> Please suggest an efficient approach. I have been searching and finding a
>> lot of approaches but unable to decide what will work best. As this
>> transfer needs to be as fast as possible.
>> The files to be transferred will be almost 10 GB of Json files not more
>> than 6kb each file.
>>
>> Thanking You,
>>
>>
>> --
>> Regards,
>> Ouch Whisper
>> 010101010101
>>
>
>
--
Nitin Pawar
Re: Copy files from remote folder to HDFS
Posted by Nitin Pawar <ni...@gmail.com>.
if this is one time activity
then just download hadoop binaries from apache
replace the hdfs-site.xml and core-site.xml with the one you have on
hadoop cluster
allow this machine to connect with hadoop cluster
then you can just do it with hadoop command line scripts.
On Fri, Jan 25, 2013 at 1:01 PM, Mahesh Balija
<ba...@gmail.com>wrote:
> Hi Panshul,
>
> I am also working on similar requirement, one approach is,
> mount your remote folder on your hadoop master node.
> And simply write a shell script to copy the files to HDFS
> using crontab.
>
> I believe Flume is literally a wrong choice as Flume is a
> data collection and aggregation framework and NOT a file transfer tool and
> may NOT be a good choice when you actually want to copy the files as-is
> onto your cluster (NOT 100% sure as I am also working on that).
>
> Thanks,
> Mahesh Balija,
> CalsoftLabs.
>
> On Fri, Jan 25, 2013 at 6:39 AM, Panshul Whisper <ou...@gmail.com>wrote:
>
>> Hello,
>>
>> I am trying to copy files, Json files from a remote folder - (a folder on
>> my local system, Cloudfiles folder or a folder on S3 server) to the HDFS of
>> a cluster running at a remote location.
>> The job submitting Application is based on Spring Hadoop.
>>
>> Can someone please suggest or point me in the right direction for best
>> option to achieve the above task:
>> 1. Use Spring Integration data pipelines to poll the folders for files
>> and copy them to the HDFS as they arrive in the source folder. - I have
>> tried to implement the solution in Spring Data book, but it does not run -
>> no idea what is wrong as it does not generate logs.
>>
>> 2. Use some other script method to transfer files.
>>
>> Main requirement, I need to transfer files from a remote folder to HDFS
>> everyday at a fixed time for processing in the hadoop cluster. These files
>> are collecting from various sources in the remote folders.
>>
>> Please suggest an efficient approach. I have been searching and finding a
>> lot of approaches but unable to decide what will work best. As this
>> transfer needs to be as fast as possible.
>> The files to be transferred will be almost 10 GB of Json files not more
>> than 6kb each file.
>>
>> Thanking You,
>>
>>
>> --
>> Regards,
>> Ouch Whisper
>> 010101010101
>>
>
>
--
Nitin Pawar
Re: Copy files from remote folder to HDFS
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Panshul,
I am also working on similar requirement, one approach is,
mount your remote folder on your hadoop master node.
And simply write a shell script to copy the files to HDFS
using crontab.
I believe Flume is literally a wrong choice as Flume is a
data collection and aggregation framework and NOT a file transfer tool and
may NOT be a good choice when you actually want to copy the files as-is
onto your cluster (NOT 100% sure as I am also working on that).
Thanks,
Mahesh Balija,
CalsoftLabs.
On Fri, Jan 25, 2013 at 6:39 AM, Panshul Whisper <ou...@gmail.com>wrote:
> Hello,
>
> I am trying to copy files, Json files from a remote folder - (a folder on
> my local system, Cloudfiles folder or a folder on S3 server) to the HDFS of
> a cluster running at a remote location.
> The job submitting Application is based on Spring Hadoop.
>
> Can someone please suggest or point me in the right direction for best
> option to achieve the above task:
> 1. Use Spring Integration data pipelines to poll the folders for files and
> copy them to the HDFS as they arrive in the source folder. - I have tried
> to implement the solution in Spring Data book, but it does not run - no
> idea what is wrong as it does not generate logs.
>
> 2. Use some other script method to transfer files.
>
> Main requirement, I need to transfer files from a remote folder to HDFS
> everyday at a fixed time for processing in the hadoop cluster. These files
> are collecting from various sources in the remote folders.
>
> Please suggest an efficient approach. I have been searching and finding a
> lot of approaches but unable to decide what will work best. As this
> transfer needs to be as fast as possible.
> The files to be transferred will be almost 10 GB of Json files not more
> than 6kb each file.
>
> Thanking You,
>
>
> --
> Regards,
> Ouch Whisper
> 010101010101
>
Re: Copy files from remote folder to HDFS
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Panshul,
You might find flume <http://flume.apache.org/> useful.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Fri, Jan 25, 2013 at 6:39 AM, Panshul Whisper <ou...@gmail.com>wrote:
> Hello,
>
> I am trying to copy files, Json files from a remote folder - (a folder on
> my local system, Cloudfiles folder or a folder on S3 server) to the HDFS of
> a cluster running at a remote location.
> The job submitting Application is based on Spring Hadoop.
>
> Can someone please suggest or point me in the right direction for best
> option to achieve the above task:
> 1. Use Spring Integration data pipelines to poll the folders for files and
> copy them to the HDFS as they arrive in the source folder. - I have tried
> to implement the solution in Spring Data book, but it does not run - no
> idea what is wrong as it does not generate logs.
>
> 2. Use some other script method to transfer files.
>
> Main requirement, I need to transfer files from a remote folder to HDFS
> everyday at a fixed time for processing in the hadoop cluster. These files
> are collecting from various sources in the remote folders.
>
> Please suggest an efficient approach. I have been searching and finding a
> lot of approaches but unable to decide what will work best. As this
> transfer needs to be as fast as possible.
> The files to be transferred will be almost 10 GB of Json files not more
> than 6kb each file.
>
> Thanking You,
>
>
> --
> Regards,
> Ouch Whisper
> 010101010101
>
Re: Copy files from remote folder to HDFS
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Panshul,
I am also working on similar requirement, one approach is,
mount your remote folder on your hadoop master node.
And simply write a shell script to copy the files to HDFS
using crontab.
I believe Flume is literally a wrong choice as Flume is a
data collection and aggregation framework and NOT a file transfer tool and
may NOT be a good choice when you actually want to copy the files as-is
onto your cluster (NOT 100% sure as I am also working on that).
Thanks,
Mahesh Balija,
CalsoftLabs.
On Fri, Jan 25, 2013 at 6:39 AM, Panshul Whisper <ou...@gmail.com>wrote:
> Hello,
>
> I am trying to copy files, Json files from a remote folder - (a folder on
> my local system, Cloudfiles folder or a folder on S3 server) to the HDFS of
> a cluster running at a remote location.
> The job submitting Application is based on Spring Hadoop.
>
> Can someone please suggest or point me in the right direction for best
> option to achieve the above task:
> 1. Use Spring Integration data pipelines to poll the folders for files and
> copy them to the HDFS as they arrive in the source folder. - I have tried
> to implement the solution in Spring Data book, but it does not run - no
> idea what is wrong as it does not generate logs.
>
> 2. Use some other script method to transfer files.
>
> Main requirement, I need to transfer files from a remote folder to HDFS
> everyday at a fixed time for processing in the hadoop cluster. These files
> are collecting from various sources in the remote folders.
>
> Please suggest an efficient approach. I have been searching and finding a
> lot of approaches but unable to decide what will work best. As this
> transfer needs to be as fast as possible.
> The files to be transferred will be almost 10 GB of Json files not more
> than 6kb each file.
>
> Thanking You,
>
>
> --
> Regards,
> Ouch Whisper
> 010101010101
>
Re: Copy files from remote folder to HDFS
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Panshul,
You might find flume <http://flume.apache.org/> useful.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Fri, Jan 25, 2013 at 6:39 AM, Panshul Whisper <ou...@gmail.com>wrote:
> Hello,
>
> I am trying to copy files, Json files from a remote folder - (a folder on
> my local system, Cloudfiles folder or a folder on S3 server) to the HDFS of
> a cluster running at a remote location.
> The job submitting Application is based on Spring Hadoop.
>
> Can someone please suggest or point me in the right direction for best
> option to achieve the above task:
> 1. Use Spring Integration data pipelines to poll the folders for files and
> copy them to the HDFS as they arrive in the source folder. - I have tried
> to implement the solution in Spring Data book, but it does not run - no
> idea what is wrong as it does not generate logs.
>
> 2. Use some other script method to transfer files.
>
> Main requirement, I need to transfer files from a remote folder to HDFS
> everyday at a fixed time for processing in the hadoop cluster. These files
> are collecting from various sources in the remote folders.
>
> Please suggest an efficient approach. I have been searching and finding a
> lot of approaches but unable to decide what will work best. As this
> transfer needs to be as fast as possible.
> The files to be transferred will be almost 10 GB of Json files not more
> than 6kb each file.
>
> Thanking You,
>
>
> --
> Regards,
> Ouch Whisper
> 010101010101
>
Re: Copy files from remote folder to HDFS
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Panshul,
I am also working on similar requirement, one approach is,
mount your remote folder on your hadoop master node.
And simply write a shell script to copy the files to HDFS
using crontab.
I believe Flume is literally a wrong choice as Flume is a
data collection and aggregation framework and NOT a file transfer tool and
may NOT be a good choice when you actually want to copy the files as-is
onto your cluster (NOT 100% sure as I am also working on that).
Thanks,
Mahesh Balija,
CalsoftLabs.
On Fri, Jan 25, 2013 at 6:39 AM, Panshul Whisper <ou...@gmail.com>wrote:
> Hello,
>
> I am trying to copy files, Json files from a remote folder - (a folder on
> my local system, Cloudfiles folder or a folder on S3 server) to the HDFS of
> a cluster running at a remote location.
> The job submitting Application is based on Spring Hadoop.
>
> Can someone please suggest or point me in the right direction for best
> option to achieve the above task:
> 1. Use Spring Integration data pipelines to poll the folders for files and
> copy them to the HDFS as they arrive in the source folder. - I have tried
> to implement the solution in Spring Data book, but it does not run - no
> idea what is wrong as it does not generate logs.
>
> 2. Use some other script method to transfer files.
>
> Main requirement, I need to transfer files from a remote folder to HDFS
> everyday at a fixed time for processing in the hadoop cluster. These files
> are collecting from various sources in the remote folders.
>
> Please suggest an efficient approach. I have been searching and finding a
> lot of approaches but unable to decide what will work best. As this
> transfer needs to be as fast as possible.
> The files to be transferred will be almost 10 GB of Json files not more
> than 6kb each file.
>
> Thanking You,
>
>
> --
> Regards,
> Ouch Whisper
> 010101010101
>
Re: Copy files from remote folder to HDFS
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Panshul,
I am also working on similar requirement, one approach is,
mount your remote folder on your hadoop master node.
And simply write a shell script to copy the files to HDFS
using crontab.
I believe Flume is literally a wrong choice as Flume is a
data collection and aggregation framework and NOT a file transfer tool and
may NOT be a good choice when you actually want to copy the files as-is
onto your cluster (NOT 100% sure as I am also working on that).
Thanks,
Mahesh Balija,
CalsoftLabs.
On Fri, Jan 25, 2013 at 6:39 AM, Panshul Whisper <ou...@gmail.com>wrote:
> Hello,
>
> I am trying to copy files, Json files from a remote folder - (a folder on
> my local system, Cloudfiles folder or a folder on S3 server) to the HDFS of
> a cluster running at a remote location.
> The job submitting Application is based on Spring Hadoop.
>
> Can someone please suggest or point me in the right direction for best
> option to achieve the above task:
> 1. Use Spring Integration data pipelines to poll the folders for files and
> copy them to the HDFS as they arrive in the source folder. - I have tried
> to implement the solution in Spring Data book, but it does not run - no
> idea what is wrong as it does not generate logs.
>
> 2. Use some other script method to transfer files.
>
> Main requirement, I need to transfer files from a remote folder to HDFS
> everyday at a fixed time for processing in the hadoop cluster. These files
> are collecting from various sources in the remote folders.
>
> Please suggest an efficient approach. I have been searching and finding a
> lot of approaches but unable to decide what will work best. As this
> transfer needs to be as fast as possible.
> The files to be transferred will be almost 10 GB of Json files not more
> than 6kb each file.
>
> Thanking You,
>
>
> --
> Regards,
> Ouch Whisper
> 010101010101
>
Re: Copy files from remote folder to HDFS
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Panshul,
You might find flume <http://flume.apache.org/> useful.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Fri, Jan 25, 2013 at 6:39 AM, Panshul Whisper <ou...@gmail.com>wrote:
> Hello,
>
> I am trying to copy files, Json files from a remote folder - (a folder on
> my local system, Cloudfiles folder or a folder on S3 server) to the HDFS of
> a cluster running at a remote location.
> The job submitting Application is based on Spring Hadoop.
>
> Can someone please suggest or point me in the right direction for best
> option to achieve the above task:
> 1. Use Spring Integration data pipelines to poll the folders for files and
> copy them to the HDFS as they arrive in the source folder. - I have tried
> to implement the solution in Spring Data book, but it does not run - no
> idea what is wrong as it does not generate logs.
>
> 2. Use some other script method to transfer files.
>
> Main requirement, I need to transfer files from a remote folder to HDFS
> everyday at a fixed time for processing in the hadoop cluster. These files
> are collecting from various sources in the remote folders.
>
> Please suggest an efficient approach. I have been searching and finding a
> lot of approaches but unable to decide what will work best. As this
> transfer needs to be as fast as possible.
> The files to be transferred will be almost 10 GB of Json files not more
> than 6kb each file.
>
> Thanking You,
>
>
> --
> Regards,
> Ouch Whisper
> 010101010101
>
Re: Copy files from remote folder to HDFS
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Panshul,
You might find flume <http://flume.apache.org/> useful.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Fri, Jan 25, 2013 at 6:39 AM, Panshul Whisper <ou...@gmail.com>wrote:
> Hello,
>
> I am trying to copy files, Json files from a remote folder - (a folder on
> my local system, Cloudfiles folder or a folder on S3 server) to the HDFS of
> a cluster running at a remote location.
> The job submitting Application is based on Spring Hadoop.
>
> Can someone please suggest or point me in the right direction for best
> option to achieve the above task:
> 1. Use Spring Integration data pipelines to poll the folders for files and
> copy them to the HDFS as they arrive in the source folder. - I have tried
> to implement the solution in Spring Data book, but it does not run - no
> idea what is wrong as it does not generate logs.
>
> 2. Use some other script method to transfer files.
>
> Main requirement, I need to transfer files from a remote folder to HDFS
> everyday at a fixed time for processing in the hadoop cluster. These files
> are collecting from various sources in the remote folders.
>
> Please suggest an efficient approach. I have been searching and finding a
> lot of approaches but unable to decide what will work best. As this
> transfer needs to be as fast as possible.
> The files to be transferred will be almost 10 GB of Json files not more
> than 6kb each file.
>
> Thanking You,
>
>
> --
> Regards,
> Ouch Whisper
> 010101010101
>