You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Divya Gehlot <di...@gmail.com> on 2015/06/19 06:26:17 UTC
copy data from one hadoop cluster to another hadoop cluster + cant
use distcp
Hi,
I need to copy data from first hadoop cluster to second hadoop cluster.
I cant access second hadoop cluster from first hadoop cluster due to some
security issue.
Can any point me how can I do apart from distcp command.
For instance
Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
zone
Thanks,
Divya
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by max scalf <or...@gmail.com>.
Not to hijack this post but how would you deal with data that is maintained
by hive(Orc format file, hive created tables etc..)...Would we copy the
hivemetastore(MySQL) and move that over to new cluster?
On Friday, June 19, 2015, Joep Rottinghuis <jr...@gmail.com> wrote:
> You can't set up a proxy ?
> You probably want to avoid writing to local file system because aside from
> that being slow, it limits the size of your file to the free space on your
> local disc.
>
> If you do need to go commando and go through a single client machine that
> can see both clusters you probably want to pipe a get to a put.
>
> Any kind of serious data volume pulled through a "straw" is going to be
> rather slow though.
>
> Cheers,
>
> Joep
>
> Sent from my iPhone
>
> On Jun 19, 2015, at 12:09 AM, Nitin Pawar <nitinpawar432@gmail.com
> <javascript:_e(%7B%7D,'cvml','nitinpawar432@gmail.com');>> wrote:
>
> yes
>
> On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot <divya.htconex@gmail.com
> <javascript:_e(%7B%7D,'cvml','divya.htconex@gmail.com');>> wrote:
>
>> In thats It will be like three step process .
>> 1. first cluster (secure zone) HDFS -> copytoLocal -> user local file
>> system
>> 2. user local space -> copy data -> second cluster user local file system
>> 3. second cluster user local file system -> copyfromlocal -> second
>> clusterHDFS
>>
>> Am I on the right track ?
>>
>>
>>
>> On 19 June 2015 at 12:38, Nitin Pawar <nitinpawar432@gmail.com
>> <javascript:_e(%7B%7D,'cvml','nitinpawar432@gmail.com');>> wrote:
>>
>>> What's the size of the data?
>>> If you can not do distcp between clusters then other way is doing hdfs
>>> get on the data and then hdfs put on another cluster
>>> On 19-Jun-2015 9:56 am, "Divya Gehlot" <divya.htconex@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','divya.htconex@gmail.com');>> wrote:
>>>
>>>> Hi,
>>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>>> I cant access second hadoop cluster from first hadoop cluster due to
>>>> some security issue.
>>>> Can any point me how can I do apart from distcp command.
>>>> For instance
>>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non
>>>> secured zone
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Divya
>>>>
>>>>
>>>>
>>
>
>
> --
> Nitin Pawar
>
>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by max scalf <or...@gmail.com>.
Not to hijack this post but how would you deal with data that is maintained
by hive(Orc format file, hive created tables etc..)...Would we copy the
hivemetastore(MySQL) and move that over to new cluster?
On Friday, June 19, 2015, Joep Rottinghuis <jr...@gmail.com> wrote:
> You can't set up a proxy ?
> You probably want to avoid writing to local file system because aside from
> that being slow, it limits the size of your file to the free space on your
> local disc.
>
> If you do need to go commando and go through a single client machine that
> can see both clusters you probably want to pipe a get to a put.
>
> Any kind of serious data volume pulled through a "straw" is going to be
> rather slow though.
>
> Cheers,
>
> Joep
>
> Sent from my iPhone
>
> On Jun 19, 2015, at 12:09 AM, Nitin Pawar <nitinpawar432@gmail.com
> <javascript:_e(%7B%7D,'cvml','nitinpawar432@gmail.com');>> wrote:
>
> yes
>
> On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot <divya.htconex@gmail.com
> <javascript:_e(%7B%7D,'cvml','divya.htconex@gmail.com');>> wrote:
>
>> In thats It will be like three step process .
>> 1. first cluster (secure zone) HDFS -> copytoLocal -> user local file
>> system
>> 2. user local space -> copy data -> second cluster user local file system
>> 3. second cluster user local file system -> copyfromlocal -> second
>> clusterHDFS
>>
>> Am I on the right track ?
>>
>>
>>
>> On 19 June 2015 at 12:38, Nitin Pawar <nitinpawar432@gmail.com
>> <javascript:_e(%7B%7D,'cvml','nitinpawar432@gmail.com');>> wrote:
>>
>>> What's the size of the data?
>>> If you can not do distcp between clusters then other way is doing hdfs
>>> get on the data and then hdfs put on another cluster
>>> On 19-Jun-2015 9:56 am, "Divya Gehlot" <divya.htconex@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','divya.htconex@gmail.com');>> wrote:
>>>
>>>> Hi,
>>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>>> I cant access second hadoop cluster from first hadoop cluster due to
>>>> some security issue.
>>>> Can any point me how can I do apart from distcp command.
>>>> For instance
>>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non
>>>> secured zone
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Divya
>>>>
>>>>
>>>>
>>
>
>
> --
> Nitin Pawar
>
>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by max scalf <or...@gmail.com>.
Not to hijack this post but how would you deal with data that is maintained
by hive(Orc format file, hive created tables etc..)...Would we copy the
hivemetastore(MySQL) and move that over to new cluster?
On Friday, June 19, 2015, Joep Rottinghuis <jr...@gmail.com> wrote:
> You can't set up a proxy ?
> You probably want to avoid writing to local file system because aside from
> that being slow, it limits the size of your file to the free space on your
> local disc.
>
> If you do need to go commando and go through a single client machine that
> can see both clusters you probably want to pipe a get to a put.
>
> Any kind of serious data volume pulled through a "straw" is going to be
> rather slow though.
>
> Cheers,
>
> Joep
>
> Sent from my iPhone
>
> On Jun 19, 2015, at 12:09 AM, Nitin Pawar <nitinpawar432@gmail.com
> <javascript:_e(%7B%7D,'cvml','nitinpawar432@gmail.com');>> wrote:
>
> yes
>
> On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot <divya.htconex@gmail.com
> <javascript:_e(%7B%7D,'cvml','divya.htconex@gmail.com');>> wrote:
>
>> In thats It will be like three step process .
>> 1. first cluster (secure zone) HDFS -> copytoLocal -> user local file
>> system
>> 2. user local space -> copy data -> second cluster user local file system
>> 3. second cluster user local file system -> copyfromlocal -> second
>> clusterHDFS
>>
>> Am I on the right track ?
>>
>>
>>
>> On 19 June 2015 at 12:38, Nitin Pawar <nitinpawar432@gmail.com
>> <javascript:_e(%7B%7D,'cvml','nitinpawar432@gmail.com');>> wrote:
>>
>>> What's the size of the data?
>>> If you can not do distcp between clusters then other way is doing hdfs
>>> get on the data and then hdfs put on another cluster
>>> On 19-Jun-2015 9:56 am, "Divya Gehlot" <divya.htconex@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','divya.htconex@gmail.com');>> wrote:
>>>
>>>> Hi,
>>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>>> I cant access second hadoop cluster from first hadoop cluster due to
>>>> some security issue.
>>>> Can any point me how can I do apart from distcp command.
>>>> For instance
>>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non
>>>> secured zone
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Divya
>>>>
>>>>
>>>>
>>
>
>
> --
> Nitin Pawar
>
>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by max scalf <or...@gmail.com>.
Not to hijack this post but how would you deal with data that is maintained
by hive(Orc format file, hive created tables etc..)...Would we copy the
hivemetastore(MySQL) and move that over to new cluster?
On Friday, June 19, 2015, Joep Rottinghuis <jr...@gmail.com> wrote:
> You can't set up a proxy ?
> You probably want to avoid writing to local file system because aside from
> that being slow, it limits the size of your file to the free space on your
> local disc.
>
> If you do need to go commando and go through a single client machine that
> can see both clusters you probably want to pipe a get to a put.
>
> Any kind of serious data volume pulled through a "straw" is going to be
> rather slow though.
>
> Cheers,
>
> Joep
>
> Sent from my iPhone
>
> On Jun 19, 2015, at 12:09 AM, Nitin Pawar <nitinpawar432@gmail.com
> <javascript:_e(%7B%7D,'cvml','nitinpawar432@gmail.com');>> wrote:
>
> yes
>
> On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot <divya.htconex@gmail.com
> <javascript:_e(%7B%7D,'cvml','divya.htconex@gmail.com');>> wrote:
>
>> In thats It will be like three step process .
>> 1. first cluster (secure zone) HDFS -> copytoLocal -> user local file
>> system
>> 2. user local space -> copy data -> second cluster user local file system
>> 3. second cluster user local file system -> copyfromlocal -> second
>> clusterHDFS
>>
>> Am I on the right track ?
>>
>>
>>
>> On 19 June 2015 at 12:38, Nitin Pawar <nitinpawar432@gmail.com
>> <javascript:_e(%7B%7D,'cvml','nitinpawar432@gmail.com');>> wrote:
>>
>>> What's the size of the data?
>>> If you can not do distcp between clusters then other way is doing hdfs
>>> get on the data and then hdfs put on another cluster
>>> On 19-Jun-2015 9:56 am, "Divya Gehlot" <divya.htconex@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','divya.htconex@gmail.com');>> wrote:
>>>
>>>> Hi,
>>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>>> I cant access second hadoop cluster from first hadoop cluster due to
>>>> some security issue.
>>>> Can any point me how can I do apart from distcp command.
>>>> For instance
>>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non
>>>> secured zone
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Divya
>>>>
>>>>
>>>>
>>
>
>
> --
> Nitin Pawar
>
>
Re: copy data from one hadoop cluster to another hadoop cluster + cant use distcp
Posted by Joep Rottinghuis <jr...@gmail.com>.
You can't set up a proxy ?
You probably want to avoid writing to local file system because aside from that being slow, it limits the size of your file to the free space on your local disc.
If you do need to go commando and go through a single client machine that can see both clusters you probably want to pipe a get to a put.
Any kind of serious data volume pulled through a "straw" is going to be rather slow though.
Cheers,
Joep
Sent from my iPhone
> On Jun 19, 2015, at 12:09 AM, Nitin Pawar <ni...@gmail.com> wrote:
>
> yes
>
>> On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot <di...@gmail.com> wrote:
>> In thats It will be like three step process .
>> 1. first cluster (secure zone) HDFS -> copytoLocal -> user local file system
>> 2. user local space -> copy data -> second cluster user local file system
>> 3. second cluster user local file system -> copyfromlocal -> second clusterHDFS
>>
>> Am I on the right track ?
>>
>>
>>
>>> On 19 June 2015 at 12:38, Nitin Pawar <ni...@gmail.com> wrote:
>>> What's the size of the data?
>>> If you can not do distcp between clusters then other way is doing hdfs get on the data and then hdfs put on another cluster
>>>
>>>> On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
>>>> Hi,
>>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>>> I cant access second hadoop cluster from first hadoop cluster due to some security issue.
>>>> Can any point me how can I do apart from distcp command.
>>>> For instance
>>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured zone
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Divya
>
>
>
> --
> Nitin Pawar
Re: copy data from one hadoop cluster to another hadoop cluster + cant use distcp
Posted by Joep Rottinghuis <jr...@gmail.com>.
You can't set up a proxy ?
You probably want to avoid writing to local file system because aside from that being slow, it limits the size of your file to the free space on your local disc.
If you do need to go commando and go through a single client machine that can see both clusters you probably want to pipe a get to a put.
Any kind of serious data volume pulled through a "straw" is going to be rather slow though.
Cheers,
Joep
Sent from my iPhone
> On Jun 19, 2015, at 12:09 AM, Nitin Pawar <ni...@gmail.com> wrote:
>
> yes
>
>> On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot <di...@gmail.com> wrote:
>> In thats It will be like three step process .
>> 1. first cluster (secure zone) HDFS -> copytoLocal -> user local file system
>> 2. user local space -> copy data -> second cluster user local file system
>> 3. second cluster user local file system -> copyfromlocal -> second clusterHDFS
>>
>> Am I on the right track ?
>>
>>
>>
>>> On 19 June 2015 at 12:38, Nitin Pawar <ni...@gmail.com> wrote:
>>> What's the size of the data?
>>> If you can not do distcp between clusters then other way is doing hdfs get on the data and then hdfs put on another cluster
>>>
>>>> On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
>>>> Hi,
>>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>>> I cant access second hadoop cluster from first hadoop cluster due to some security issue.
>>>> Can any point me how can I do apart from distcp command.
>>>> For instance
>>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured zone
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Divya
>
>
>
> --
> Nitin Pawar
Re: copy data from one hadoop cluster to another hadoop cluster + cant use distcp
Posted by Joep Rottinghuis <jr...@gmail.com>.
You can't set up a proxy ?
You probably want to avoid writing to local file system because aside from that being slow, it limits the size of your file to the free space on your local disc.
If you do need to go commando and go through a single client machine that can see both clusters you probably want to pipe a get to a put.
Any kind of serious data volume pulled through a "straw" is going to be rather slow though.
Cheers,
Joep
Sent from my iPhone
> On Jun 19, 2015, at 12:09 AM, Nitin Pawar <ni...@gmail.com> wrote:
>
> yes
>
>> On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot <di...@gmail.com> wrote:
>> In thats It will be like three step process .
>> 1. first cluster (secure zone) HDFS -> copytoLocal -> user local file system
>> 2. user local space -> copy data -> second cluster user local file system
>> 3. second cluster user local file system -> copyfromlocal -> second clusterHDFS
>>
>> Am I on the right track ?
>>
>>
>>
>>> On 19 June 2015 at 12:38, Nitin Pawar <ni...@gmail.com> wrote:
>>> What's the size of the data?
>>> If you can not do distcp between clusters then other way is doing hdfs get on the data and then hdfs put on another cluster
>>>
>>>> On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
>>>> Hi,
>>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>>> I cant access second hadoop cluster from first hadoop cluster due to some security issue.
>>>> Can any point me how can I do apart from distcp command.
>>>> For instance
>>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured zone
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Divya
>
>
>
> --
> Nitin Pawar
Re: copy data from one hadoop cluster to another hadoop cluster + cant use distcp
Posted by Joep Rottinghuis <jr...@gmail.com>.
You can't set up a proxy ?
You probably want to avoid writing to local file system because aside from that being slow, it limits the size of your file to the free space on your local disc.
If you do need to go commando and go through a single client machine that can see both clusters you probably want to pipe a get to a put.
Any kind of serious data volume pulled through a "straw" is going to be rather slow though.
Cheers,
Joep
Sent from my iPhone
> On Jun 19, 2015, at 12:09 AM, Nitin Pawar <ni...@gmail.com> wrote:
>
> yes
>
>> On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot <di...@gmail.com> wrote:
>> In thats It will be like three step process .
>> 1. first cluster (secure zone) HDFS -> copytoLocal -> user local file system
>> 2. user local space -> copy data -> second cluster user local file system
>> 3. second cluster user local file system -> copyfromlocal -> second clusterHDFS
>>
>> Am I on the right track ?
>>
>>
>>
>>> On 19 June 2015 at 12:38, Nitin Pawar <ni...@gmail.com> wrote:
>>> What's the size of the data?
>>> If you can not do distcp between clusters then other way is doing hdfs get on the data and then hdfs put on another cluster
>>>
>>>> On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
>>>> Hi,
>>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>>> I cant access second hadoop cluster from first hadoop cluster due to some security issue.
>>>> Can any point me how can I do apart from distcp command.
>>>> For instance
>>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured zone
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Divya
>
>
>
> --
> Nitin Pawar
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Nitin Pawar <ni...@gmail.com>.
yes
On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot <di...@gmail.com>
wrote:
> In thats It will be like three step process .
> 1. first cluster (secure zone) HDFS -> copytoLocal -> user local file
> system
> 2. user local space -> copy data -> second cluster user local file system
> 3. second cluster user local file system -> copyfromlocal -> second
> clusterHDFS
>
> Am I on the right track ?
>
>
>
> On 19 June 2015 at 12:38, Nitin Pawar <ni...@gmail.com> wrote:
>
>> What's the size of the data?
>> If you can not do distcp between clusters then other way is doing hdfs
>> get on the data and then hdfs put on another cluster
>> On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
>>
>>> Hi,
>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>> I cant access second hadoop cluster from first hadoop cluster due to
>>> some security issue.
>>> Can any point me how can I do apart from distcp command.
>>> For instance
>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>>> zone
>>>
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>>
>>>
>
--
Nitin Pawar
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Nitin Pawar <ni...@gmail.com>.
yes
On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot <di...@gmail.com>
wrote:
> In thats It will be like three step process .
> 1. first cluster (secure zone) HDFS -> copytoLocal -> user local file
> system
> 2. user local space -> copy data -> second cluster user local file system
> 3. second cluster user local file system -> copyfromlocal -> second
> clusterHDFS
>
> Am I on the right track ?
>
>
>
> On 19 June 2015 at 12:38, Nitin Pawar <ni...@gmail.com> wrote:
>
>> What's the size of the data?
>> If you can not do distcp between clusters then other way is doing hdfs
>> get on the data and then hdfs put on another cluster
>> On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
>>
>>> Hi,
>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>> I cant access second hadoop cluster from first hadoop cluster due to
>>> some security issue.
>>> Can any point me how can I do apart from distcp command.
>>> For instance
>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>>> zone
>>>
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>>
>>>
>
--
Nitin Pawar
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Nitin Pawar <ni...@gmail.com>.
yes
On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot <di...@gmail.com>
wrote:
> In thats It will be like three step process .
> 1. first cluster (secure zone) HDFS -> copytoLocal -> user local file
> system
> 2. user local space -> copy data -> second cluster user local file system
> 3. second cluster user local file system -> copyfromlocal -> second
> clusterHDFS
>
> Am I on the right track ?
>
>
>
> On 19 June 2015 at 12:38, Nitin Pawar <ni...@gmail.com> wrote:
>
>> What's the size of the data?
>> If you can not do distcp between clusters then other way is doing hdfs
>> get on the data and then hdfs put on another cluster
>> On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
>>
>>> Hi,
>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>> I cant access second hadoop cluster from first hadoop cluster due to
>>> some security issue.
>>> Can any point me how can I do apart from distcp command.
>>> For instance
>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>>> zone
>>>
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>>
>>>
>
--
Nitin Pawar
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Nitin Pawar <ni...@gmail.com>.
yes
On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot <di...@gmail.com>
wrote:
> In thats It will be like three step process .
> 1. first cluster (secure zone) HDFS -> copytoLocal -> user local file
> system
> 2. user local space -> copy data -> second cluster user local file system
> 3. second cluster user local file system -> copyfromlocal -> second
> clusterHDFS
>
> Am I on the right track ?
>
>
>
> On 19 June 2015 at 12:38, Nitin Pawar <ni...@gmail.com> wrote:
>
>> What's the size of the data?
>> If you can not do distcp between clusters then other way is doing hdfs
>> get on the data and then hdfs put on another cluster
>> On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
>>
>>> Hi,
>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>> I cant access second hadoop cluster from first hadoop cluster due to
>>> some security issue.
>>> Can any point me how can I do apart from distcp command.
>>> For instance
>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>>> zone
>>>
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>>
>>>
>
--
Nitin Pawar
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Divya Gehlot <di...@gmail.com>.
In thats It will be like three step process .
1. first cluster (secure zone) HDFS -> copytoLocal -> user local file
system
2. user local space -> copy data -> second cluster user local file system
3. second cluster user local file system -> copyfromlocal -> second
clusterHDFS
Am I on the right track ?
On 19 June 2015 at 12:38, Nitin Pawar <ni...@gmail.com> wrote:
> What's the size of the data?
> If you can not do distcp between clusters then other way is doing hdfs get
> on the data and then hdfs put on another cluster
> On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
>
>> Hi,
>> I need to copy data from first hadoop cluster to second hadoop cluster.
>> I cant access second hadoop cluster from first hadoop cluster due to some
>> security issue.
>> Can any point me how can I do apart from distcp command.
>> For instance
>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>> zone
>>
>>
>>
>> Thanks,
>> Divya
>>
>>
>>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Divya Gehlot <di...@gmail.com>.
In thats It will be like three step process .
1. first cluster (secure zone) HDFS -> copytoLocal -> user local file
system
2. user local space -> copy data -> second cluster user local file system
3. second cluster user local file system -> copyfromlocal -> second
clusterHDFS
Am I on the right track ?
On 19 June 2015 at 12:38, Nitin Pawar <ni...@gmail.com> wrote:
> What's the size of the data?
> If you can not do distcp between clusters then other way is doing hdfs get
> on the data and then hdfs put on another cluster
> On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
>
>> Hi,
>> I need to copy data from first hadoop cluster to second hadoop cluster.
>> I cant access second hadoop cluster from first hadoop cluster due to some
>> security issue.
>> Can any point me how can I do apart from distcp command.
>> For instance
>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>> zone
>>
>>
>>
>> Thanks,
>> Divya
>>
>>
>>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Divya Gehlot <di...@gmail.com>.
In thats It will be like three step process .
1. first cluster (secure zone) HDFS -> copytoLocal -> user local file
system
2. user local space -> copy data -> second cluster user local file system
3. second cluster user local file system -> copyfromlocal -> second
clusterHDFS
Am I on the right track ?
On 19 June 2015 at 12:38, Nitin Pawar <ni...@gmail.com> wrote:
> What's the size of the data?
> If you can not do distcp between clusters then other way is doing hdfs get
> on the data and then hdfs put on another cluster
> On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
>
>> Hi,
>> I need to copy data from first hadoop cluster to second hadoop cluster.
>> I cant access second hadoop cluster from first hadoop cluster due to some
>> security issue.
>> Can any point me how can I do apart from distcp command.
>> For instance
>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>> zone
>>
>>
>>
>> Thanks,
>> Divya
>>
>>
>>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Divya Gehlot <di...@gmail.com>.
In thats It will be like three step process .
1. first cluster (secure zone) HDFS -> copytoLocal -> user local file
system
2. user local space -> copy data -> second cluster user local file system
3. second cluster user local file system -> copyfromlocal -> second
clusterHDFS
Am I on the right track ?
On 19 June 2015 at 12:38, Nitin Pawar <ni...@gmail.com> wrote:
> What's the size of the data?
> If you can not do distcp between clusters then other way is doing hdfs get
> on the data and then hdfs put on another cluster
> On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
>
>> Hi,
>> I need to copy data from first hadoop cluster to second hadoop cluster.
>> I cant access second hadoop cluster from first hadoop cluster due to some
>> security issue.
>> Can any point me how can I do apart from distcp command.
>> For instance
>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>> zone
>>
>>
>>
>> Thanks,
>> Divya
>>
>>
>>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Nitin Pawar <ni...@gmail.com>.
What's the size of the data?
If you can not do distcp between clusters then other way is doing hdfs get
on the data and then hdfs put on another cluster
On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
> Hi,
> I need to copy data from first hadoop cluster to second hadoop cluster.
> I cant access second hadoop cluster from first hadoop cluster due to some
> security issue.
> Can any point me how can I do apart from distcp command.
> For instance
> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
> zone
>
>
>
> Thanks,
> Divya
>
>
>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by hadoop hive <ha...@gmail.com>.
You can use node.js for this.
On Tue, Jun 23, 2015 at 8:15 PM, Divya Gehlot <di...@gmail.com>
wrote:
> Can you please elaborate it more.
> On 20 Jun 2015 2:46 pm, "SF Hadoop" <sf...@gmail.com> wrote:
>
>> Really depends on your requirements for the format of the data.
>>
>> The easiest way I can think of is to "stream" batches of data into a pub
>> sub system that the target system can access and then consume.
>>
>> Verify each batch and then ditch them.
>>
>> You can throttle the size of the intermediary infrastructure based on
>> your batches.
>>
>> Seems the most efficient approach.
>>
>> On Thursday, June 18, 2015, Divya Gehlot <di...@gmail.com> wrote:
>>
>>> Hi,
>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>> I cant access second hadoop cluster from first hadoop cluster due to
>>> some security issue.
>>> Can any point me how can I do apart from distcp command.
>>> For instance
>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>>> zone
>>>
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>>
>>>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by hadoop hive <ha...@gmail.com>.
You can use node.js for this.
On Tue, Jun 23, 2015 at 8:15 PM, Divya Gehlot <di...@gmail.com>
wrote:
> Can you please elaborate it more.
> On 20 Jun 2015 2:46 pm, "SF Hadoop" <sf...@gmail.com> wrote:
>
>> Really depends on your requirements for the format of the data.
>>
>> The easiest way I can think of is to "stream" batches of data into a pub
>> sub system that the target system can access and then consume.
>>
>> Verify each batch and then ditch them.
>>
>> You can throttle the size of the intermediary infrastructure based on
>> your batches.
>>
>> Seems the most efficient approach.
>>
>> On Thursday, June 18, 2015, Divya Gehlot <di...@gmail.com> wrote:
>>
>>> Hi,
>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>> I cant access second hadoop cluster from first hadoop cluster due to
>>> some security issue.
>>> Can any point me how can I do apart from distcp command.
>>> For instance
>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>>> zone
>>>
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>>
>>>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by hadoop hive <ha...@gmail.com>.
You can use node.js for this.
On Tue, Jun 23, 2015 at 8:15 PM, Divya Gehlot <di...@gmail.com>
wrote:
> Can you please elaborate it more.
> On 20 Jun 2015 2:46 pm, "SF Hadoop" <sf...@gmail.com> wrote:
>
>> Really depends on your requirements for the format of the data.
>>
>> The easiest way I can think of is to "stream" batches of data into a pub
>> sub system that the target system can access and then consume.
>>
>> Verify each batch and then ditch them.
>>
>> You can throttle the size of the intermediary infrastructure based on
>> your batches.
>>
>> Seems the most efficient approach.
>>
>> On Thursday, June 18, 2015, Divya Gehlot <di...@gmail.com> wrote:
>>
>>> Hi,
>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>> I cant access second hadoop cluster from first hadoop cluster due to
>>> some security issue.
>>> Can any point me how can I do apart from distcp command.
>>> For instance
>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>>> zone
>>>
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>>
>>>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by hadoop hive <ha...@gmail.com>.
You can use node.js for this.
On Tue, Jun 23, 2015 at 8:15 PM, Divya Gehlot <di...@gmail.com>
wrote:
> Can you please elaborate it more.
> On 20 Jun 2015 2:46 pm, "SF Hadoop" <sf...@gmail.com> wrote:
>
>> Really depends on your requirements for the format of the data.
>>
>> The easiest way I can think of is to "stream" batches of data into a pub
>> sub system that the target system can access and then consume.
>>
>> Verify each batch and then ditch them.
>>
>> You can throttle the size of the intermediary infrastructure based on
>> your batches.
>>
>> Seems the most efficient approach.
>>
>> On Thursday, June 18, 2015, Divya Gehlot <di...@gmail.com> wrote:
>>
>>> Hi,
>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>> I cant access second hadoop cluster from first hadoop cluster due to
>>> some security issue.
>>> Can any point me how can I do apart from distcp command.
>>> For instance
>>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>>> zone
>>>
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>>
>>>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Divya Gehlot <di...@gmail.com>.
Can you please elaborate it more.
On 20 Jun 2015 2:46 pm, "SF Hadoop" <sf...@gmail.com> wrote:
> Really depends on your requirements for the format of the data.
>
> The easiest way I can think of is to "stream" batches of data into a pub
> sub system that the target system can access and then consume.
>
> Verify each batch and then ditch them.
>
> You can throttle the size of the intermediary infrastructure based on your
> batches.
>
> Seems the most efficient approach.
>
> On Thursday, June 18, 2015, Divya Gehlot <di...@gmail.com> wrote:
>
>> Hi,
>> I need to copy data from first hadoop cluster to second hadoop cluster.
>> I cant access second hadoop cluster from first hadoop cluster due to some
>> security issue.
>> Can any point me how can I do apart from distcp command.
>> For instance
>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>> zone
>>
>>
>>
>> Thanks,
>> Divya
>>
>>
>>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Divya Gehlot <di...@gmail.com>.
Can you please elaborate it more.
On 20 Jun 2015 2:46 pm, "SF Hadoop" <sf...@gmail.com> wrote:
> Really depends on your requirements for the format of the data.
>
> The easiest way I can think of is to "stream" batches of data into a pub
> sub system that the target system can access and then consume.
>
> Verify each batch and then ditch them.
>
> You can throttle the size of the intermediary infrastructure based on your
> batches.
>
> Seems the most efficient approach.
>
> On Thursday, June 18, 2015, Divya Gehlot <di...@gmail.com> wrote:
>
>> Hi,
>> I need to copy data from first hadoop cluster to second hadoop cluster.
>> I cant access second hadoop cluster from first hadoop cluster due to some
>> security issue.
>> Can any point me how can I do apart from distcp command.
>> For instance
>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>> zone
>>
>>
>>
>> Thanks,
>> Divya
>>
>>
>>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Divya Gehlot <di...@gmail.com>.
Can you please elaborate it more.
On 20 Jun 2015 2:46 pm, "SF Hadoop" <sf...@gmail.com> wrote:
> Really depends on your requirements for the format of the data.
>
> The easiest way I can think of is to "stream" batches of data into a pub
> sub system that the target system can access and then consume.
>
> Verify each batch and then ditch them.
>
> You can throttle the size of the intermediary infrastructure based on your
> batches.
>
> Seems the most efficient approach.
>
> On Thursday, June 18, 2015, Divya Gehlot <di...@gmail.com> wrote:
>
>> Hi,
>> I need to copy data from first hadoop cluster to second hadoop cluster.
>> I cant access second hadoop cluster from first hadoop cluster due to some
>> security issue.
>> Can any point me how can I do apart from distcp command.
>> For instance
>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>> zone
>>
>>
>>
>> Thanks,
>> Divya
>>
>>
>>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Divya Gehlot <di...@gmail.com>.
Can you please elaborate it more.
On 20 Jun 2015 2:46 pm, "SF Hadoop" <sf...@gmail.com> wrote:
> Really depends on your requirements for the format of the data.
>
> The easiest way I can think of is to "stream" batches of data into a pub
> sub system that the target system can access and then consume.
>
> Verify each batch and then ditch them.
>
> You can throttle the size of the intermediary infrastructure based on your
> batches.
>
> Seems the most efficient approach.
>
> On Thursday, June 18, 2015, Divya Gehlot <di...@gmail.com> wrote:
>
>> Hi,
>> I need to copy data from first hadoop cluster to second hadoop cluster.
>> I cant access second hadoop cluster from first hadoop cluster due to some
>> security issue.
>> Can any point me how can I do apart from distcp command.
>> For instance
>> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
>> zone
>>
>>
>>
>> Thanks,
>> Divya
>>
>>
>>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by SF Hadoop <sf...@gmail.com>.
Really depends on your requirements for the format of the data.
The easiest way I can think of is to "stream" batches of data into a pub
sub system that the target system can access and then consume.
Verify each batch and then ditch them.
You can throttle the size of the intermediary infrastructure based on your
batches.
Seems the most efficient approach.
On Thursday, June 18, 2015, Divya Gehlot <di...@gmail.com> wrote:
> Hi,
> I need to copy data from first hadoop cluster to second hadoop cluster.
> I cant access second hadoop cluster from first hadoop cluster due to some
> security issue.
> Can any point me how can I do apart from distcp command.
> For instance
> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
> zone
>
>
>
> Thanks,
> Divya
>
>
>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Nitin Pawar <ni...@gmail.com>.
What's the size of the data?
If you can not do distcp between clusters then other way is doing hdfs get
on the data and then hdfs put on another cluster
On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
> Hi,
> I need to copy data from first hadoop cluster to second hadoop cluster.
> I cant access second hadoop cluster from first hadoop cluster due to some
> security issue.
> Can any point me how can I do apart from distcp command.
> For instance
> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
> zone
>
>
>
> Thanks,
> Divya
>
>
>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Nitin Pawar <ni...@gmail.com>.
What's the size of the data?
If you can not do distcp between clusters then other way is doing hdfs get
on the data and then hdfs put on another cluster
On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
> Hi,
> I need to copy data from first hadoop cluster to second hadoop cluster.
> I cant access second hadoop cluster from first hadoop cluster due to some
> security issue.
> Can any point me how can I do apart from distcp command.
> For instance
> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
> zone
>
>
>
> Thanks,
> Divya
>
>
>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by SF Hadoop <sf...@gmail.com>.
Really depends on your requirements for the format of the data.
The easiest way I can think of is to "stream" batches of data into a pub
sub system that the target system can access and then consume.
Verify each batch and then ditch them.
You can throttle the size of the intermediary infrastructure based on your
batches.
Seems the most efficient approach.
On Thursday, June 18, 2015, Divya Gehlot <di...@gmail.com> wrote:
> Hi,
> I need to copy data from first hadoop cluster to second hadoop cluster.
> I cant access second hadoop cluster from first hadoop cluster due to some
> security issue.
> Can any point me how can I do apart from distcp command.
> For instance
> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
> zone
>
>
>
> Thanks,
> Divya
>
>
>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by SF Hadoop <sf...@gmail.com>.
Really depends on your requirements for the format of the data.
The easiest way I can think of is to "stream" batches of data into a pub
sub system that the target system can access and then consume.
Verify each batch and then ditch them.
You can throttle the size of the intermediary infrastructure based on your
batches.
Seems the most efficient approach.
On Thursday, June 18, 2015, Divya Gehlot <di...@gmail.com> wrote:
> Hi,
> I need to copy data from first hadoop cluster to second hadoop cluster.
> I cant access second hadoop cluster from first hadoop cluster due to some
> security issue.
> Can any point me how can I do apart from distcp command.
> For instance
> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
> zone
>
>
>
> Thanks,
> Divya
>
>
>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by Nitin Pawar <ni...@gmail.com>.
What's the size of the data?
If you can not do distcp between clusters then other way is doing hdfs get
on the data and then hdfs put on another cluster
On 19-Jun-2015 9:56 am, "Divya Gehlot" <di...@gmail.com> wrote:
> Hi,
> I need to copy data from first hadoop cluster to second hadoop cluster.
> I cant access second hadoop cluster from first hadoop cluster due to some
> security issue.
> Can any point me how can I do apart from distcp command.
> For instance
> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
> zone
>
>
>
> Thanks,
> Divya
>
>
>
Re: copy data from one hadoop cluster to another hadoop cluster +
cant use distcp
Posted by SF Hadoop <sf...@gmail.com>.
Really depends on your requirements for the format of the data.
The easiest way I can think of is to "stream" batches of data into a pub
sub system that the target system can access and then consume.
Verify each batch and then ditch them.
You can throttle the size of the intermediary infrastructure based on your
batches.
Seems the most efficient approach.
On Thursday, June 18, 2015, Divya Gehlot <di...@gmail.com> wrote:
> Hi,
> I need to copy data from first hadoop cluster to second hadoop cluster.
> I cant access second hadoop cluster from first hadoop cluster due to some
> security issue.
> Can any point me how can I do apart from distcp command.
> For instance
> Cluster 1 secured zone -> copy hdfs data to -> cluster 2 in non secured
> zone
>
>
>
> Thanks,
> Divya
>
>
>