You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Arindam Choudhury <ar...@gmail.com> on 2012/05/11 15:45:19 UTC
transferring between HDFS which reside in different subnet
Hi,
I have a question to the hadoop experts:
I have two HDFS, in different subnet.
HDFS1 : 192.168.*.*
HDFS2: 10.10.*.*
the namenode of HDFS2 has two NIC. One connected to 192.168.*.* and another
to 10.10.*.*.
So, is it possible to transfer data from HDFS1 to HDFS2 and vice versa.
Regards,
Arindam
Re: transferring between HDFS which reside in different subnet
Posted by Arindam Choudhury <ar...@gmail.com>.
So,
hadoop dfs -cp hdfs://.... hdfs://...
this will work.
On Fri, May 11, 2012 at 4:14 PM, Rajesh Sai T <ts...@gmail.com> wrote:
> Looks like both are private subnets, so you got to route via a public
> default gateway. Try adding route using route command if your in
> linux(windows i have no idea). Just a thought i havent tried it though.
>
> Thanks,
> Rajesh
>
> Typed from mobile, please bear with typos.
> On May 11, 2012 10:03 AM, "Arindam Choudhury" <arindamchoudhury0@gmail.com
> >
> wrote:
>
> > I can not cross access HDFS. Though HDFS2 has two NIC the HDFS is running
> > on the other subnet.
> >
> > On Fri, May 11, 2012 at 3:57 PM, Shi Yu <sh...@uchicago.edu> wrote:
> >
> > > If you could cross-access HDFS from both name nodes, then it should be
> > > transferable using /distcp /command.
> > >
> > > Shi *
> > > *
> > >
> > > On 5/11/2012 8:45 AM, Arindam Choudhury wrote:
> > >
> > >> Hi,
> > >>
> > >> I have a question to the hadoop experts:
> > >>
> > >> I have two HDFS, in different subnet.
> > >>
> > >> HDFS1 : 192.168.*.*
> > >> HDFS2: 10.10.*.*
> > >>
> > >> the namenode of HDFS2 has two NIC. One connected to 192.168.*.* and
> > >> another
> > >> to 10.10.*.*.
> > >>
> > >> So, is it possible to transfer data from HDFS1 to HDFS2 and vice
> versa.
> > >>
> > >> Regards,
> > >> Arindam
> > >>
> > >>
> > >
> >
>
Re: transferring between HDFS which reside in different subnet
Posted by Rajesh Sai T <ts...@gmail.com>.
Looks like both are private subnets, so you got to route via a public
default gateway. Try adding route using route command if your in
linux(windows i have no idea). Just a thought i havent tried it though.
Thanks,
Rajesh
Typed from mobile, please bear with typos.
On May 11, 2012 10:03 AM, "Arindam Choudhury" <ar...@gmail.com>
wrote:
> I can not cross access HDFS. Though HDFS2 has two NIC the HDFS is running
> on the other subnet.
>
> On Fri, May 11, 2012 at 3:57 PM, Shi Yu <sh...@uchicago.edu> wrote:
>
> > If you could cross-access HDFS from both name nodes, then it should be
> > transferable using /distcp /command.
> >
> > Shi *
> > *
> >
> > On 5/11/2012 8:45 AM, Arindam Choudhury wrote:
> >
> >> Hi,
> >>
> >> I have a question to the hadoop experts:
> >>
> >> I have two HDFS, in different subnet.
> >>
> >> HDFS1 : 192.168.*.*
> >> HDFS2: 10.10.*.*
> >>
> >> the namenode of HDFS2 has two NIC. One connected to 192.168.*.* and
> >> another
> >> to 10.10.*.*.
> >>
> >> So, is it possible to transfer data from HDFS1 to HDFS2 and vice versa.
> >>
> >> Regards,
> >> Arindam
> >>
> >>
> >
>
Re: transferring between HDFS which reside in different subnet
Posted by Shi Yu <sh...@uchicago.edu>.
It seems in your case HDFS2 could access HDFS, so you should be able to
transfer HDFS data to HDFS2.
If you want to cross-transfer, you don't need to do distcp on cluster
nodes, if any client node (not necessary to be namenode, datanode,
secondary node, etc.) could access to both HDFSs, then run transfer
command on that client node.
On 5/11/2012 9:03 AM, Arindam Choudhury wrote:
> I can not cross access HDFS. Though HDFS2 has two NIC the HDFS is running
> on the other subnet.
>
> On Fri, May 11, 2012 at 3:57 PM, Shi Yu<sh...@uchicago.edu> wrote:
>
>> If you could cross-access HDFS from both name nodes, then it should be
>> transferable using /distcp /command.
>>
>> Shi *
>> *
>>
>> On 5/11/2012 8:45 AM, Arindam Choudhury wrote:
>>
>>> Hi,
>>>
>>> I have a question to the hadoop experts:
>>>
>>> I have two HDFS, in different subnet.
>>>
>>> HDFS1 : 192.168.*.*
>>> HDFS2: 10.10.*.*
>>>
>>> the namenode of HDFS2 has two NIC. One connected to 192.168.*.* and
>>> another
>>> to 10.10.*.*.
>>>
>>> So, is it possible to transfer data from HDFS1 to HDFS2 and vice versa.
>>>
>>> Regards,
>>> Arindam
>>>
>>>
Re: transferring between HDFS which reside in different subnet
Posted by Arindam Choudhury <ar...@gmail.com>.
I can not cross access HDFS. Though HDFS2 has two NIC the HDFS is running
on the other subnet.
On Fri, May 11, 2012 at 3:57 PM, Shi Yu <sh...@uchicago.edu> wrote:
> If you could cross-access HDFS from both name nodes, then it should be
> transferable using /distcp /command.
>
> Shi *
> *
>
> On 5/11/2012 8:45 AM, Arindam Choudhury wrote:
>
>> Hi,
>>
>> I have a question to the hadoop experts:
>>
>> I have two HDFS, in different subnet.
>>
>> HDFS1 : 192.168.*.*
>> HDFS2: 10.10.*.*
>>
>> the namenode of HDFS2 has two NIC. One connected to 192.168.*.* and
>> another
>> to 10.10.*.*.
>>
>> So, is it possible to transfer data from HDFS1 to HDFS2 and vice versa.
>>
>> Regards,
>> Arindam
>>
>>
>
Re: transferring between HDFS which reside in different subnet
Posted by Shi Yu <sh...@uchicago.edu>.
If you could cross-access HDFS from both name nodes, then it should be
transferable using /distcp /command.
Shi *
*
On 5/11/2012 8:45 AM, Arindam Choudhury wrote:
> Hi,
>
> I have a question to the hadoop experts:
>
> I have two HDFS, in different subnet.
>
> HDFS1 : 192.168.*.*
> HDFS2: 10.10.*.*
>
> the namenode of HDFS2 has two NIC. One connected to 192.168.*.* and another
> to 10.10.*.*.
>
> So, is it possible to transfer data from HDFS1 to HDFS2 and vice versa.
>
> Regards,
> Arindam
>