You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by samir das mohapatra <sa...@gmail.com> on 2013/05/28 13:47:37 UTC
Pulling data from secured hadoop cluster to another hadoop cluster
Hi All,
I could able to connect the hadoop (source ) cluster after ssh is
established.
But i wanted to know, If I want to pull some data using distcp from source
secured hadoop box to another hadoop cluster , I could not able to ping
name node machine. In this approach how to run distcp command from target
cluster in with secured connection.
Source: hadoop.server1 (ssh secured)
Target: hadoop.server2 (runing distcp here)
running command:
distcp hftp://hadoop.server1:50070/dataSet
hdfs://hadoop.server2:54310/targetDataSet
Regards,
samir.
Re: Pulling data from secured hadoop cluster to another hadoop cluster
Posted by samir das mohapatra <sa...@gmail.com>.
it is not hadoop security issue, the security is in host , I Mean to say in
network level.
I could not able to ping bcz source system is designed such a way that
only you can connecto through ssh .
If this is the case how to over come this problem.
What extra parameter i need to add i ssh level so that i could able to ping
the machine. All the servers are in same domain.
On Tue, May 28, 2013 at 7:35 PM, Shahab Yunus <sh...@gmail.com>wrote:
> Also Samir, when you say 'secured', by any chance that cluster is secured
> with Kerberos (rather than ssh)?
>
> -Shahab
>
>
> On Tue, May 28, 2013 at 8:29 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> hadoop daemons do not use ssh to communicate.
>>
>> if your distcp job could not connect to remote server then either the
>> connection was rejected by the target namenode or the it was not able to
>> establish the network connection.
>>
>> were you able to see the hdfs on server1 from server2?
>>
>>
>> On Tue, May 28, 2013 at 5:17 PM, samir das mohapatra <
>> samir.helpdoc@gmail.com> wrote:
>>
>>> Hi All,
>>> I could able to connect the hadoop (source ) cluster after ssh is
>>> established.
>>>
>>> But i wanted to know, If I want to pull some data using distcp from
>>> source secured hadoop box to another hadoop cluster , I could not able to
>>> ping name node machine. In this approach how to run distcp command from
>>> target cluster in with secured connection.
>>>
>>> Source: hadoop.server1 (ssh secured)
>>> Target: hadoop.server2 (runing distcp here)
>>>
>>>
>>> running command:
>>>
>>> distcp hftp://hadoop.server1:50070/dataSet
>>> hdfs://hadoop.server2:54310/targetDataSet
>>>
>>>
>>> Regards,
>>> samir.
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
Re: Pulling data from secured hadoop cluster to another hadoop cluster
Posted by samir das mohapatra <sa...@gmail.com>.
it is not hadoop security issue, the security is in host , I Mean to say in
network level.
I could not able to ping bcz source system is designed such a way that
only you can connecto through ssh .
If this is the case how to over come this problem.
What extra parameter i need to add i ssh level so that i could able to ping
the machine. All the servers are in same domain.
On Tue, May 28, 2013 at 7:35 PM, Shahab Yunus <sh...@gmail.com>wrote:
> Also Samir, when you say 'secured', by any chance that cluster is secured
> with Kerberos (rather than ssh)?
>
> -Shahab
>
>
> On Tue, May 28, 2013 at 8:29 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> hadoop daemons do not use ssh to communicate.
>>
>> if your distcp job could not connect to remote server then either the
>> connection was rejected by the target namenode or the it was not able to
>> establish the network connection.
>>
>> were you able to see the hdfs on server1 from server2?
>>
>>
>> On Tue, May 28, 2013 at 5:17 PM, samir das mohapatra <
>> samir.helpdoc@gmail.com> wrote:
>>
>>> Hi All,
>>> I could able to connect the hadoop (source ) cluster after ssh is
>>> established.
>>>
>>> But i wanted to know, If I want to pull some data using distcp from
>>> source secured hadoop box to another hadoop cluster , I could not able to
>>> ping name node machine. In this approach how to run distcp command from
>>> target cluster in with secured connection.
>>>
>>> Source: hadoop.server1 (ssh secured)
>>> Target: hadoop.server2 (runing distcp here)
>>>
>>>
>>> running command:
>>>
>>> distcp hftp://hadoop.server1:50070/dataSet
>>> hdfs://hadoop.server2:54310/targetDataSet
>>>
>>>
>>> Regards,
>>> samir.
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
Re: Pulling data from secured hadoop cluster to another hadoop cluster
Posted by samir das mohapatra <sa...@gmail.com>.
it is not hadoop security issue, the security is in host , I Mean to say in
network level.
I could not able to ping bcz source system is designed such a way that
only you can connecto through ssh .
If this is the case how to over come this problem.
What extra parameter i need to add i ssh level so that i could able to ping
the machine. All the servers are in same domain.
On Tue, May 28, 2013 at 7:35 PM, Shahab Yunus <sh...@gmail.com>wrote:
> Also Samir, when you say 'secured', by any chance that cluster is secured
> with Kerberos (rather than ssh)?
>
> -Shahab
>
>
> On Tue, May 28, 2013 at 8:29 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> hadoop daemons do not use ssh to communicate.
>>
>> if your distcp job could not connect to remote server then either the
>> connection was rejected by the target namenode or the it was not able to
>> establish the network connection.
>>
>> were you able to see the hdfs on server1 from server2?
>>
>>
>> On Tue, May 28, 2013 at 5:17 PM, samir das mohapatra <
>> samir.helpdoc@gmail.com> wrote:
>>
>>> Hi All,
>>> I could able to connect the hadoop (source ) cluster after ssh is
>>> established.
>>>
>>> But i wanted to know, If I want to pull some data using distcp from
>>> source secured hadoop box to another hadoop cluster , I could not able to
>>> ping name node machine. In this approach how to run distcp command from
>>> target cluster in with secured connection.
>>>
>>> Source: hadoop.server1 (ssh secured)
>>> Target: hadoop.server2 (runing distcp here)
>>>
>>>
>>> running command:
>>>
>>> distcp hftp://hadoop.server1:50070/dataSet
>>> hdfs://hadoop.server2:54310/targetDataSet
>>>
>>>
>>> Regards,
>>> samir.
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
Re: Pulling data from secured hadoop cluster to another hadoop cluster
Posted by samir das mohapatra <sa...@gmail.com>.
it is not hadoop security issue, the security is in host , I Mean to say in
network level.
I could not able to ping bcz source system is designed such a way that
only you can connecto through ssh .
If this is the case how to over come this problem.
What extra parameter i need to add i ssh level so that i could able to ping
the machine. All the servers are in same domain.
On Tue, May 28, 2013 at 7:35 PM, Shahab Yunus <sh...@gmail.com>wrote:
> Also Samir, when you say 'secured', by any chance that cluster is secured
> with Kerberos (rather than ssh)?
>
> -Shahab
>
>
> On Tue, May 28, 2013 at 8:29 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> hadoop daemons do not use ssh to communicate.
>>
>> if your distcp job could not connect to remote server then either the
>> connection was rejected by the target namenode or the it was not able to
>> establish the network connection.
>>
>> were you able to see the hdfs on server1 from server2?
>>
>>
>> On Tue, May 28, 2013 at 5:17 PM, samir das mohapatra <
>> samir.helpdoc@gmail.com> wrote:
>>
>>> Hi All,
>>> I could able to connect the hadoop (source ) cluster after ssh is
>>> established.
>>>
>>> But i wanted to know, If I want to pull some data using distcp from
>>> source secured hadoop box to another hadoop cluster , I could not able to
>>> ping name node machine. In this approach how to run distcp command from
>>> target cluster in with secured connection.
>>>
>>> Source: hadoop.server1 (ssh secured)
>>> Target: hadoop.server2 (runing distcp here)
>>>
>>>
>>> running command:
>>>
>>> distcp hftp://hadoop.server1:50070/dataSet
>>> hdfs://hadoop.server2:54310/targetDataSet
>>>
>>>
>>> Regards,
>>> samir.
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
Re: Pulling data from secured hadoop cluster to another hadoop cluster
Posted by Shahab Yunus <sh...@gmail.com>.
Also Samir, when you say 'secured', by any chance that cluster is secured
with Kerberos (rather than ssh)?
-Shahab
On Tue, May 28, 2013 at 8:29 AM, Nitin Pawar <ni...@gmail.com>wrote:
> hadoop daemons do not use ssh to communicate.
>
> if your distcp job could not connect to remote server then either the
> connection was rejected by the target namenode or the it was not able to
> establish the network connection.
>
> were you able to see the hdfs on server1 from server2?
>
>
> On Tue, May 28, 2013 at 5:17 PM, samir das mohapatra <
> samir.helpdoc@gmail.com> wrote:
>
>> Hi All,
>> I could able to connect the hadoop (source ) cluster after ssh is
>> established.
>>
>> But i wanted to know, If I want to pull some data using distcp from
>> source secured hadoop box to another hadoop cluster , I could not able to
>> ping name node machine. In this approach how to run distcp command from
>> target cluster in with secured connection.
>>
>> Source: hadoop.server1 (ssh secured)
>> Target: hadoop.server2 (runing distcp here)
>>
>>
>> running command:
>>
>> distcp hftp://hadoop.server1:50070/dataSet
>> hdfs://hadoop.server2:54310/targetDataSet
>>
>>
>> Regards,
>> samir.
>>
>
>
>
> --
> Nitin Pawar
>
Re: Pulling data from secured hadoop cluster to another hadoop cluster
Posted by Shahab Yunus <sh...@gmail.com>.
Also Samir, when you say 'secured', by any chance that cluster is secured
with Kerberos (rather than ssh)?
-Shahab
On Tue, May 28, 2013 at 8:29 AM, Nitin Pawar <ni...@gmail.com>wrote:
> hadoop daemons do not use ssh to communicate.
>
> if your distcp job could not connect to remote server then either the
> connection was rejected by the target namenode or the it was not able to
> establish the network connection.
>
> were you able to see the hdfs on server1 from server2?
>
>
> On Tue, May 28, 2013 at 5:17 PM, samir das mohapatra <
> samir.helpdoc@gmail.com> wrote:
>
>> Hi All,
>> I could able to connect the hadoop (source ) cluster after ssh is
>> established.
>>
>> But i wanted to know, If I want to pull some data using distcp from
>> source secured hadoop box to another hadoop cluster , I could not able to
>> ping name node machine. In this approach how to run distcp command from
>> target cluster in with secured connection.
>>
>> Source: hadoop.server1 (ssh secured)
>> Target: hadoop.server2 (runing distcp here)
>>
>>
>> running command:
>>
>> distcp hftp://hadoop.server1:50070/dataSet
>> hdfs://hadoop.server2:54310/targetDataSet
>>
>>
>> Regards,
>> samir.
>>
>
>
>
> --
> Nitin Pawar
>
Re: Pulling data from secured hadoop cluster to another hadoop cluster
Posted by Shahab Yunus <sh...@gmail.com>.
Also Samir, when you say 'secured', by any chance that cluster is secured
with Kerberos (rather than ssh)?
-Shahab
On Tue, May 28, 2013 at 8:29 AM, Nitin Pawar <ni...@gmail.com>wrote:
> hadoop daemons do not use ssh to communicate.
>
> if your distcp job could not connect to remote server then either the
> connection was rejected by the target namenode or the it was not able to
> establish the network connection.
>
> were you able to see the hdfs on server1 from server2?
>
>
> On Tue, May 28, 2013 at 5:17 PM, samir das mohapatra <
> samir.helpdoc@gmail.com> wrote:
>
>> Hi All,
>> I could able to connect the hadoop (source ) cluster after ssh is
>> established.
>>
>> But i wanted to know, If I want to pull some data using distcp from
>> source secured hadoop box to another hadoop cluster , I could not able to
>> ping name node machine. In this approach how to run distcp command from
>> target cluster in with secured connection.
>>
>> Source: hadoop.server1 (ssh secured)
>> Target: hadoop.server2 (runing distcp here)
>>
>>
>> running command:
>>
>> distcp hftp://hadoop.server1:50070/dataSet
>> hdfs://hadoop.server2:54310/targetDataSet
>>
>>
>> Regards,
>> samir.
>>
>
>
>
> --
> Nitin Pawar
>
Re: Pulling data from secured hadoop cluster to another hadoop cluster
Posted by Shahab Yunus <sh...@gmail.com>.
Also Samir, when you say 'secured', by any chance that cluster is secured
with Kerberos (rather than ssh)?
-Shahab
On Tue, May 28, 2013 at 8:29 AM, Nitin Pawar <ni...@gmail.com>wrote:
> hadoop daemons do not use ssh to communicate.
>
> if your distcp job could not connect to remote server then either the
> connection was rejected by the target namenode or the it was not able to
> establish the network connection.
>
> were you able to see the hdfs on server1 from server2?
>
>
> On Tue, May 28, 2013 at 5:17 PM, samir das mohapatra <
> samir.helpdoc@gmail.com> wrote:
>
>> Hi All,
>> I could able to connect the hadoop (source ) cluster after ssh is
>> established.
>>
>> But i wanted to know, If I want to pull some data using distcp from
>> source secured hadoop box to another hadoop cluster , I could not able to
>> ping name node machine. In this approach how to run distcp command from
>> target cluster in with secured connection.
>>
>> Source: hadoop.server1 (ssh secured)
>> Target: hadoop.server2 (runing distcp here)
>>
>>
>> running command:
>>
>> distcp hftp://hadoop.server1:50070/dataSet
>> hdfs://hadoop.server2:54310/targetDataSet
>>
>>
>> Regards,
>> samir.
>>
>
>
>
> --
> Nitin Pawar
>
Re: Pulling data from secured hadoop cluster to another hadoop cluster
Posted by Nitin Pawar <ni...@gmail.com>.
hadoop daemons do not use ssh to communicate.
if your distcp job could not connect to remote server then either the
connection was rejected by the target namenode or the it was not able to
establish the network connection.
were you able to see the hdfs on server1 from server2?
On Tue, May 28, 2013 at 5:17 PM, samir das mohapatra <
samir.helpdoc@gmail.com> wrote:
> Hi All,
> I could able to connect the hadoop (source ) cluster after ssh is
> established.
>
> But i wanted to know, If I want to pull some data using distcp from source
> secured hadoop box to another hadoop cluster , I could not able to ping
> name node machine. In this approach how to run distcp command from target
> cluster in with secured connection.
>
> Source: hadoop.server1 (ssh secured)
> Target: hadoop.server2 (runing distcp here)
>
>
> running command:
>
> distcp hftp://hadoop.server1:50070/dataSet
> hdfs://hadoop.server2:54310/targetDataSet
>
>
> Regards,
> samir.
>
--
Nitin Pawar
Re: Pulling data from secured hadoop cluster to another hadoop cluster
Posted by Nitin Pawar <ni...@gmail.com>.
hadoop daemons do not use ssh to communicate.
if your distcp job could not connect to remote server then either the
connection was rejected by the target namenode or the it was not able to
establish the network connection.
were you able to see the hdfs on server1 from server2?
On Tue, May 28, 2013 at 5:17 PM, samir das mohapatra <
samir.helpdoc@gmail.com> wrote:
> Hi All,
> I could able to connect the hadoop (source ) cluster after ssh is
> established.
>
> But i wanted to know, If I want to pull some data using distcp from source
> secured hadoop box to another hadoop cluster , I could not able to ping
> name node machine. In this approach how to run distcp command from target
> cluster in with secured connection.
>
> Source: hadoop.server1 (ssh secured)
> Target: hadoop.server2 (runing distcp here)
>
>
> running command:
>
> distcp hftp://hadoop.server1:50070/dataSet
> hdfs://hadoop.server2:54310/targetDataSet
>
>
> Regards,
> samir.
>
--
Nitin Pawar
Re: Pulling data from secured hadoop cluster to another hadoop cluster
Posted by Nitin Pawar <ni...@gmail.com>.
hadoop daemons do not use ssh to communicate.
if your distcp job could not connect to remote server then either the
connection was rejected by the target namenode or the it was not able to
establish the network connection.
were you able to see the hdfs on server1 from server2?
On Tue, May 28, 2013 at 5:17 PM, samir das mohapatra <
samir.helpdoc@gmail.com> wrote:
> Hi All,
> I could able to connect the hadoop (source ) cluster after ssh is
> established.
>
> But i wanted to know, If I want to pull some data using distcp from source
> secured hadoop box to another hadoop cluster , I could not able to ping
> name node machine. In this approach how to run distcp command from target
> cluster in with secured connection.
>
> Source: hadoop.server1 (ssh secured)
> Target: hadoop.server2 (runing distcp here)
>
>
> running command:
>
> distcp hftp://hadoop.server1:50070/dataSet
> hdfs://hadoop.server2:54310/targetDataSet
>
>
> Regards,
> samir.
>
--
Nitin Pawar
Re: Pulling data from secured hadoop cluster to another hadoop cluster
Posted by Nitin Pawar <ni...@gmail.com>.
hadoop daemons do not use ssh to communicate.
if your distcp job could not connect to remote server then either the
connection was rejected by the target namenode or the it was not able to
establish the network connection.
were you able to see the hdfs on server1 from server2?
On Tue, May 28, 2013 at 5:17 PM, samir das mohapatra <
samir.helpdoc@gmail.com> wrote:
> Hi All,
> I could able to connect the hadoop (source ) cluster after ssh is
> established.
>
> But i wanted to know, If I want to pull some data using distcp from source
> secured hadoop box to another hadoop cluster , I could not able to ping
> name node machine. In this approach how to run distcp command from target
> cluster in with secured connection.
>
> Source: hadoop.server1 (ssh secured)
> Target: hadoop.server2 (runing distcp here)
>
>
> running command:
>
> distcp hftp://hadoop.server1:50070/dataSet
> hdfs://hadoop.server2:54310/targetDataSet
>
>
> Regards,
> samir.
>
--
Nitin Pawar