You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@whirr.apache.org by Doug Daniels <dd...@mortardata.com> on 2011/06/02 07:02:09 UTC

trouble reaching whirr hadoop cluster from ec2 instance

Hi,

Using whirr 0.4.0 I'm able to start up and communicate with a hadoop cluster from my local machine, but I'm having trouble doing so from an ec2 instance.

Everything seems to start fine, but after I start the hadoop-proxy.sh I get this error message when I try to do 'hadoop fs -ls /':

ubuntu@domU-12-31-39-04-1E-48:~$ hadoop fs -ls /
11/06/02 04:57:16 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Bad connection to FS. command aborted. exception: Call to ec2-174-129-68-49.compute-1.amazonaws.com/10.192.214.192:8020 failed on local exception: java.io.EOFException
ubuntu@domU-12-31-39-04-1E-48:~$ 

I can ssh directly to the machines, but can't seem to communicate using hadoop.

Does anyone have any ideas what might be happening there?

Thanks,
Doug

Re: trouble reaching whirr hadoop cluster from ec2 instance

Posted by Doug Daniels <dd...@mortardata.com>.

Figured it out — this turned out to be because I had a different version of hadoop on the instance where I was running whirr than on the machines booted up in the cluster.

On Jun 2, 2011, at 10:30 AM, Doug Daniels wrote:

> Thanks Andrei, but I'm still having the same problem when I replace the ec2-174-129-68-49.compute-1.amazonaws.com with the public IP address.  Now I get:
> 
> ubuntu@domU-12-31-39-00-85-D1:~$ hadoop fs -ls /
> 11/06/02 14:23:06 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
> Bad connection to FS. command aborted. exception: Call to /50.16.169.253:8020 failed on local exception: java.io.EOFException
> 
> My situation is actually a bit different than you described.  I was not trying to reuse the same cluster from my local machine, but rather had booted up a new cluster directly from the ec2 machine.  Using the same configuration I used on my local machine, I get the behavior above.
> 
> I think you're right that it has something to do with the public/private IP, but I'm not sure exactly how.
> 
> Thanks,
> Doug
> 
> On Jun 2, 2011, at 6:07 AM, Andrei Savu wrote:
> 
>> Hi Daniels,
>> 
>> Let me see if I understand. You are starting a Hadoop cluster using
>> Whirr 0.4.0 from your local machine and you want to use it from a
>> different ec2 instance running in another security group.
>> 
>> The issue you are seeing is probably related to how DNS resolution
>> works inside and outside the Amazon network (e.g.
>> ec2-174-129-68-49.compute-1.amazonaws.com resolves to the public IP
>> outside the Amazon network and to the private IP inside).
>> 
>> I've done some testing by creating a similar environment and it seems
>> like the easiest workaround is to replace the hostname with the public
>> IP in hadoop-site.xml (the one generate on the local machine) and
>> updating hadoop-proxy.sh in a similar fashion.
>> 
>> I am not sure but I think we should consider to update the code to
>> replace the hostnames with IPs by default.
>> 
>> Let me know if you need more help with this one.
>> 
>> Cheers,
>> 
>> -- Andrei Savu / andreisavu.ro
>> 
>> On Thu, Jun 2, 2011 at 8:02 AM, Doug Daniels <dd...@mortardata.com> wrote:
>>> Hi,
>>> 
>>> Using whirr 0.4.0 I'm able to start up and communicate with a hadoop cluster from my local machine, but I'm having trouble doing so from an ec2 instance.
>>> 
>>> Everything seems to start fine, but after I start the hadoop-proxy.sh I get this error message when I try to do 'hadoop fs -ls /':
>>> 
>>> ubuntu@domU-12-31-39-04-1E-48:~$ hadoop fs -ls /
>>> 11/06/02 04:57:16 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
>>> Bad connection to FS. command aborted. exception: Call to ec2-174-129-68-49.compute-1.amazonaws.com/10.192.214.192:8020 failed on local exception: java.io.EOFException
>>> ubuntu@domU-12-31-39-04-1E-48:~$
>>> 
>>> I can ssh directly to the machines, but can't seem to communicate using hadoop.
>>> 
>>> Does anyone have any ideas what might be happening there?
>>> 
>>> Thanks,
>>> Doug
>

Re: trouble reaching whirr hadoop cluster from ec2 instance

Posted by Doug Daniels <dd...@mortardata.com>.

Thanks Andrei, but I'm still having the same problem when I replace the ec2-174-129-68-49.compute-1.amazonaws.com with the public IP address.  Now I get:

ubuntu@domU-12-31-39-00-85-D1:~$ hadoop fs -ls /
11/06/02 14:23:06 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Bad connection to FS. command aborted. exception: Call to /50.16.169.253:8020 failed on local exception: java.io.EOFException

My situation is actually a bit different than you described.  I was not trying to reuse the same cluster from my local machine, but rather had booted up a new cluster directly from the ec2 machine.  Using the same configuration I used on my local machine, I get the behavior above.

I think you're right that it has something to do with the public/private IP, but I'm not sure exactly how.

Thanks,
Doug

On Jun 2, 2011, at 6:07 AM, Andrei Savu wrote:

> Hi Daniels,
> 
> Let me see if I understand. You are starting a Hadoop cluster using
> Whirr 0.4.0 from your local machine and you want to use it from a
> different ec2 instance running in another security group.
> 
> The issue you are seeing is probably related to how DNS resolution
> works inside and outside the Amazon network (e.g.
> ec2-174-129-68-49.compute-1.amazonaws.com resolves to the public IP
> outside the Amazon network and to the private IP inside).
> 
> I've done some testing by creating a similar environment and it seems
> like the easiest workaround is to replace the hostname with the public
> IP in hadoop-site.xml (the one generate on the local machine) and
> updating hadoop-proxy.sh in a similar fashion.
> 
> I am not sure but I think we should consider to update the code to
> replace the hostnames with IPs by default.
> 
> Let me know if you need more help with this one.
> 
> Cheers,
> 
> -- Andrei Savu / andreisavu.ro
> 
> On Thu, Jun 2, 2011 at 8:02 AM, Doug Daniels <dd...@mortardata.com> wrote:
>> Hi,
>> 
>> Using whirr 0.4.0 I'm able to start up and communicate with a hadoop cluster from my local machine, but I'm having trouble doing so from an ec2 instance.
>> 
>> Everything seems to start fine, but after I start the hadoop-proxy.sh I get this error message when I try to do 'hadoop fs -ls /':
>> 
>> ubuntu@domU-12-31-39-04-1E-48:~$ hadoop fs -ls /
>> 11/06/02 04:57:16 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
>> Bad connection to FS. command aborted. exception: Call to ec2-174-129-68-49.compute-1.amazonaws.com/10.192.214.192:8020 failed on local exception: java.io.EOFException
>> ubuntu@domU-12-31-39-04-1E-48:~$
>> 
>> I can ssh directly to the machines, but can't seem to communicate using hadoop.
>> 
>> Does anyone have any ideas what might be happening there?
>> 
>> Thanks,
>> Doug

Re: trouble reaching whirr hadoop cluster from ec2 instance

Posted by Andrei Savu <sa...@gmail.com>.

Hi Daniels,

Let me see if I understand. You are starting a Hadoop cluster using
Whirr 0.4.0 from your local machine and you want to use it from a
different ec2 instance running in another security group.

The issue you are seeing is probably related to how DNS resolution
works inside and outside the Amazon network (e.g.
ec2-174-129-68-49.compute-1.amazonaws.com resolves to the public IP
outside the Amazon network and to the private IP inside).

I've done some testing by creating a similar environment and it seems
like the easiest workaround is to replace the hostname with the public
IP in hadoop-site.xml (the one generate on the local machine) and
updating hadoop-proxy.sh in a similar fashion.

I am not sure but I think we should consider to update the code to
replace the hostnames with IPs by default.

Let me know if you need more help with this one.

Cheers,

-- Andrei Savu / andreisavu.ro

On Thu, Jun 2, 2011 at 8:02 AM, Doug Daniels <dd...@mortardata.com> wrote:
> Hi,
>
> Using whirr 0.4.0 I'm able to start up and communicate with a hadoop cluster from my local machine, but I'm having trouble doing so from an ec2 instance.
>
> Everything seems to start fine, but after I start the hadoop-proxy.sh I get this error message when I try to do 'hadoop fs -ls /':
>
> ubuntu@domU-12-31-39-04-1E-48:~$ hadoop fs -ls /
> 11/06/02 04:57:16 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
> Bad connection to FS. command aborted. exception: Call to ec2-174-129-68-49.compute-1.amazonaws.com/10.192.214.192:8020 failed on local exception: java.io.EOFException
> ubuntu@domU-12-31-39-04-1E-48:~$
>
> I can ssh directly to the machines, but can't seem to communicate using hadoop.
>
> Does anyone have any ideas what might be happening there?
>
> Thanks,
> Doug