You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@whirr.apache.org by Alex Heneveld <al...@cloudsoftcorp.com> on 2012/05/09 10:39:47 UTC
whirr with hadoop 1.0.2 in aws ?
hi folks,
i'm trying to get the latest hadoop (1.0.2) deployed into aws. simply
setting "whirr.hadoop.version" *almost* does the trick.
however there now seems to be a different strategy to determine which
interface to bind to. hadoop is failing because trying to bind to its
external IP address ... which of course in AWS is not known at the
machine itself. see log below [2].
i'll poke around, there's probably a config parameter to tweak, but i
wondered if anyone had the solution at their fingertips. :)
if not, i'll report back.
btw my motivation for 1.0.2 is the hadoop-client dependency item [1]
which cuts the wars size by more than half. nice!
--a
[1] https://issues.apache.org/jira/browse/HADOOP-8009
[2] log:
2012-05-08 15:22:41,357 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host =
ip-10-58-231-212.eu-west-1.compute.internal/10.58.231.212
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 -r
1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012
************************************************************/
// ~15 lines removed
2012-05-08 15:22:41,916 FATAL org.apache.hadoop.mapred.JobTracker:
java.net.BindException: Problem binding to /79.125.41.39:8021 : Cannot
assign requested address
Re: whirr with hadoop 1.0.2 in aws ?
Posted by Andrei Savu <sa...@gmail.com>.
Thanks Alex! I will review and merge your changes later today.
On Thu, May 10, 2012 at 12:20 PM, Alex Heneveld <
alex.heneveld@cloudsoftcorp.com> wrote:
>
> thanks andrei, i've added patch to #459.
> working reliably for me now. :)
>
> --a
>
>
>
> On 10/05/2012 10:55, Andrei Savu wrote:
>
>
> On Thu, May 10, 2012 at 1:11 AM, Alex Heneveld <
> alex.heneveld@cloudsoftcorp.com> wrote:
>
>> i've got a patch which attempts to use the nodeMetadata.hostname if DNS
>> doesn't resolve, which i'm testing, so far so good.
>>
>> would this be useful to raise as an issue and submit?
>>
>
> That would be great! It's much more robust to get hostname information via
> API.
>
> Over the last few months we've had many DNS related issues:
>
> https://issues.apache.org/jira/browse/WHIRR-459
> https://issues.apache.org/jira/browse/WHIRR-577
> https://issues.apache.org/jira/browse/WHIRR-511
> https://issues.apache.org/jira/browse/WHIRR-492
>
> Note: unfortunately this will only work for ec2
>
>
>
Re: whirr with hadoop 1.0.2 in aws ?
Posted by Alex Heneveld <al...@cloudsoftcorp.com>.
thanks andrei, i've added patch to #459.
working reliably for me now. :)
--a
On 10/05/2012 10:55, Andrei Savu wrote:
>
> On Thu, May 10, 2012 at 1:11 AM, Alex Heneveld
> <alex.heneveld@cloudsoftcorp.com
> <ma...@cloudsoftcorp.com>> wrote:
>
> i've got a patch which attempts to use the nodeMetadata.hostname
> if DNS doesn't resolve, which i'm testing, so far so good.
>
> would this be useful to raise as an issue and submit?
>
>
> That would be great! It's much more robust to get hostname information
> via API.
>
> Over the last few months we've had many DNS related issues:
>
> https://issues.apache.org/jira/browse/WHIRR-459
> https://issues.apache.org/jira/browse/WHIRR-577
> https://issues.apache.org/jira/browse/WHIRR-511
> https://issues.apache.org/jira/browse/WHIRR-492
>
> Note: unfortunately this will only work for ec2
Re: whirr with hadoop 1.0.2 in aws ?
Posted by Andrei Savu <sa...@gmail.com>.
On Thu, May 10, 2012 at 1:11 AM, Alex Heneveld <
alex.heneveld@cloudsoftcorp.com> wrote:
> i've got a patch which attempts to use the nodeMetadata.hostname if DNS
> doesn't resolve, which i'm testing, so far so good.
>
> would this be useful to raise as an issue and submit?
>
That would be great! It's much more robust to get hostname information via
API.
Over the last few months we've had many DNS related issues:
https://issues.apache.org/jira/browse/WHIRR-459
https://issues.apache.org/jira/browse/WHIRR-577
https://issues.apache.org/jira/browse/WHIRR-511
https://issues.apache.org/jira/browse/WHIRR-492
Note: unfortunately this will only work for ec2
Re: whirr with hadoop 1.0.2 in aws ?
Posted by Alex Heneveld <al...@cloudsoftcorp.com>.
thanks. i've tracked it down a bit more ... looks like it isn't hadoop
but rather something in my ecosystem (random wifi hotspots perhaps?) not
doing reverse-dns lookup, so it falls back to the hostip as a string.
so i'd expect the same behaviour with the default version.
i've got a patch which attempts to use the nodeMetadata.hostname if DNS
doesn't resolve, which i'm testing, so far so good.
would this be useful to raise as an issue and submit?
--a
On 10/05/2012 00:44, Andrei Savu wrote:
>
> however there now seems to be a different strategy to determine
> which interface to bind to. hadoop is failing because trying to
> bind to its external IP address ... which of course in AWS is not
> known at the machine itself. see log below [2].
>
>
> That log file looks a bit strange to me because it starts by
> mentioning the private hostname and the private IP and later on it
> tries to bind to the public IP.
>
> i'll poke around, there's probably a config parameter to tweak,
> but i wondered if anyone had the solution at their fingertips. :)
> if not, i'll report back.
>
>
> I have no answer ready but it should be easy to understand what needs
> to be changed.
>
> btw my motivation for 1.0.2 is the hadoop-client dependency item
> [1] which cuts the wars size by more than half. nice!
>
>
> cool! (and finally)
>
>
> --a
>
>
> [1] https://issues.apache.org/jira/browse/HADOOP-8009
>
> [2] log:
>
> 2012-05-08 15:22:41,357 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host =
> ip-10-58-231-212.eu-west-1.compute.internal/10.58.231.212
> <http://10.58.231.212>
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2
> -r 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012
> ************************************************************/
> // ~15 lines removed
> 2012-05-08 15:22:41,916 FATAL org.apache.hadoop.mapred.JobTracker:
> java.net.BindException: Problem binding to /79.125.41.39:8021
> <http://79.125.41.39:8021> : Cannot assign requested address
>
>
>
Re: whirr with hadoop 1.0.2 in aws ?
Posted by Andrei Savu <sa...@gmail.com>.
>
> however there now seems to be a different strategy to determine which
> interface to bind to. hadoop is failing because trying to bind to its
> external IP address ... which of course in AWS is not known at the machine
> itself. see log below [2].
>
That log file looks a bit strange to me because it starts by mentioning the
private hostname and the private IP and later on it tries to bind to the
public IP.
> i'll poke around, there's probably a config parameter to tweak, but i
> wondered if anyone had the solution at their fingertips. :)
> if not, i'll report back.
>
I have no answer ready but it should be easy to understand what needs to be
changed.
> btw my motivation for 1.0.2 is the hadoop-client dependency item [1] which
> cuts the wars size by more than half. nice!
>
cool! (and finally)
>
> --a
>
>
> [1] https://issues.apache.org/**jira/browse/HADOOP-8009<https://issues.apache.org/jira/browse/HADOOP-8009>
>
> [2] log:
>
> 2012-05-08 15:22:41,357 INFO org.apache.hadoop.mapred.**JobTracker:
> STARTUP_MSG:
> /****************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host = ip-10-58-231-212.eu-west-1.**compute.internal/
> 10.58.231.212
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.2
> STARTUP_MSG: build = https://svn.apache.org/repos/**
> asf/hadoop/common/branches/**branch-1.0.2<https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2>-r 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012
> ****************************************************************/
> // ~15 lines removed
> 2012-05-08 15:22:41,916 FATAL org.apache.hadoop.mapred.**JobTracker:
> java.net.BindException: Problem binding to /79.125.41.39:8021 : Cannot
> assign requested address
>
>
>