Hadoop EC2 clusters -- internal DNS for tasktracker/datanode UI hyperlinks?

[Evan Pollan <ev...@gmail.com>]

This is a minor annoyance, but I'm wondering if anyone has found a way
around this.

When whirr creates a hadoop cluster in EC2, I've found that the admin UI
(i.e. the web interface served up at <jobtracker>:50030, e.g.) uses the
public DNS names for all systems until you get down to the individual tasks
on a particular task tracker and want to download some or all of the logs
for a particular log.  At this point, the URLs are rendered using the
private DNS names and are not resolvable outside EC2.

Is there any way to get around this, other than pulling down the cluster
instance manifest and generating /etc/host entries all the private DNS
names pointing to each instance's public IPs?

Re: Hadoop EC2 clusters -- internal DNS for tasktracker/datanode UI hyperlinks?

[Andrei Savu <sa...@gmail.com>]

On Mon, Feb 13, 2012 at 8:27 PM, Evan Pollan <ev...@gmail.com> wrote:

> No worries.  This was more of a hadoop question than a whirr question.
>  I'm assuming all the datanode/tasktracker bits in the web UI are based on
> those services self-reporting their hostnames -- it'd be nice to be able to
> have them use their public DNS names, instead of the private ones.


I agree - unfortunately I think this is also a Hadoop issue - I'm not 100%
sure.

Re: Hadoop EC2 clusters -- internal DNS for tasktracker/datanode UI hyperlinks?

[Evan Pollan <ev...@gmail.com>]

I saw that, but I need to be able to hit other AWS resources directly, so
I'd end up having to pull down the instance manifest and build a proxy
config file dynamically, just as I'm doing for the /etc/hosts file now.

No worries.  This was more of a hadoop question than a whirr question.  I'm
assuming all the datanode/tasktracker bits in the web UI are based on those
services self-reporting their hostnames -- it'd be nice to be able to have
them use their public DNS names, instead of the private ones.



On Mon, Feb 13, 2012 at 9:29 AM, Andrei Savu <sa...@gmail.com> wrote:

> The recommended way is to set your browser to use the local SOCKS proxy.
>
> See "Run a proxy" in
> http://whirr.apache.org/docs/0.7.0/quick-start-guide.html
>
> We are also working on making that .pac file part of the distribution:
> https://issues.apache.org/jira/browse/WHIRR-35
>
> -- Andrei
>
>
> On Mon, Feb 13, 2012 at 5:26 PM, Evan Pollan <ev...@gmail.com>wrote:
>
>> This is a minor annoyance, but I'm wondering if anyone has found a way
>> around this.
>>
>> When whirr creates a hadoop cluster in EC2, I've found that the admin UI
>> (i.e. the web interface served up at <jobtracker>:50030, e.g.) uses the
>> public DNS names for all systems until you get down to the individual tasks
>> on a particular task tracker and want to download some or all of the logs
>> for a particular log.  At this point, the URLs are rendered using the
>> private DNS names and are not resolvable outside EC2.
>>
>> Is there any way to get around this, other than pulling down the cluster
>> instance manifest and generating /etc/host entries all the private DNS
>> names pointing to each instance's public IPs?
>>
>
>

Re: Hadoop EC2 clusters -- internal DNS for tasktracker/datanode UI hyperlinks?

[Marco Didonna <m....@gmail.com>]

On 13 February 2012 16:29, Andrei Savu <sa...@gmail.com> wrote:
> The recommended way is to set your browser to use the local SOCKS proxy.
>
> See "Run a proxy"
> in http://whirr.apache.org/docs/0.7.0/quick-start-guide.html
>
> We are also working on making that .pac file part of the distribution:
> https://issues.apache.org/jira/browse/WHIRR-35
>
> -- Andrei

If I may add my 2 cents I'd recommend to make sure your foxyproxy is
in "Use proxies based on their pre-defined patterns and priority" mode
since it is not the default mode of operation. Secondly remember that
foxyproxy doesn't record the .pac file, it loads it every time it
encounters a url that matches the pattern so you cannot delete the
.pac file or move it.

Marco

Re: Hadoop EC2 clusters -- internal DNS for tasktracker/datanode UI hyperlinks?

[Andrei Savu <sa...@gmail.com>]

The recommended way is to set your browser to use the local SOCKS proxy.

See "Run a proxy" in
http://whirr.apache.org/docs/0.7.0/quick-start-guide.html

We are also working on making that .pac file part of the distribution:
https://issues.apache.org/jira/browse/WHIRR-35

-- Andrei

On Mon, Feb 13, 2012 at 5:26 PM, Evan Pollan <ev...@gmail.com> wrote:

> This is a minor annoyance, but I'm wondering if anyone has found a way
> around this.
>
> When whirr creates a hadoop cluster in EC2, I've found that the admin UI
> (i.e. the web interface served up at <jobtracker>:50030, e.g.) uses the
> public DNS names for all systems until you get down to the individual tasks
> on a particular task tracker and want to download some or all of the logs
> for a particular log.  At this point, the URLs are rendered using the
> private DNS names and are not resolvable outside EC2.
>
> Is there any way to get around this, other than pulling down the cluster
> instance manifest and generating /etc/host entries all the private DNS
> names pointing to each instance's public IPs?
>