Issue launching slider applications

[Ed Kohlwey <ek...@gmail.com>]

I'm attempting to use slider on one of our clusters.

Slider fails to launch it's AM and leaves this in the AM logs:

LogType: slider-err.txt
LogLength: 61
Log Contents:
/bin/bash: {{JAVA_HOME}}/bin/java: No such file or directory

This appears to be a yarn-ish error - I traced through to
ApplicationConstants.java and found that the {{ and }} characters are
supposed to be interpreted by the node manager that launches the command,
to do platform-specific environment variable interpolation (since windows
and unix use different characters).

Clearly this isn't happening. Has anyone seen this before?

Re: Issue launching slider applications

[Steve Loughran <st...@hortonworks.com>]

(sorry, thought I'd reply to this ... apologies)

The latest release - 0.50.1 is built against 2.4.1, as of earlier this week
we've switch development to 2.6 for ongoing work related to YARN-896
features our colleagues are setting up for us, and some things we are
adding ourselves.

there's one bit of code in the AM that won't link against Hadoop 2.2,
related to AM restart. On Hadoop 2.4.1+, when the AM restarts all the
containers keep running, the newly restarted AM gets the list of containers
and rebuilds its state. This added some new fields to the AM registration
operations, and while the protobuf wire format is robust, the protoc code
hides all that from the java code, where you get new get/set methods,
instead of being able to ask for a field by number. As a result: no linkage
on 2.2 where the relevant classes compiled into hadoop-yarn.jar are
different.

AM restart is one of those things that's been put in with us as the first
customer; we'd originally introspected to bind to it, but that actually
caused us to miss a bug in Hadoop 2.3...the tests weren't being fussy about
whether they passed or failed. Hence the switch to 2.4, then 2.4.1. The big
leap to 2.6 will get us use: token-renewal for long lived applications
(YARN-941), the Credential Store, the YARN registry and labelled nodes:
http://wiki.apache.org/hadoop/Roadmap




On 18 August 2014 19:13, Ed Kohlwey <ek...@gmail.com> wrote:

> Looks like I have Hadoop 2.2.
>
> Do you think its worthwhile to try recompiling to get this to work or do
> you think I'll hit other hadoop version compatibility snags? We also still
> have Hoya running from earlier this year... hopefully there haven't been
> too many other changes like this?
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Issue launching slider applications

[Ed Kohlwey <ek...@gmail.com>]

Looks like I have Hadoop 2.2.

Do you think its worthwhile to try recompiling to get this to work or do
you think I'll hit other hadoop version compatibility snags? We also still
have Hoya running from earlier this year... hopefully there haven't been
too many other changes like this?

Re: Issue launching slider applications

[Steve Loughran <st...@hortonworks.com>]

what version of Hadoop are you using? I suspect that this expansion feature
may be hadoop 2.3+

-steve


On 18 August 2014 18:48, Ed Kohlwey <ek...@gmail.com> wrote:

> I'm attempting to use slider on one of our clusters.
>
> Slider fails to launch it's AM and leaves this in the AM logs:
>
> LogType: slider-err.txt
> LogLength: 61
> Log Contents:
> /bin/bash: {{JAVA_HOME}}/bin/java: No such file or directory
>
> This appears to be a yarn-ish error - I traced through to
> ApplicationConstants.java and found that the {{ and }} characters are
> supposed to be interpreted by the node manager that launches the command,
> to do platform-specific environment variable interpolation (since windows
> and unix use different characters).
>
> Clearly this isn't happening. Has anyone seen this before?
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.