You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2014/08/18 12:10:18 UTC
[jira] [Created] (SLIDER-330) Slider to upload all JARs to AM via
distributed cache
Steve Loughran created SLIDER-330:
-------------------------------------
Summary: Slider to upload all JARs to AM via distributed cache
Key: SLIDER-330
URL: https://issues.apache.org/jira/browse/SLIDER-330
Project: Slider
Issue Type: New Feature
Components: client
Affects Versions: Slider 0.50
Reporter: Steve Loughran
Currently slider only uploads the JARs we know/think aren't on the YARN classpath, and sets the CP from that and the {{yarn.application.classpath}} conf option
This is brittle against Hadoop versions and installations
# the option {{yarn.application.classpath}} may be missing or wrong, at which point the AM refuses to start
# if YARN adds a different version of a dependency we push up, we end up in >1 version on the CP pain.
# if the Hadoop installation itself has some older JARs which don't link up to our code (e.g. AM uses protobuf fields not in hadoop-yarn JARs), then AM won't start with linkage errors.
The solution: isolate our classpath by pushing up all dependencies.
This requires Slider to know all its dependencies, which can be done via one of
* static coding (painful, brittle)
* chained .asm dependency analysis (As Twill does). Efficient and near-invisible, but breaks for any introspective binding.
* upload {{$SLIDER_HOME/lib/*.jar}}. Relies on mvn to build the files, the assembly code to add them to RPM and zip.
The last option, upload {{lib/*.jar}} seems the easiest and should be reliable, provided {{$SLIDER_HOME/lib/*.jar}} is set.
For slider-client-as-API, the homedir may be unset. Here we propose letting the client config define the homedir, having that over-ride any env var.
For performance, especially long-haul, the distributed cache should be used
--
This message was sent by Atlassian JIRA
(v6.2#6252)