You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2014/08/18 12:10:18 UTC

[jira] [Created] (SLIDER-330) Slider to upload all JARs to AM via distributed cache

Steve Loughran created SLIDER-330:
-------------------------------------

             Summary: Slider to upload all JARs to AM via distributed cache
                 Key: SLIDER-330
                 URL: https://issues.apache.org/jira/browse/SLIDER-330
             Project: Slider
          Issue Type: New Feature
          Components: client
    Affects Versions: Slider 0.50
            Reporter: Steve Loughran


Currently slider only uploads the JARs we know/think aren't on the YARN classpath, and sets the CP from that and the {{yarn.application.classpath}} conf option

This is brittle against Hadoop versions and installations
# the option {{yarn.application.classpath}}  may be missing or wrong, at which point the AM refuses to start
# if YARN adds a different version of a dependency we push up, we end up in >1 version on the CP pain.
# if the Hadoop installation itself has some older JARs which don't link up to our code (e.g. AM uses protobuf fields not in hadoop-yarn JARs), then AM won't start with linkage errors.

The solution: isolate our classpath by pushing up all dependencies.

This requires Slider to know all its dependencies, which can be done via one of
* static coding (painful, brittle)
* chained .asm dependency analysis (As Twill does). Efficient and near-invisible, but breaks for any introspective binding.
* upload {{$SLIDER_HOME/lib/*.jar}}. Relies on mvn to build the files, the assembly code to add them to RPM and zip.

The last option, upload {{lib/*.jar}} seems the easiest and should be reliable, provided {{$SLIDER_HOME/lib/*.jar}} is set. 

For slider-client-as-API, the homedir may be unset. Here we propose letting the client config define the homedir, having that over-ride any env var.

For performance, especially long-haul, the distributed cache should be used



--
This message was sent by Atlassian JIRA
(v6.2#6252)