[jira] [Commented] (SLIDER-330) Slider to upload all JARs to AM via distributed cache

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SLIDER-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115822#comment-14115822 ] 

Steve Loughran commented on SLIDER-330:
---------------------------------------

...maybe there is a way at build time to create a manifest of dependencies and place it in a file in the resources/ tree, so that it can be read in automatically

> Slider to upload all JARs to AM via distributed cache
> -----------------------------------------------------
>
>                 Key: SLIDER-330
>                 URL: https://issues.apache.org/jira/browse/SLIDER-330
>             Project: Slider
>          Issue Type: New Feature
>          Components: client
>    Affects Versions: Slider 0.50
>            Reporter: Steve Loughran
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently slider only uploads the JARs we know/think aren't on the YARN classpath, and sets the CP from that and the {{yarn.application.classpath}} conf option
> This is brittle against Hadoop versions and installations
> # the option {{yarn.application.classpath}}  may be missing or wrong, at which point the AM refuses to start
> # if YARN adds a different version of a dependency we push up, we end up in >1 version on the CP pain.
> # if the Hadoop installation itself has some older JARs which don't link up to our code (e.g. AM uses protobuf fields not in hadoop-yarn JARs), then AM won't start with linkage errors.
> The solution: isolate our classpath by pushing up all dependencies.
> This requires Slider to know all its dependencies, which can be done via one of
> * static coding (painful, brittle)
> * chained .asm dependency analysis (As Twill does). Efficient and near-invisible, but breaks for any introspective binding.
> * upload {{$SLIDER_HOME/lib/*.jar}}. Relies on mvn to build the files, the assembly code to add them to RPM and zip.
> The last option, upload {{lib/*.jar}} seems the easiest and should be reliable, provided {{$SLIDER_HOME/lib/*.jar}} is set. 
> For slider-client-as-API, the homedir may be unset. Here we propose letting the client config define the homedir, having that over-ride any env var.
> For performance, especially long-haul, the distributed cache should be used



--
This message was sent by Atlassian JIRA
(v6.2#6252)