You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by "Gour Saha (JIRA)" <ji...@apache.org> on 2015/10/15 15:58:05 UTC
[jira] [Comment Edited] (SLIDER-777) Provide slider dependencies as
a self contained versioned tarball
[ https://issues.apache.org/jira/browse/SLIDER-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952532#comment-14952532 ]
Gour Saha edited comment on SLIDER-777 at 10/15/15 1:58 PM:
------------------------------------------------------------
To keep matters simple this feature has been implemented as follows -
# Install slider (client version say *0.81.0-incubating-SNAPSHOT*) in a node (gateway) in the cluster
# sudo as user {{hdfs}}
# Upload the versioned tarball by calling -
{code}
slider dependency --upload
{code}
In a cluster, this cmd needs to be called only once, for a specific version of the client.
# The above cmd uploads a dependency tarball to the following location in hdfs -
{code}
/apps/<slider_version>/slider/slider.tar.gz
{code}
So for client version *0.81.0-incubating-SNAPSHOT* the location will be -
{code}
/apps/0.81.0-incubating-SNAPSHOT/slider/slider.tar.gz
{code}
# All subsequent slider app creation jobs submitted using slider client (or the API via slider-core.jar) version *0.81.0-incubating-SNAPSHOT*, will use this uploaded dependency tarball.
App submissions become significantly faster, as ~50MB size of files are not required to be uploaded for every create call. In some clusters it has been seen to take less than a second, where it used to take north of 10 secs without the dependency uploaded.
Note, the above steps are not mandatory. Slider client will continue to work in its legacy way, if the dependency upload command is never invoked in a cluster.
was (Author: gsaha):
To keep matters simple this feature has been implemented as follows -
# Install slider (client version say *0.81.0-incubating-SNAPSHOT*) in a node (gateway) in the cluster
# sudo as user {{hdfs}}
# Upload the versioned tarball by calling -
{code}
slider dependency --upload
{code}
In a cluster, this cmd needs to be called only once, for a specific version of the client.
# The above cmd uploads a dependency tarball to the following location in hdfs -
{code}
/apps/<slider_version>/slider/slider.tar.gz
{code}
So for client version *0.81.0-incubating-SNAPSHOT* the location will be -
{code}
/apps/0.81.0-incubating-SNAPSHOT/slider/slider.tar.gz
{code}
# All subsequent slider app creation jobs submitted using slider client version *0.81.0-incubating-SNAPSHOT*, will use this uploaded dependency tarball.
App submissions become significantly faster, as ~50MB size of files are not required to be uploaded for every create call. In some clusters it has been seen to take less than a second, where it used to take north of 10 secs without the dependency uploaded.
Note, the above steps are not mandatory. Slider client will continue to work in its legacy way, if the dependency upload command is never invoked in a cluster.
> Provide slider dependencies as a self contained versioned tarball
> -----------------------------------------------------------------
>
> Key: SLIDER-777
> URL: https://issues.apache.org/jira/browse/SLIDER-777
> Project: Slider
> Issue Type: Bug
> Components: build, client
> Affects Versions: Slider 0.70
> Reporter: Sumit Mohanty
> Assignee: Gour Saha
> Priority: Critical
> Fix For: Slider 0.81
>
> Time Spent: 32h
> Remaining Estimate: 10h
>
> Currently, slider client picks up all the dependencies from the lib folder. This causes an issue for code that embed Slider client where either they need to specify a folder which has the all the required (but no more) jar files or their own lib directory which contains jar files of the embedding application as well.
> Slider should explore (similar to Tez or MR) the possibility of packaging all dependencies into a tarball that can be stored in a version aware fashion in HDFS or local FS.
> A Slider client can choose to use a specific version when creating the an instance of an application. Now, it is possible for a client of certain version to create an app using another version of the tarball. _This requires further investigation in terms of how easy to debug the issues if there is no compatibility between versions of the client and the tarball._
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)