You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by Sumit Mohanty <sm...@hortonworks.com> on 2014/08/30 20:55:24 UTC
Self containing all Slider app artifacts
Should we also move slider-client.tar.gz and application package into the
application instance specific folder in HDFS? This along with the changes
due to SLIDER-330 will likely make the application bits completely self
contained.
-Sumit
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Self containing all Slider app artifacts
Posted by Sumit Mohanty <sm...@hortonworks.com>.
On Sat, Aug 30, 2014 at 1:18 PM, Jon Maron <jm...@hortonworks.com> wrote:
>
> On Aug 30, 2014, at 2:55 PM, Sumit Mohanty <sm...@hortonworks.com>
> wrote:
>
> > Should we also move slider-client.tar.gz and application package into the
> > application instance specific folder in HDFS? This along with the changes
> > due to SLIDER-330 will likely make the application bits completely self
> > contained.
>
> Maybe I’ve been buried in security related work for too long so…
>
> 1) slider-client.tar.gz? Do you mean slider-agent.tar.gz or is there a
> client installer?
>
Yes, slider-agent.tar.gz. And no, we haven't secretly added another tarball
:-)
> 2) what is the “application instance specific folder”?
> hdfs://user/<username>/slider or the like?
>
Subfolders within hdfs:///user/<username>/.slider/cluster/<cluster name>.
Some AM artifacts are already there under tmp.
3) Do we really want to group Slider/AM artifacts and various application
> artifacts together? Perhaps under a shared parent directory rather than
> the same directory?
>
>
We can define a folder structure that keeps various artifacts separate.
There are two classes of artifacts, "application definition", and "Slider
infrastructure". The later includes slider-agent, yarn libraries, AppMaster
libraries, etc.
* At least the slider-agent.tar.gz could move in as part of the Slider
infra. There are two choices here and second one is just an optimization
** Store all Slider infra in a sub-folder within
hdfs:///user/<username>/.slider/cluster/<cluster name>
** Create a common area in HDFS and refer to the infrastructure bits from
there (a common world readable location)
* The application definition zip file can also be moved in to say
hdfs:///user/<username>/.slider/cluster/<cluster name>/definition
The advantages will be that the application will be self contained and
upgrades, to package or slider, can happen independently for each
application instance. Again it may be a disadvantage if one wants to
upgrade all instances together but that is really a tooling issues.
Are there any TEZ/MR best practices that we can refer to?
>
> > -Sumit
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Self containing all Slider app artifacts
Posted by Jon Maron <jm...@hortonworks.com>.
On Aug 30, 2014, at 2:55 PM, Sumit Mohanty <sm...@hortonworks.com> wrote:
> Should we also move slider-client.tar.gz and application package into the
> application instance specific folder in HDFS? This along with the changes
> due to SLIDER-330 will likely make the application bits completely self
> contained.
Maybe I’ve been buried in security related work for too long so…
1) slider-client.tar.gz? Do you mean slider-agent.tar.gz or is there a client installer?
2) what is the “application instance specific folder”? hdfs://user/<username>/slider or the like?
3) Do we really want to group Slider/AM artifacts and various application artifacts together? Perhaps under a shared parent directory rather than the same directory?
>
> -Sumit
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.