Installation instructions imply/recommend that Spot-ML runs on YARN Node Manager, but should be Edge Server

[Jeremy Nelson <je...@digitalminion.com>]

On https://spot.incubator.apache.org/doc/ in section “2. Deployment
Recommendations”,
we are recommending that the “spot-ml” software (which is all
scala-spark-streaming)
should be run on the YARN node manager. On the "service layout", it is
implied that
the spot-ml software is installed on "Worker" node.

But these jobs are intended to be launched from the edge node, where the
YARN service packages them up and publishes them to worker nodes. Thus, to
make this simpler and more obvious for the users, the "ML" task should be
shown as being installed, configured, and executed from the Edge node.

Jeremy

Re: Installation instructions imply/recommend that Spot-ML runs on YARN Node Manager, but should be Edge Server

[Tadd Wood <ta...@apache.org>]

Agreed Jeremy, I think the diagram is a little misleading.  If you want,
checkout this JIRA that discusses this issue as well as a few other parts
of the documentation we need to revise:
https://issues.apache.org/jira/projects/SPOT/issues/SPOT-224

Thank you,
Tadd Wood

On Mon, Feb 10, 2020 at 2:29 PM Jeremy Nelson <je...@digitalminion.com>
wrote:

> On https://spot.incubator.apache.org/doc/ in section “2. Deployment
> Recommendations”,
> we are recommending that the “spot-ml” software (which is all
> scala-spark-streaming)
> should be run on the YARN node manager. On the "service layout", it is
> implied that
> the spot-ml software is installed on "Worker" node.
>
> But these jobs are intended to be launched from the edge node, where the
> YARN service packages them up and publishes them to worker nodes. Thus, to
> make this simpler and more obvious for the users, the "ML" task should be
> shown as being installed, configured, and executed from the Edge node.
>
> Jeremy
>