[SPOT-266] ml_ops.sh script should use Spark 2?

[Tadd Wood <ta...@apache.org>]

I notice there’s a dependency on Spark 2.1+ that didn’t get resolved per
this JIRA (https://issues.apache.org/jira/projects/SPOT/issues/SPOT-266). I
propose adding a check in this script to verify the user’s version of Spark
and if it's not at least 2.1 the script will exit and indicate that the
user should upgrade their version of Spark. We also need to replace
spark-submit with spark-submit2. Any thoughts for/against updating this
script?

Also would anyone be willing to update the documentation on the Spot
website to reflect the correct required version of Spark to run Spot?

Thank you,
Tadd Wood

Re: [SPOT-266] ml_ops.sh script should use Spark 2?

[skip cruse <f8...@gmail.com>]

That seems like a fair solution. I noticed there’s also a spark-submit
command that’s required when running this pipeline:worke
<https://github.com/apache/incubator-spot/blob/9f63ab724e075fc3da458bd8ff86a777ddb94d68/spot-ingest/pipelines/proxy/worker.py>
r.py . We should make the update here as well since I believe this also
requries Spark 2.1+.

On Fri, Mar 20, 2020 at 4:24 PM Tadd Wood <ta...@apache.org> wrote:

> I notice there’s a dependency on Spark 2.1+ that didn’t get resolved per
> this JIRA (https://issues.apache.org/jira/projects/SPOT/issues/SPOT-266).
> I
> propose adding a check in this script to verify the user’s version of Spark
> and if it's not at least 2.1 the script will exit and indicate that the
> user should upgrade their version of Spark. We also need to replace
> spark-submit with spark-submit2. Any thoughts for/against updating this
> script?
>
> Also would anyone be willing to update the documentation on the Spot
> website to reflect the correct required version of Spark to run Spot?
>
> Thank you,
> Tadd Wood
>


-- 
if( bool halfWayThere == true);
echo "WOOOOOOOOOOOOOOOOAH! LIVIN ON A PRAYER";