[jira] [Created] (BIGTOP-2834) spark charm: refactor for restricted networks; lib cleanup

["Kevin W Monroe (JIRA)" <ji...@apache.org>]

Kevin W Monroe created BIGTOP-2834:
--------------------------------------

             Summary: spark charm: refactor for restricted networks; lib cleanup
                 Key: BIGTOP-2834
                 URL: https://issues.apache.org/jira/browse/BIGTOP-2834
             Project: Bigtop
          Issue Type: Improvement
          Components: deployment
    Affects Versions: 1.2.0
            Reporter: Kevin W Monroe
            Assignee: Kevin W Monroe
            Priority: Minor
             Fix For: 1.3.0


The spark charm currently grabs Spark-Bench and sample data for the pagerank benchmark from s3.  This is not good for a couple reasons:

1 - Spark-Bench hasn't been verified with Spark 2.1.0.
2 - Deployments might fail in an environment where the charm is not allowed to fetch things from s3.

#1 is an easy fix; just change the default behavior.  SB is there if a user wants it, but we shouldn't install it by default.

#2 is trickier because we *do* need the sample data for our pagerank benchmark.  Refactor this as a Juju Resource (vs s3 URL) so it comes from the charm store.

While we're here, simplify the spark charm library to remove leftover cruft from pre-Bigtop days and break out the complex configure method into more purpose-optimized methods.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)