You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by kwmonroe <gi...@git.apache.org> on 2017/07/07 00:01:26 UTC
[GitHub] bigtop pull request #246: BIGTOP-2834: spark charm: refactor for restricted ...
GitHub user kwmonroe opened a pull request:
https://github.com/apache/bigtop/pull/246
BIGTOP-2834: spark charm: refactor for restricted networks; lib cleanup
See [BIGTOP-2834](https://issues.apache.org/jira/browse/BIGTOP-2834) for details.
- Make the pagerank sample data a juju resource
- Don't install Spark-Bench by default
- cleanup the spark charm lib
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/juju-solutions/bigtop feature/BIGTOP-2834/spark-restricted-net
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/bigtop/pull/246.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #246
----
commit 7316ea6a3809b217acd370f83f231548ceaeaab3
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2017-07-05T20:30:06Z
sparkbench changes
Do not install Spark-Bench by default for a couple reasons:
- it has not been tested with Spark 2.1.x.
- restricted networks may not allow the charm to fetch from s3,
which is where we host the SB tarball.
Simplify config related to SB since the payload is not arch specific.
Include a SPARK_HOME config in /etc/environment since apps like SB
will look for default values there.
commit b1c8f795a55dd12926b267ec1bbc3ea34d5a6c6a
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2017-07-05T22:44:02Z
add a sample-data resource for pagerank now that sparkbench is not installed by default
commit c11b1299f0aa4932407d32652045149c49b2967e
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2017-07-06T04:23:11Z
need the add_dirs to create spark event dir (else history server fails to start)
commit 06c103df43e5abf2ef712f762ea6e76b8f2d974c
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2017-07-06T04:23:36Z
update pagerank action to use sample from new resource
commit 50d464ee5a46b9caa3d5567d4c9ab86c41c8afda
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2017-07-06T22:55:03Z
pagerank: copy the sample data to hdfs when spark is in yarn mode
commit 3f6389ce74b742bfcdda02791d93fbb4ac4154ee
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2017-07-06T22:57:35Z
trigger reinstall if sample data resource has changed
commit 0a0f2fefb38f86ec15963598d8fb4225ca4cfba6
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2017-07-06T23:08:43Z
update/simplify spark lib
- dont bootstrap spark
- packages handle our users/groups
- we handle the dirs as needed based on the spark exe mode
- remove setup method
- rename methods to be more accurate
- handle_sparkbench -> configure_sparkbench
- install_demo -> configure_examples
- setup_hdfs_logs -> configure_events_dir
- configure_examples ensures sparkpi and sample-data are installed correctly
- configure_events_dir creates dirs and sets perms based on spark exe mode
- BUG
- passing an empty hosts['namenode'] breaks when exe mode goes from yarn->non-yarn
- no need to nullify hosts['namenode']
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #246: BIGTOP-2834: spark charm: refactor for restricted ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/bigtop/pull/246
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---