You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/06/02 19:08:04 UTC
[jira] [Commented] (BIGTOP-2795) spark charm: fix sparkpi and
rework start/stop logic
[ https://issues.apache.org/jira/browse/BIGTOP-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035256#comment-16035256 ]
ASF GitHub Bot commented on BIGTOP-2795:
----------------------------------------
GitHub user kwmonroe opened a pull request:
https://github.com/apache/bigtop/pull/229
BIGTOP-2795: spark charm: fix sparkpi and rework start/stop logic
See [bigtop-2795](https://issues.apache.org/jira/browse/BIGTOP-2795) for details.
- add bundle descriptions to the hadoop-spark yaml files
- expose the worker ui port
- rework start/stop logic to handle ports and wait when appropriate
- fix sparkpi.sh by not hard coding the spark-examples.jar path
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/juju-solutions/bigtop bug/BIGTOP-2795/logic-fixes
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/bigtop/pull/229.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #229
----
commit 69e3e17574e67d38530fbe4beae8dd08e3dac6c2
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2017-06-01T20:57:30Z
adjust bundle description
commit e0e5d0cacdf41ae2e80c96f8b2251ec631da29d6
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2017-06-02T18:56:12Z
add worker-ui port; rework start/stop to open ports and wait for services appropriately
commit 71468acd0a47ae07f0488211df99a50309a2dabc
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2017-06-02T18:57:11Z
sparkpi: find example jar dynamically and rework the output
commit 3cd376fce4f0214b3ea9897213457c33cac9e725
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2017-06-02T19:01:24Z
add bundle descriptions to dev/local bundle yamls
----
> spark charm: fix sparkpi and rework start/stop logic
> ----------------------------------------------------
>
> Key: BIGTOP-2795
> URL: https://issues.apache.org/jira/browse/BIGTOP-2795
> Project: Bigtop
> Issue Type: Improvement
> Reporter: Kevin W Monroe
> Assignee: Kevin W Monroe
> Priority: Minor
> Fix For: 1.2.1
>
>
> The current spark charm has a couple problems:
> - The sample workload (delivered as ~/sparkpi.sh) hard codes the spark-examples.jar path, and it's incorrect for spark2.
> - Expose the spark worker ui port (8081) when appropriate.
> - We only need to open the master ui port (8080) if we're in standalone mode. There is no master process running in yarn or local modes.
> - We should only wait for master recovery if we have peers. It's important to wait for recovery before starting workers in a multi-node cluster, but for single-node, this is a waste of time.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)