You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/06/02 19:08:04 UTC

[jira] [Commented] (BIGTOP-2795) spark charm: fix sparkpi and rework start/stop logic

    [ https://issues.apache.org/jira/browse/BIGTOP-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035256#comment-16035256 ] 

ASF GitHub Bot commented on BIGTOP-2795:
----------------------------------------

GitHub user kwmonroe opened a pull request:

    https://github.com/apache/bigtop/pull/229

    BIGTOP-2795: spark charm: fix sparkpi and rework start/stop logic

    See [bigtop-2795](https://issues.apache.org/jira/browse/BIGTOP-2795) for details.
    
    - add bundle descriptions to the hadoop-spark yaml files
    - expose the worker ui port
    - rework start/stop logic to handle ports and wait when appropriate
    - fix sparkpi.sh by not hard coding the spark-examples.jar path


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/juju-solutions/bigtop bug/BIGTOP-2795/logic-fixes

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/bigtop/pull/229.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #229
    
----
commit 69e3e17574e67d38530fbe4beae8dd08e3dac6c2
Author: Kevin W Monroe <ke...@canonical.com>
Date:   2017-06-01T20:57:30Z

    adjust bundle description

commit e0e5d0cacdf41ae2e80c96f8b2251ec631da29d6
Author: Kevin W Monroe <ke...@canonical.com>
Date:   2017-06-02T18:56:12Z

    add worker-ui port; rework start/stop to open ports and wait for services appropriately

commit 71468acd0a47ae07f0488211df99a50309a2dabc
Author: Kevin W Monroe <ke...@canonical.com>
Date:   2017-06-02T18:57:11Z

    sparkpi: find example jar dynamically and rework the output

commit 3cd376fce4f0214b3ea9897213457c33cac9e725
Author: Kevin W Monroe <ke...@canonical.com>
Date:   2017-06-02T19:01:24Z

    add bundle descriptions to dev/local bundle yamls

----


> spark charm: fix sparkpi and rework start/stop logic
> ----------------------------------------------------
>
>                 Key: BIGTOP-2795
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-2795
>             Project: Bigtop
>          Issue Type: Improvement
>            Reporter: Kevin W Monroe
>            Assignee: Kevin W Monroe
>            Priority: Minor
>             Fix For: 1.2.1
>
>
> The current spark charm has a couple problems:
> - The sample workload (delivered as ~/sparkpi.sh) hard codes the spark-examples.jar path, and it's incorrect for spark2.
> - Expose the spark worker ui port (8081) when appropriate.
> - We only need to open the master ui port (8080) if we're in standalone mode.  There is no master process running in yarn or local modes.
> - We should only wait for master recovery if we have peers. It's important to wait for recovery before starting workers in a multi-node cluster, but for single-node, this is a waste of time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)