[jira] [Commented] (BIGTOP-2490) Spark in HA when Zookeeper is available

["Evans Ye (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15828248#comment-15828248 ] 

Evans Ye commented on BIGTOP-2490:
----------------------------------

I'm wondering how zookeeper_connection_string being provided?
It looks like the only appearance of that variable is under kafka:
{code}
hieradata/bigtop/cluster.yaml:kafka::server::zookeeper_connection_string: "%{hiera('bigtop::hadoop_head_node')}:2181"
{code}

It doesn't make sense to me so far. Can you elaborate your thought here?



> Spark in HA when Zookeeper is available 
> ----------------------------------------
>
>                 Key: BIGTOP-2490
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-2490
>             Project: Bigtop
>          Issue Type: Improvement
>          Components: spark
>    Affects Versions: 1.1.0
>            Reporter: Konstantinos Tsakalozos
>            Assignee: Konstantinos Tsakalozos
>            Priority: Minor
>              Labels: features
>
> Spark could work in HA mode as long as a Zookeeper deployment is available. In this improvement we add an option to the Spark configuration through which  we can set the Zookeeper ensemble string. This string is pushed down to Spark configs causing the Spark to function in HA mode.
> In HA mode Zookeeper is used to select the Spark leader and the workers need to ping all the potential leaders to pinpoint the current one. 
> You can monitor the progress of this work in: https://github.com/juju-solutions/bigtop/tree/BIGTOP-2490



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)