You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@zeppelin.apache.org by liuxunorg <gi...@git.apache.org> on 2018/07/16 02:15:44 UTC

[GitHub] zeppelin pull request #3074: [ZEPPELIN-3610] Cluster Raft module design

GitHub user liuxunorg opened a pull request:

    https://github.com/apache/zeppelin/pull/3074

    [ZEPPELIN-3610] Cluster Raft module design

    By using the Raft protocol, multiple Zeppelin-Server groups are built into a Zeppelin cluster, the cluster State Machine is maintained through the Raft protocol, and the services in the cluster are agreed upon. The Zeppelin-Server and Zeppelin-Interperter services and processes are stored in the Cluster MetaData. Metadata information;
    
    [Feature]
    
    * [x] added support for checking lambda syntax styles in the pom.xml file
    * [x] add raft algorithm copycat jar
    * [x] add  Cluster State Machine
    * [x] add state machine query command
    * [x] add state machine delete command
    * [x] add state machine put command
    
    * https://issues.apache.org/jira/browse/ZEPPELIN-3610
    
    CI pass
    
    * Does the licenses files need update? No
    * Is there breaking changes for older versions? No
    * Does this needs documentation? No


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/liuxunorg/zeppelin ZEPPELIN-3610

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/zeppelin/pull/3074.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3074
    
----
commit b2faba1010c9517d1f651e6c69df1b3651c4b13f
Author: liuxunorg <33...@...>
Date:   2018-07-14T03:25:18Z

    [ZEPPELIN-3610] Cluster Raft module design
    
    By using the Raft protocol, multiple Zeppelin-Server groups are built into a Zeppelin cluster, the cluster State Machine is maintained through the Raft protocol, and the services in the cluster are agreed upon. The Zeppelin-Server and Zeppelin-Interperter services and processes are stored in the Cluster MetaData. Metadata information;
    
    [Feature]
    
    * [x] added support for checking lambda syntax styles in the pom.xml file
    * [x] add raft algorithm copycat jar
    * [x] add  Cluster State Machine
    * [x] add state machine query command
    * [x] add state machine delete command
    * [x] add state machine put command
    
    * https://issues.apache.org/jira/browse/ZEPPELIN-3610
    
    CI pass
    
    * Does the licenses files need update? No
    * Is there breaking changes for older versions? No
    * Does this needs documentation? No

----


---

[GitHub] zeppelin issue #3074: [ZEPPELIN-3610] Cluster Raft module design

Posted by zjffdu <gi...@git.apache.org>.

Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/3074
  
    @liuxunorg Another suggestion is that could you send a mail to user/dev mail list about the zeppelin HA feature and attach the design doc, it is better to let the community aware this feature beforehand and you may get more feedback and comments. 


---

[GitHub] zeppelin issue #3074: [ZEPPELIN-3610] Cluster Raft module design

Posted by jongyoul <gi...@git.apache.org>.

Github user jongyoul commented on the issue:

    https://github.com/apache/zeppelin/pull/3074
  
    Got it. I hope not to spend much time merging this feature into the master
    branch after finishing implementations.
    
    On Mon, Jul 23, 2018 at 3:22 PM, Jeff Zhang <no...@github.com>
    wrote:
    
    > @jongyoul <https://github.com/jongyoul> This is for the zeppelin HA
    > feature which will involve many PRs. Here's the pros and cons of creating
    > new branch.
    > Pros.
    >
    >    1. Move fast of implementing this feature
    >    2. Keep the master branch stable.
    >    3. Interested users can try this feature earlier.
    >    4. This feature branch will become much stable before merging into
    >    master branch.
    >
    > Cons.
    >
    >    1. Merging into master will take a lot time.
    >
    > That's why I think creating new branch for this feature is necessary.
    >
    > —
    > You are receiving this because you were mentioned.
    > Reply to this email directly, view it on GitHub
    > <https://github.com/apache/zeppelin/pull/3074#issuecomment-406952014>, or mute
    > the thread
    > <https://github.com/notifications/unsubscribe-auth/ADcflotPVllHE6hB6xB2FCSzH14c-GGsks5uJWuygaJpZM4VQfJN>
    > .
    >
    
    
    
    -- 
    이종열, Jongyoul Lee, 李宗烈
    http://madeng.net



---

[GitHub] zeppelin issue #3074: [ZEPPELIN-3610] Cluster Raft module design

Posted by zjffdu <gi...@git.apache.org>.

Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/3074
  
    @liuxunorg Could you do rebase and retrigger the travis ? If CI pass, I will create new branch for you, then you can work on the new branch. 


---

[GitHub] zeppelin issue #3074: [ZEPPELIN-3610] Cluster Raft module design

Posted by liuxunorg <gi...@git.apache.org>.

Github user liuxunorg commented on the issue:

    https://github.com/apache/zeppelin/pull/3074
  
    hi @Tagar , Increasing the service state consistency of the system through zookeeper is a common practice in distributed systems. But this will cause zeppelin to rely on an external service zookeeper, this increases the complexity of zeppelin deployment. the Raft protocol is an easy-to-understand service state consistency protocol. there are some mature Raft algorithm libraries (for example: copycat), we can directly introduce the copycat package in zeppelin-server, and directly let several zeppelin-server directly form a zeppelin cluster.


---

[GitHub] zeppelin issue #3074: [ZEPPELIN-3610] Cluster Raft module design

Posted by jongyoul <gi...@git.apache.org>.

Github user jongyoul commented on the issue:

    https://github.com/apache/zeppelin/pull/3074
  
    Deleted?


---

[GitHub] zeppelin issue #3074: [ZEPPELIN-3610] Cluster Raft module design

Posted by zjffdu <gi...@git.apache.org>.

Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/3074
  
    @liuxunorg You don't need to create a new PR for the same ticket. You can reopen it to trigger the build if this is what you want. 


---

[GitHub] zeppelin issue #3074: [ZEPPELIN-3610] Cluster Raft module design

Posted by jongyoul <gi...@git.apache.org>.

Github user jongyoul commented on the issue:

    https://github.com/apache/zeppelin/pull/3074
  
    I believe we need design docs to describe this feature. I'm not sure which component would be located which places. @liuxunorg Could you describe it with a diagram?


---

[GitHub] zeppelin issue #3074: [ZEPPELIN-3610] Cluster Raft module design

Posted by liuxunorg <gi...@git.apache.org>.

Github user liuxunorg commented on the issue:

    https://github.com/apache/zeppelin/pull/3074
  
    @Tagar @jongyoul @zjffdu  If the community feels that zeppelin has been added to the clustering feature, the changes are relatively large. Is it possible to add a new code branch?
    This can be more convenient. I will incorporate the code related to this feature earlier. You can test it together to see the running effect. If you feel good, merge it into the master branch.
    Because of this distributed clustering function, it has been used in our company for a period of time and has a good use effect.


---

[GitHub] zeppelin issue #3074: [ZEPPELIN-3610] Cluster Raft module design

Posted by liuxunorg <gi...@git.apache.org>.

Github user liuxunorg commented on the issue:

    https://github.com/apache/zeppelin/pull/3074
  
    @jongyoul very sorry, I executed the wrong git command, Rebuilt the `liuxunorg:ZEPPELIN-3610` branch, Causes this pull to automatically close. I can't reopen this pull now. `Liuxunorg:ZEPPELIN-3610` branch was rebuilt, It may not be useful to reopen this pull.
    
    Am I requesting a pull again?


---

[GitHub] zeppelin issue #3074: [ZEPPELIN-3610] Cluster Raft module design

Posted by Tagar <gi...@git.apache.org>.

Github user Tagar commented on the issue:

    https://github.com/apache/zeppelin/pull/3074
  
    Just out of curiosity, why Raft Protocol? Have you considered alternatives, for example, Zookeeper could do here, I guess. I think what the design document is missing is some sort of pros and cons compared to alternatives. Thanks! 


---

[GitHub] zeppelin issue #3074: [ZEPPELIN-3610] Cluster Raft module design

Posted by liuxunorg <gi...@git.apache.org>.

Github user liuxunorg commented on the issue:

    https://github.com/apache/zeppelin/pull/3074
  
    Ok, let me improve the documentation and add a diagram, then send an email to user/dev mail list.


---

[GitHub] zeppelin issue #3074: [ZEPPELIN-3610] Cluster Raft module design

Posted by liuxunorg <gi...@git.apache.org>.

Github user liuxunorg commented on the issue:

    https://github.com/apache/zeppelin/pull/3074
  
    CI pass. https://travis-ci.org/liuxunorg/zeppelin/builds/407058117


---

[GitHub] zeppelin issue #3074: [ZEPPELIN-3610] Cluster Raft module design

Posted by zjffdu <gi...@git.apache.org>.

Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/3074
  
    It make sense @liuxunorg , I will create a new branch for this feature. 


---

[GitHub] zeppelin issue #3074: [ZEPPELIN-3610] Cluster Raft module design

Posted by liuxunorg <gi...@git.apache.org>.

Github user liuxunorg commented on the issue:

    https://github.com/apache/zeppelin/pull/3074
  
    @zjffdu Sorry, I will pay attention next time.


---