[jira] [Commented] (STORM-2306) Redeisgn Messaging Subsystem and switch to JCTools Queues

["Roshan Naik (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16280106#comment-16280106 ] 

Roshan Naik commented on STORM-2306:
------------------------------------

Since the PR webpage has become terribly slow due to the large number of comments, moving the non-code-review type conversation to the JIRA. 

The PR is ready to move forward with additional reviews and testing by anyone interested in test driving their topologies on this (much appreciated!).

- I believe all the key issues raised so far should be addressed, please take a look.  Also includes fixes to issues discovered during testing and perf runs.
- Have added a new design doc detailing the BackPressure model. That is the part that has undergone the most change off late.
- Based on my observations from perf runs and prior feedback on the PR, the new defaults have been tweaked to make it easy for existing workloads to transition to this with minimal or no tweaking but still get good perf.
- My colleague is in the process of running some perf numbers comparing master vs the latest 2306. Will share them soon.

cc: [~revans2] 

> Redeisgn Messaging Subsystem and switch to JCTools Queues
> ---------------------------------------------------------
>
>                 Key: STORM-2306
>                 URL: https://issues.apache.org/jira/browse/STORM-2306
>             Project: Apache Storm
>          Issue Type: Sub-task
>          Components: storm-core
>    Affects Versions: 2.0.0
>            Reporter: Roshan Naik
>            Assignee: Roshan Naik
>              Labels: pull-request-available
>          Time Spent: 33h
>  Remaining Estimate: 0h
>
> Details in these documents:
> 1) *Redesign of the messaging subsystem*
> https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit?usp=sharing
> This doc discusses the new design for the messaging system. Plus some of the optimizations being made.
> 2) *Choosing a high performance messaging queue:*
> https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing
> This doc looks into how fast hardware can do inter-thread messaging and why we chose the JCTools queues.
> 3) *Backpressure Model*
> https://docs.google.com/document/d/1Z9pRdI5wtnK-hVwE3Spe6VGCTsz9g8TkgxbTFcbL3jM/edit?usp=sharing
> Describes the Backpressure model integrated into the new messaging subsystem.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)