[jira] [Resolved] (SAMZA-2118) Improve the shutdown sequence of AsyncRunLoop.

["Xinyu Liu (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xinyu Liu resolved SAMZA-2118.
------------------------------
       Resolution: Fixed
    Fix Version/s: 1.1

> Improve the shutdown sequence of AsyncRunLoop.
> ----------------------------------------------
>
>                 Key: SAMZA-2118
>                 URL: https://issues.apache.org/jira/browse/SAMZA-2118
>             Project: Samza
>          Issue Type: Improvement
>            Reporter: Shanthoosh Venkataraman
>            Assignee: Shanthoosh Venkataraman
>            Priority: Major
>             Fix For: 1.1
>
>          Time Spent: 50m
>  Remaining Estimate: 0h
>
> A samza container is comprised of multiple tasks and event-loop. Event loop is the main orchestration layer between the different stages of a task(process, commit, window) and delivers messages to task by polling them from different consumers. 
> Currently in samza, the event loop dispatches messages to the tasks as per user-defined configuration `task.max.concurrency`. After dispatching messages with count equal to `task.max.concurrency` to the tasks,  the event loop is blocked until the processing of at-least a single dispatched message finishes(aka the task implementation invokes `taskCallBack.complete()` for at-least a dispatched message). 
> When the event loop is blocked by this, the user thread would sometimes try to shutdown the event loop. This shutdown intent marked by the user-thread will take effect only when the processing of a single message completes. In some-cases, when the task implementation does retry on failure for a message, this could prolong the shutdown of the samza event loop.
> This introduces tight coupling between the task processing time and the event loop shutdown time.
> This behavior was observed predominantly in standalone applications where the samza container is shutdown as a part of the re-balancing phase and task processing time was significantly larger.
> To alleviate this problem, as a part of this ticket, we propose to wake-up the samza event-loop thread when the user-thread marks the intent to shutdown the event-loop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)