[jira] [Commented] (FLINK-10331) Fix unnecessary flush requests to the network stack

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614128#comment-16614128 ] 

ASF GitHub Bot commented on FLINK-10331:
----------------------------------------

NicoK opened a new pull request #6692: [FLINK-10331][network] reduce unnecesary flushing
URL: https://github.com/apache/flink/pull/6692
 
 
   ## What is the purpose of the change
   
   With the re-design of the record writer interaction with the result(sub)partitions, flush requests can currently pile up in these scenarios:
   - a previous flush request has not been completely handled yet and/or is still enqueued or
   - the network stack is still polling from this subpartition and doesn't need a new notification
   
   These lead to increased notifications in low latency settings (low output flusher intervals) which can be avoided.
   
   ## Brief change log
   
   - do not flush (again) in the scenarios mentioned above, relying on `flushRequested` and the `buffer` queue size
   - add intensive sanity checks to `SpillingAdaptiveSpanningRecordDeserializer`
   - several smaller improvement hotfixes (please see the individual commits)
   
   ## Verifying this change
   
   This change is already covered by existing tests plus a few new tests in `PipelinedSubpartitionTest`.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): **no**
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
     - The serializers: **no**
     - The runtime per-record code paths (performance sensitive): **yes** (depending on output flusher interval, rather per buffer)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
     - The S3 file system connector: **no**
   
   ## Documentation
   
     - Does this pull request introduce a new feature? **no**
     - If yes, how is the feature documented? **JavaDocs**
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Fix unnecessary flush requests to the network stack
> ---------------------------------------------------
>
>                 Key: FLINK-10331
>                 URL: https://issues.apache.org/jira/browse/FLINK-10331
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Network
>    Affects Versions: 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0
>            Reporter: Nico Kruber
>            Assignee: Nico Kruber
>            Priority: Major
>              Labels: pull-request-available
>
> With the re-design of the record writer interaction with the result(sub)partitions, flush requests can currently pile up in these scenarios:
> - a previous flush request has not been completely handled yet and/or is still enqueued or
> - the network stack is still polling from this subpartition and doesn't need a new notification
> These lead to increased notifications in low latency settings (low output flusher intervals) which can be avoided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)