[jira] [Commented] (FLINK-8582) Introduce BufferConsumer concept

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355524#comment-16355524 ] 

ASF GitHub Bot commented on FLINK-8582:
---------------------------------------

GitHub user pnowojski opened a pull request:

    https://github.com/apache/flink/pull/5423

    Low latency network changes

    This big PR depends on #4552 and #5314. Main purpose of this change is to increase network throughput/performance in low latency cases (benchmark results will be posted here later). On its own, #4552 and #5314 are causing huge performance degradation for ~1ms flushing intervals (on top of already very poor Flink's performance in such case). This PR is fixing making throughput in ~1ms more or less similar to ~100ms flushing interval.
    
    ## Brief change log
    
    - #4552 and #5314 dependencies
    - bunch of hotfixes/prerequisiting fixes
    - [FLINK-8582][runtime] Introduce BufferConsumer
    - bunch of hotfixes/prerequisiting fixes
    - [FLINK-8583] Pass BufferConsumer to subpartitions
    - bunch of hotfixes/prerequisiting fixes
    - [FLINK-8591][runtime] Pass unfinished bufferConsumers to subpartitions
    - some hotfixes
    
    This last one ([FLINK-8591]) is the one commit that actually improves the performance by allowing sender to append a records to a memory segment, while `PartitionRequestQueue` in Netty is busy handling/processing/flushing previous memory segment and when it is blocked for a new credit to arrive.
    
    ## Verifying this change
    
    This change is a trivial rework ;)
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): (yes / **no**)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
      - The serializers: (yes / **no** / don't know)
      - The runtime per-record code paths (performance sensitive): (**yes** / no / don't know)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
      - The S3 file system connector: (yes / **no** / don't know)
    
    ## Documentation
    
      - Does this pull request introduce a new feature? (yes / **no**)


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/pnowojski/flink buffer-consumer

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5423.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5423
    
----
commit c6b1f913b01db5acdd1364063b3998ccecb8317d
Author: Zhijiang <wa...@...>
Date:   2017-09-30T06:36:19Z

    [FLINK-7456][network] Implement Netty sender incoming pipeline for credit-based

commit 7dd59f25a11b98e896c8a514f409e8ebb3127529
Author: Zhijiang <wa...@...>
Date:   2018-01-18T15:30:34Z

    [FLINK-7456][fixup] Add the related parameters in the document

commit 62e16f0f0a52b8f831f4438c6ed99b8aa72c0145
Author: Nico Kruber <ni...@...>
Date:   2018-01-18T19:54:14Z

    [FLINK-7456][network] revert integrating initialCredit into unannouncedCredit
    
    A PartitionRequest could actually be sent multiple times and each time it must
    contain the same credit (only one of those messages will actually be used).

commit 7bdcae31953339c75927b6c1292d164da52915f1
Author: Zhijiang <wa...@...>
Date:   2018-01-17T06:15:04Z

    [hotfix] Add the switch for keeping both the old mode and the new credit-based mode

commit 2bd12a1d3c6cf51778dec0b043fa8b06fcaf06d8
Author: Nico Kruber <ni...@...>
Date:   2018-01-18T16:24:59Z

    fixup! [hotfix] Add the switch for keeping both the old mode and the new credit-based mode

commit 321221d6b00d72100088a64c57b121e9571d5dc7
Author: Nico Kruber <ni...@...>
Date:   2018-01-18T20:27:46Z

    [FLINK-8425][network] fix SpilledSubpartitionView not protected against concurrent release calls

commit 0ed000e12a74dfdeff925f9640c0998cdacb4d26
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-12T15:01:29Z

    [hotfix][network] Invert if check in SpanningRecordSerializer to improve readability

commit 42eb594eedc6089d1193df39119f68820391a238
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-15T11:06:31Z

    [hotfix][tests] Do not hide original exception in Serialization tests

commit df89bfc7697e9d5ce2cd0fe2a334f5834f94ebc7
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-15T08:37:05Z

    [hotfix][runtime] Drop one of the two clear methods in RecordSerializer
    
    This simplifies an API a little bit

commit e753102b6ed139e18eda3221f0ff083a3f7e3610
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-16T10:08:09Z

    [hotfix][tests] Deduplicate code in LargeRecordsTest
    
    Deduplicated code was identical.

commit d24180d972ba56f53201102ac3b34242dbd08858
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-16T10:37:06Z

    [hotfix][test] Deduplicate code in LargeRecordsTest and SpanningRecordSerializationTest
    
    Dedupilcated code was effectively identical, but implemented in a slightly different way.

commit 2204d76231b9532d05318ecdbe4956b3d415ed19
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-04T15:42:46Z

    [FLINK-8582][runtime] Introduce BufferConsumer
    
    BufferConsumer will be used in the future for reading partially written
    MemorySegments. On flushes instead of requesting new MemorySegment BufferConsumer
    code will allow to continue writting to partially filled up MemmorySegment.

commit 53905ece9f2aa2cd21ad4e9e9812270753be3be8
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-18T08:14:40Z

    [hotfix][test] Simplify RecordWriterTest

commit 442980b27d9633eb4cf1031f54d30434692f35ae
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-18T09:22:23Z

    [hotfix][runtime] Refactor ResultPartition for cleaner recycle path

commit 5abd37404ed39a94b3cb08e7900e57ca9c87bdfd
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-18T09:22:52Z

    [hotfix][runtime] Fix recycleBuffer in ResultPartitionTest

commit 351bea462ddc1e0580c2462871354a3403586ce4
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-18T09:41:18Z

    [hotfix][runtime] Deduplicate code in PipelinedSubpartition

commit 3c31567d71d81b8027ea1b9aa5b4245b6d5f8585
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-18T11:09:15Z

    [hotfix][runtime] Deduplicate buffersInBacklog code in Pipelined and Spillable subtartitions

commit 6e6ce1f7082ac5798b62896db03e149adda71121
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-23T14:34:57Z

    [hotfix][runtime-tests] Immediatelly fail test when one of the futures fails

commit c1885a1c7c2a64abb8efeb244ecf896e8d701de3
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-23T16:28:13Z

    [hotfix][runtime-tests] Deduplicate CollectingResultPartitionWriters classes

commit e88c0b2618abb0f168eb366bb11224ed2d8e73e1
Author: Piotr Nowojski <pi...@...>
Date:   2018-02-07T12:34:49Z

    [hotfix][tests] Reduce mockito usage in StreamTaskTest

commit 845e8607b1176f4f8bd7b445aa1c939a77b921f3
Author: Piotr Nowojski <pi...@...>
Date:   2018-02-06T09:03:03Z

    [FLINK-8590][runtime] Drop addBufferConsumerToAllSubpartitions method

commit ebe4c7c8e7fc1b6a444712c1fe3a128f01e0b30c
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-18T09:28:48Z

    [FLINK-8583] Pass BufferConsumer to subpartitions

commit bca5301c438218def5fd56886bdc91bde657b6b6
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-24T14:41:22Z

    [FLINK-8584] handle read-only buffers in deserializer

commit da7eb1550df71b6bdb0747d4d481b36f271749f4
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-25T14:19:35Z

    [hotfix][runtime] Simplify RecordWriter code

commit 145c2fbc39547b4825d701e4115039cbc62854c4
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-25T14:31:01Z

    [hotfix][java-docs] Improve ResultSubpartition java doc

commit 386ad771def518e9adc2eb56157aad8193317a7d
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-25T14:50:42Z

    [hotfix][runtime] Simplify PipelinedSubpartition
    
    notifyBuffersAvailable is a quick call that doesn't need to be executed outside of the lock

commit d59edad1683967bba9a0cfa86bd476e0710d4876
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-25T15:23:36Z

    [hotfix][runtime] Drop unused throws IOException

commit 36a448432705e5c87b3cdc5b8d97d633a46f6dc6
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-30T08:17:50Z

    [FLINK-8586][tests] Clean up hard to maintain tests
    
    SpilledSubpartitionViewTest duplicates a lot of production logic (TestSubpartitionConsumer is a
    duplicated logic of LocalInputChannel and mix of CreditBasedSequenceNumberingViewReader with PartitionRequestQueue.
    Also it seems like most of the logic is covered by SpillableSubpartitionTest.

commit 4c28d4b8b414e3ef41aa01a070b90f33ff3eabbc
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-29T12:00:33Z

    [hotfix][tests] Properly close StreamRecordWriter in network benchmarks

commit f8b334cf9e4c9b8d5a51c3a40a71304c3f03dfc4
Author: Piotr Nowojski <pi...@...>
Date:   2018-01-30T16:03:14Z

    [hotfix][tests] Correctly set moreAvailable flag in StreamTestSingleInputGate and handle redundant data notifications

----


> Introduce BufferConsumer concept
> --------------------------------
>
>                 Key: FLINK-8582
>                 URL: https://issues.apache.org/jira/browse/FLINK-8582
>             Project: Flink
>          Issue Type: Sub-task
>            Reporter: Piotr Nowojski
>            Assignee: Piotr Nowojski
>            Priority: Major
>
> BufferConsumer will be responsible for creating Buffer slices from a shared memory segment and together with BufferBuilder it will allow to append data to a memory segment that's already visible/passed to Netty threads. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)