You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Congxian Qiu(klion26) (JIRA)" <ji...@apache.org> on 2019/06/26 08:39:00 UTC
[jira] [Commented] (FLINK-12536) Make
BufferOrEventSequence#getNext() non-blocking
[ https://issues.apache.org/jira/browse/FLINK-12536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873088#comment-16873088 ]
Congxian Qiu(klion26) commented on FLINK-12536:
-----------------------------------------------
As discussed with [~pnowojski] offline, will use a buffer pool(default 2, can be configurable) and reuse the read thread in IOManager to read the BufferOrEvent asynchronous.
Create an issue FLINK-12994 to track the improvement for buffer processing performance in SpilledBufferOrEventSequence#getNext.
The benchmark result is as following:
Benchmark Mode Cnt Score Error Units
ConsumerBench.testBuffer thrpt 30 99.528 ± 1.923 ops/ms
ConsumerBench.testBufferWithWrap thrpt 30 827.931 ± 15.148 ops/ms
ConsumerBench.testCancelCheckpointMarkerEvent thrpt 30 40144.914 ± 1244.112 ops/ms
ConsumerBench.testCheckpointBarrierEvent thrpt 30 29784.258 ± 467.440 ops/ms
ConsumerBench.testEndOfPartitionEvent thrpt 30 48562.810 ± 1599.328 ops/ms
FileIOBench.test1SyncIO thrpt 30 359.367 ± 3.203 ops/ms
FileIOBench.testAsyncIO thrpt 30 66.573 ± 3.384 ops/ms
[1] `testBuffer` test `SpilledBufferOrEventSequence#getNext` using `MemorySegmentFactory.allocateUnpooledSegment(pageSize)`, `testBufferWithWrap` test `SpilledBufferOrEventSequence#getNext` using `MemorySegmentFactory.wrap()`, `test***Event` test all the event processing, `test1SyncIO` tests the `FileChannel` and `testAsyncIO` tests the `AsynchronousFileChannel`
[2] [https://github.com/klion26/FileIOBench/blob/8618fc4d6e745d8dd762b87505102e6cec78dc9b/src/main/java/com/klion26/data/ConsumerBench.java#L130]
[3] https://github.com/klion26/FileIOBench/blob/8618fc4d6e745d8dd762b87505102e6cec78dc9b/src/main/java/com/klion26/data/ConsumerBench.java#L169
> Make BufferOrEventSequence#getNext() non-blocking
> -------------------------------------------------
>
> Key: FLINK-12536
> URL: https://issues.apache.org/jira/browse/FLINK-12536
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Affects Versions: 1.9.0
> Reporter: Piotr Nowojski
> Assignee: Congxian Qiu(klion26)
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently it is non-blocking in case of credit-based flow control (default), however for \{{SpilledBufferOrEventSequence}} it is blocking on reading from file. We might want to consider reimplementing it to be non blocking with {{CompletableFuture<?> isAvailable()}} method.
>
> Otherwise we will block mailbox processing for the duration of reading from file - for example we will block processing time timers and potentially in the future network flushes.
>
> This is not a high priority change, since it affects non-default configuration option AND at the moment only processing time timers are planned to be moved to the mailbox for 1.9.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)