You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "zhijiang (Jira)" <ji...@apache.org> on 2019/12/11 05:41:00 UTC
[jira] [Assigned] (FLINK-15140) Shuffle data compression does not
work with BroadcastRecordWriter.
[ https://issues.apache.org/jira/browse/FLINK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhijiang reassigned FLINK-15140:
--------------------------------
Assignee: Yingjie Cao
> Shuffle data compression does not work with BroadcastRecordWriter.
> ------------------------------------------------------------------
>
> Key: FLINK-15140
> URL: https://issues.apache.org/jira/browse/FLINK-15140
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
> Affects Versions: 1.10.0
> Reporter: Yingjie Cao
> Assignee: Yingjie Cao
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I tested the newest code of master branch last weekend with more test cases. Unfortunately, several problems were encountered, including a bug of compression.
> When BroadcastRecordWriter is used, for pipelined mode, because the compressor copies the data back to the input buffer, however, the underlying buffer is shared when BroadcastRecordWriter is used. So we can not copy the compressed buffer back to the input buffer if the underlying buffer is shared. For blocking mode, we wrongly recycle the buffer when buffer is not compressed, and the problem is also triggered when BroadcastRecordWriter is used.
> To fix the problem, for blocking shuffle, the reference counter should be maintained correctly, for pipelined shuffle, the simplest way maybe disable compression when the underlying buffer is shared. I will open a PR to fix the problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)