You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yun Tang (Jira)" <ji...@apache.org> on 2020/12/21 12:43:00 UTC
[jira] [Commented] (FLINK-20684) Add compression type option in
RocksDBConfigurableOptions
[ https://issues.apache.org/jira/browse/FLINK-20684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252832#comment-17252832 ]
Yun Tang commented on FLINK-20684:
----------------------------------
Before we decide to add this option in \{{RocksDBConfigurableOptions}}, we have several questions to answer:
* What kind of options we want to support. We can set the overall compression type or we can also set different compression type at different level.
* Can this option break the checkpoint compatibility? If user choose snappy as previous compression type, and then he decide to switch to LZ4 as current compression type. Can the new job restore from previous job correctly in both incremental and full checkpoint format?
I think adding this option is not something complicated, but we might meet the risk to break the checkpoint compatibility. Last but not least, we do not provide such option in Flink currently, user still could config the compression type programmatically.
> Add compression type option in RocksDBConfigurableOptions
> ---------------------------------------------------------
>
> Key: FLINK-20684
> URL: https://issues.apache.org/jira/browse/FLINK-20684
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Affects Versions: 1.11.3
> Reporter: Jiayi Liao
> Priority: Major
>
> Currently RocksDB uses snappy compression by default, but this could be a huge cost on CPU time, and for most cases we may prefer using more disk space than CPU.
> I think Flink should provide an compression type option for users to choose in different use cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)