You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Andrey Zagrebin (JIRA)" <ji...@apache.org> on 2019/03/25 09:59:00 UTC

[jira] [Updated] (FLINK-10471) State TTL cleanup using RocksDb compaction filter

     [ https://issues.apache.org/jira/browse/FLINK-10471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrey Zagrebin updated FLINK-10471:
------------------------------------
    Description: 
In case of rocksdb backend, we can piggy back compaction using [custom compaction filter|https://github.com/facebook/rocksdb/wiki/Compaction-Filter] which checks our last access timestamp and discards expired values. It requires contributing a C++ Flink TTL specific filter to Rocksdb, like for [cassandra|https://github.com/facebook/rocksdb/blob/master/utilities/cassandra/cassandra_compaction_filter.cc]. At the moment RocksDB does not support compaction filter plugins (see [PR discussion|https://github.com/facebook/rocksdb/pull/4463]), it is under development. 

Meanwhile, we can apply to strategies to enable this feature in Flink:
 * Release and maintain a temporary fork of RocksDB for Flink: [FRocksDB|https://github.com/dataArtisans/frocksdb] and merge [TTL filter|https://github.com/dataArtisans/frocksdb/pull/1] into this fork (used in Flink 1.8)
 * Build C++ TTL filter separately, pack this C++ lib into its JNI java client jar and load it in Flink additionally to vanila RocksDB (Flink RocksDB extensions, planned for Flink 1.9)

The second strategy is more flexible in the long run.

> State TTL cleanup using RocksDb compaction filter
> -------------------------------------------------
>
>                 Key: FLINK-10471
>                 URL: https://issues.apache.org/jira/browse/FLINK-10471
>             Project: Flink
>          Issue Type: New Feature
>          Components: Runtime / State Backends
>    Affects Versions: 1.7.0
>            Reporter: Andrey Zagrebin
>            Assignee: Andrey Zagrebin
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.8.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> In case of rocksdb backend, we can piggy back compaction using [custom compaction filter|https://github.com/facebook/rocksdb/wiki/Compaction-Filter] which checks our last access timestamp and discards expired values. It requires contributing a C++ Flink TTL specific filter to Rocksdb, like for [cassandra|https://github.com/facebook/rocksdb/blob/master/utilities/cassandra/cassandra_compaction_filter.cc]. At the moment RocksDB does not support compaction filter plugins (see [PR discussion|https://github.com/facebook/rocksdb/pull/4463]), it is under development. 
> Meanwhile, we can apply to strategies to enable this feature in Flink:
>  * Release and maintain a temporary fork of RocksDB for Flink: [FRocksDB|https://github.com/dataArtisans/frocksdb] and merge [TTL filter|https://github.com/dataArtisans/frocksdb/pull/1] into this fork (used in Flink 1.8)
>  * Build C++ TTL filter separately, pack this C++ lib into its JNI java client jar and load it in Flink additionally to vanila RocksDB (Flink RocksDB extensions, planned for Flink 1.9)
> The second strategy is more flexible in the long run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)