You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yun Tang (Jira)" <ji...@apache.org> on 2021/10/29 08:33:00 UTC
[jira] [Commented] (FLINK-21543) when using FIFO compaction, I
found sst being deleted on the first checkpoint
[ https://issues.apache.org/jira/browse/FLINK-21543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17435847#comment-17435847 ]
Yun Tang commented on FLINK-21543:
----------------------------------
Since RocksDB's FIFO compaction would cause data lost quitely, it cannot be used in open-source Flink, I'll close this ticket.
[~zhoujira86] your special use case is intresting, and mabe you could share more information if still choosing this way.
> when using FIFO compaction, I found sst being deleted on the first checkpoint
> -----------------------------------------------------------------------------
>
> Key: FLINK-21543
> URL: https://issues.apache.org/jira/browse/FLINK-21543
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
> Reporter: xiaogang zhou
> Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
> Attachments: LOG (2), image-2021-03-03-11-35-11-458.png, image-2021-03-03-13-09-01-695.png
>
>
> 2021/03/01-18:51:01.202049 7f59042fc700 (Original Log Time 2021/03/01-18:51:01.200883) [/compaction/compaction_picker_fifo.cc:107] [_timer_state/processing_user-timers] FIFO compaction: picking file 1710 with creation time 0 for deletion
>
> the configuration is like
> currentOptions.setCompactionStyle(getCompactionStyle());
> currentOptions.setLevel0FileNumCompactionTrigger(8);
> // currentOptions.setMaxTableFilesSizeFIFO(MemorySize.parse("2gb").getBytes());
> CompactionOptionsFIFO compactionOptionsFIFO = new CompactionOptionsFIFO();
> compactionOptionsFIFO.setMaxTableFilesSize(MemorySize.parse("8gb").getBytes());
> compactionOptionsFIFO.setAllowCompaction(true);
>
> the rocksdb version is
> <dependency>
> <groupId>io.github.myasuka</groupId>
> <artifactId>frocksdbjni</artifactId>
> <version>6.10.2-ververica-3.0</version>
> </dependency>
>
> I think the problem is caused by tableproperty is lost by snapshot. Can any one suggest how i can skip this problem?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)