[jira] [Comment Edited] (FLINK-29577) Disable rocksdb wal when restore from full snapshot

["Yuan Mei (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-29577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17620706#comment-17620706 ] 

Yuan Mei edited comment on FLINK-29577 at 10/20/22 3:48 AM:
------------------------------------------------------------

[~cailiuyang] 

Would you share some performance numbers here? It would be interesting to see:
 # restore speed/time spent
 # disk usage (as you mentioned)

I can take a look at the PR if the improvement is promising.

 


was (Author: ym):
[~cailiuyang] 

Would you share some performance numbers here? It would be interesting to see:
 # restore speed/time spent
 # disk usage (as you mentioned)

I can take a look if the improvement is promising.

 

> Disable rocksdb wal when restore from full snapshot
> ---------------------------------------------------
>
>                 Key: FLINK-29577
>                 URL: https://issues.apache.org/jira/browse/FLINK-29577
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / State Backends
>            Reporter: Cai Liuyang
>            Assignee: Cai Liuyang
>            Priority: Major
>              Labels: pull-request-available
>
> For now, RocksDBFullRestoreOperation and RocksDBHeapTimersFullRestoreOperation does's pass RocksDB::WriteOptions to RocksDBWriteBatchWrapper when restore kv-data, which will use RocksDBWriteBatchWrapper‘s default WriteOptions(doesn't disable rocksdb wal explicitly, see code below), so during restoring from full snapshot, wal is enabled(use more disk and maybe affect rocksdb-write-performance when restoring)
>  
> {code:java}
> // First: RocksDBHeapTimersFullRestoreOperation::restoreKVStateData() doesn't pass WriteOptions to RocksDBWriteBatchWrapper(null as default)
> private void restoreKVStateData(
>         ThrowingIterator<KeyGroup> keyGroups,
>         Map<Integer, ColumnFamilyHandle> columnFamilies,
>         Map<Integer, HeapPriorityQueueSnapshotRestoreWrapper<?>> restoredPQStates)
>         throws IOException, RocksDBException, StateMigrationException {
>     // for all key-groups in the current state handle...
>     try (RocksDBWriteBatchWrapper writeBatchWrapper =
>             new RocksDBWriteBatchWrapper(this.rocksHandle.getDb(), writeBatchSize)) {
>         HeapPriorityQueueSnapshotRestoreWrapper<HeapPriorityQueueElement> restoredPQ = null;
>         ColumnFamilyHandle handle = null;
>    ......
> }
> // Second: RocksDBWriteBatchWrapper::flush function doesn't disable wal explicitly when user doesn't pass WriteOptions to RocksDBWriteBatchWrapper
> public void flush() throws RocksDBException {
>     if (options != null) {
>         db.write(options, batch);
>     } else {
>         // use the default WriteOptions, if wasn't provided.
>         try (WriteOptions writeOptions = new WriteOptions()) {
>             db.write(writeOptions, batch);
>         }
>     }
>     batch.clear();
> }
> {code}
>  
>  
> As we known, rocksdb's wal is usesless for flink, so i think we can disable wal for RocksDBWriteBatchWrapper's default WriteOptions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)