You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Cai Liuyang (Jira)" <ji...@apache.org> on 2022/10/11 08:27:00 UTC
[jira] [Created] (FLINK-29577) Disable rocksdb wal when restore from full snapshot
Cai Liuyang created FLINK-29577:
-----------------------------------
Summary: Disable rocksdb wal when restore from full snapshot
Key: FLINK-29577
URL: https://issues.apache.org/jira/browse/FLINK-29577
Project: Flink
Issue Type: Improvement
Components: Runtime / State Backends
Reporter: Cai Liuyang
For now, RocksDBFullRestoreOperation and RocksDBHeapTimersFullRestoreOperation does's pass RocksDB::WriteOptions to RocksDBWriteBatchWrapper when restore kv-data, which will use RocksDBWriteBatchWrapper‘s default WriteOptions(doesn't disable rocksdb wal explicitly, see code below), so during restoring from full snapshot, wal is enabled(use more disk and affect rocksdb-write-performance when restoring)
{code:java}
// First: RocksDBHeapTimersFullRestoreOperation::restoreKVStateData() doesn't pass WriteOptions to RocksDBWriteBatchWrapper(null as default)
private void restoreKVStateData(
ThrowingIterator<KeyGroup> keyGroups,
Map<Integer, ColumnFamilyHandle> columnFamilies,
Map<Integer, HeapPriorityQueueSnapshotRestoreWrapper<?>> restoredPQStates)
throws IOException, RocksDBException, StateMigrationException {
// for all key-groups in the current state handle...
try (RocksDBWriteBatchWrapper writeBatchWrapper =
new RocksDBWriteBatchWrapper(this.rocksHandle.getDb(), writeBatchSize)) {
HeapPriorityQueueSnapshotRestoreWrapper<HeapPriorityQueueElement> restoredPQ = null;
ColumnFamilyHandle handle = null;
......
}
// Second: RocksDBWriteBatchWrapper::flush function doesn't disable wal explicitly when user doesn't pass WriteOptions to RocksDBWriteBatchWrapper
public void flush() throws RocksDBException {
if (options != null) {
db.write(options, batch);
} else {
// use the default WriteOptions, if wasn't provided.
try (WriteOptions writeOptions = new WriteOptions()) {
db.write(writeOptions, batch);
}
}
batch.clear();
}
{code}
As we known, rocksdb's wal is usesless for flink, so i think we can disable wal for RocksDBWriteBatchWrapper's default WriteOptions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)