[GitHub] [flink] anonymouscodeholic commented on a change in pull request #13393: [FLINK-19238] [RocksDB] Sanity check for arena block size

[GitBox <gi...@apache.org>]

anonymouscodeholic commented on a change in pull request #13393:
URL: https://github.com/apache/flink/pull/13393#discussion_r507679715



##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBMemoryControllerUtils.java
##########
@@ -95,4 +95,45 @@ static Cache createCache(long cacheCapacity, double highPriorityPoolRatio) {
 	static WriteBufferManager createWriteBufferManager(long writeBufferManagerCapacity, Cache cache) {
 		return new WriteBufferManager(writeBufferManagerCapacity, cache);
 	}
+
+	/**
+	 * Calculate the default arena block size as RocksDB calculates it in
+	 * <a href="https://github.com/dataArtisans/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/db/column_family.cc#L196"/>.
+	 *
+	 * @return the default arena block size
+	 * @param writeBufferSize the write buffer size (bytes)
+	 */
+	static long calculateRocksDBDefaultArenaBlockSize(long writeBufferSize) {
+		return writeBufferSize / 8;
+	}
+
+	/**
+	 * Calculate {@code mutable_limit_} as RocksDB calculates it in
+	 * <a href="https://github.com/dataArtisans/frocksdb/blob/FRocksDB-5.17.2/memtable/write_buffer_manager.cc#L54"/>.
+	 *
+	 * @param bufferSize write buffer size
+	 * @return mutableLimit
+	 */
+	static long calculateRocksDBMutableLimit(long bufferSize) {
+		return bufferSize * 7 / 8;
+	}
+
+	/**
+	 * RocksDB starts flushing the active memtable constantly in the case when the arena block size is greater than
+	 * mutable limit (as calculated in {@link #calculateRocksDBMutableLimit(long)}).
+	 *
+	 * <p>This happens because in such a case the check here
+	 * <a href="https://github.com/dataArtisans/frocksdb/blob/958f191d3f7276ae59b270f9db8390034d549ee0/include/rocksdb/write_buffer_manager.h#L47"/>
+	 * is always true.
+	 *
+	 * <p>This method checks that arena block size is smaller than mutable limit.
+	 *
+	 * @param arenaBlockSize Arena block size
+	 * @param mutableLimit mutable limit
+	 * @return whether arena block size is sensible
+	 */
+	@VisibleForTesting
+	static boolean validateArenaBlockSize(long arenaBlockSize, long mutableLimit) {
+		return arenaBlockSize < mutableLimit;

Review comment:
       Yes, equal is OK.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org