[I] [KV] Add RocksDB rate limiter support for TabletServer [fluss]

["platinumhamburg (via GitHub)" <gi...@apache.org>]

platinumhamburg opened a new issue, #2177:
URL: https://github.com/apache/fluss/issues/2177

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   
   ### Motivation
   
   RocksDB's flush and compaction operations can generate significant I/O pressure, potentially impacting:
   - **System stability**: Uncontrolled I/O may saturate disk bandwidth, affecting other operations
   - **Query latency**: High I/O contention can degrade read/write performance
   - **Resource predictability**: Difficult to manage multi-tenant TabletServer resource allocation
   
   RocksDB provides a built-in rate limiter mechanism to control flush and compaction write rates, but Fluss currently doesn't expose this capability to users.
   
   ### Solution
   
   ### 1. Shared Rate Limiter Architecture
   - **Server-level rate limiting**: A single rate limiter shared across all RocksDB instances on each TabletServer
   - **Resource isolation**: Prevents any single table from consuming excessive I/O bandwidth
   - **Cost efficiency**: Reduces memory overhead compared to per-instance rate limiters
   
   ### 2. Configuration Options
   **New configuration: `kv.shared-rate-limiter-bytes-per-sec`**
   - **Type**: MemorySize
   - **Default**: `0b` (disabled)
   - **Scope**: Cluster-level, dynamically updatable
   - **Example values**: `100MB`, `500MB`, `1GB`
   
   Static configuration in `conf/server.yaml`:
   kv.shared-rate-limiter-bytes-per-sec: 200mb### 3. Dynamic Configuration Support
   Rate limiter can be updated at runtime without server restart:
   - **Persistence**: Changes persisted to ZooKeeper
   - **Cluster-wide**: Applies to all TabletServers
   - **Zero-downtime**: Thread-safe updates via RocksDB API
   
   ### 3. Flink Stored Procedures
   SQL-based management interface for operators:
   
   **Set rate limiter:**
   -- Named argument (Flink 1.19+)
   CALL fluss_catalog.sys.set_shared_rocksdb_rate_limiter(rate_limit => '200MB');
   
   -- Indexed argument
   CALL fluss_catalog.sys.set_shared_rocksdb_rate_limiter('500MB');
   
   -- Disable rate limiter
   CALL fluss_catalog.sys.set_shared_rocksdb_rate_limiter('0MB');**Query current setting:**
   CALL fluss_catalog.sys.get_shared_rocksdb_rate_limiter();
   
   
   ### Anything else?
   
   _No response_
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [KV] Add RocksDB rate limiter support for TabletServer [fluss]

["wuchong (via GitHub)" <gi...@apache.org>]

wuchong closed issue #2177: [KV] Add RocksDB rate limiter support for TabletServer
URL: https://github.com/apache/fluss/issues/2177


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org