You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "danny0405 (via GitHub)" <gi...@apache.org> on 2026/05/28 03:08:24 UTC
[PR] docs: add flink RLI related configurations [hudi]
danny0405 opened a new pull request, #18869:
URL: https://github.com/apache/hudi/pull/18869
### Describe the issue this Pull Request addresses
The Flink indexing documentation on the `Indexes` page still described Flink support as limited to bucket index and Flink state index. It did not document the newer Flink record-level index paths or the configuration combinations needed to use global RLI and partitioned RLI.
This PR updates the local site docs so users can configure Flink RLI for native metadata-table access, RocksDB-backed bootstrap/cache, and dynamic bucket assignment with partitioned RLI.
### Summary and Changelog
Add Flink RLI configuration guidance to `website/docs/indexes.md`, including supported `index.type` values, related RLI tuning options, and common setup recipes.
#### Commit 1: docs: add flink RLI related configurations (`9425d5f`)
- Updates the Flink index support text to include record-level index.
- Adds `GLOBAL_RECORD_LEVEL_INDEX` and `RECORD_LEVEL_INDEX` to the Flink `index.type` table.
- Documents Flink RLI options including `index.global.enabled`, `index.bootstrap.enabled`, `index.bootstrap.rocksdb.path`, RLI cache settings, RLI write buffer/task settings, metadata compaction settings, and RLI file group count settings.
- Adds common configuration recipes for:
- Flink global RLI with native MDT access.
- Flink global RLI with local RocksDB cache.
- Dynamic bucket scaling with partitioned RLI.
### Impact
Documentation-only change. There is no public API, storage format, runtime behavior, or compatibility impact.
Users gain clearer guidance for configuring Flink RLI modes and related metadata table options.
### Risk Level
low
The change is limited to one markdown documentation page. Validation run: `git diff --check -- website/docs/indexes.md`.
### Documentation Update
This PR is the documentation update for Flink RLI configuration coverage.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: add flink RLI related configurations [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #18869:
URL: https://github.com/apache/hudi/pull/18869
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: add flink RLI related configurations [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #18869:
URL: https://github.com/apache/hudi/pull/18869#discussion_r3315149627
##########
website/docs/indexes.md:
##########
@@ -211,13 +211,38 @@ for more details. All these, support the index types mentioned [above](#addition
#### Flink based configs
-For Flink DataStream and Flink SQL, Bucket index and Flink state index are supported.
+For Flink DataStream and Flink SQL, Bucket index, Flink state index, and record-level index are supported.
Following are the basic configs that control the indexing behavior. Please refer [the Flink configurations](configurations.md#Flink-Options-advanced-configs) for advanced configs.
-| Config Name | Default | Description |
-|----------------------------|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| index.type | FLINK_STATE (Optional) | Index type of Flink write job, default is using state backed index. Possible values:<br /> <ul><li>FLINK_STATE</li><li>BUCKET</li></ul><br /> `Config Param: INDEX_TYPE` |
-| hoodie.index.bucket.engine | SIMPLE (Optional) | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. Possible Values: <br /> <ul><li>SIMPLE</li><li>CONSISTENT_HASHING</li></ul> |
+| Config Name | Default | Description |
+|-------------------------------------------------------------|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| index.type | FLINK_STATE (Optional) | Index type of Flink write job, default is using state backed index. Possible values:<br /> <ul><li>FLINK_STATE</li><li>BUCKET</li><li>GLOBAL_RECORD_LEVEL_INDEX</li><li>RECORD_LEVEL_INDEX</li></ul><br /> `Config Param: INDEX_TYPE` |
+| hoodie.index.bucket.engine | SIMPLE (Optional) | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. Possible Values: <br /> <ul><li>SIMPLE</li><li>CONSISTENT_HASHING</li></ul> |
+| metadata.enabled | true (Optional) | Enables the metadata table. Required for Flink record-level index lookups. |
Review Comment:
🤖 The default for `index.bootstrap.rocksdb.path` is listed as `(Optional)` with no value, but in `FlinkOptions.INDEX_BOOTSTRAP_ROCKSDB_PATH` it has a real default of `FileIOUtils.getDefaultSpillableMapBasePath()` (typically the system temp directory). Could you update the Default column to reflect this, since users relying on the documented "no default" may be surprised that a path is implicitly chosen for them?
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
website/docs/indexes.md:
##########
@@ -211,13 +211,38 @@ for more details. All these, support the index types mentioned [above](#addition
#### Flink based configs
-For Flink DataStream and Flink SQL, Bucket index and Flink state index are supported.
+For Flink DataStream and Flink SQL, Bucket index, Flink state index, and record-level index are supported.
Following are the basic configs that control the indexing behavior. Please refer [the Flink configurations](configurations.md#Flink-Options-advanced-configs) for advanced configs.
-| Config Name | Default | Description |
-|----------------------------|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| index.type | FLINK_STATE (Optional) | Index type of Flink write job, default is using state backed index. Possible values:<br /> <ul><li>FLINK_STATE</li><li>BUCKET</li></ul><br /> `Config Param: INDEX_TYPE` |
-| hoodie.index.bucket.engine | SIMPLE (Optional) | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. Possible Values: <br /> <ul><li>SIMPLE</li><li>CONSISTENT_HASHING</li></ul> |
+| Config Name | Default | Description |
+|-------------------------------------------------------------|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| index.type | FLINK_STATE (Optional) | Index type of Flink write job, default is using state backed index. Possible values:<br /> <ul><li>FLINK_STATE</li><li>BUCKET</li><li>GLOBAL_RECORD_LEVEL_INDEX</li><li>RECORD_LEVEL_INDEX</li></ul><br /> `Config Param: INDEX_TYPE` |
+| hoodie.index.bucket.engine | SIMPLE (Optional) | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. Possible Values: <br /> <ul><li>SIMPLE</li><li>CONSISTENT_HASHING</li></ul> |
Review Comment:
🤖 The default column shows `false (Optional)` but the description explains Flink may override this to `true` for `GLOBAL_RECORD_LEVEL_INDEX` at runtime. This dual default is easy to miss — it might help to make the override behavior more prominent (e.g., "Static default: `false`; runtime default for `GLOBAL_RECORD_LEVEL_INDEX`: `true` when unset"), so users don't assume the static default applies in all cases.
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
website/docs/indexes.md:
##########
@@ -211,13 +211,38 @@ for more details. All these, support the index types mentioned [above](#addition
#### Flink based configs
-For Flink DataStream and Flink SQL, Bucket index and Flink state index are supported.
+For Flink DataStream and Flink SQL, Bucket index, Flink state index, and record-level index are supported.
Following are the basic configs that control the indexing behavior. Please refer [the Flink configurations](configurations.md#Flink-Options-advanced-configs) for advanced configs.
-| Config Name | Default | Description |
-|----------------------------|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| index.type | FLINK_STATE (Optional) | Index type of Flink write job, default is using state backed index. Possible values:<br /> <ul><li>FLINK_STATE</li><li>BUCKET</li></ul><br /> `Config Param: INDEX_TYPE` |
-| hoodie.index.bucket.engine | SIMPLE (Optional) | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. Possible Values: <br /> <ul><li>SIMPLE</li><li>CONSISTENT_HASHING</li></ul> |
+| Config Name | Default | Description |
+|-------------------------------------------------------------|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| index.type | FLINK_STATE (Optional) | Index type of Flink write job, default is using state backed index. Possible values:<br /> <ul><li>FLINK_STATE</li><li>BUCKET</li><li>GLOBAL_RECORD_LEVEL_INDEX</li><li>RECORD_LEVEL_INDEX</li></ul><br /> `Config Param: INDEX_TYPE` |
+| hoodie.index.bucket.engine | SIMPLE (Optional) | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. Possible Values: <br /> <ul><li>SIMPLE</li><li>CONSISTENT_HASHING</li></ul> |
+| metadata.enabled | true (Optional) | Enables the metadata table. Required for Flink record-level index lookups. |
+| index.global.enabled | true (Optional) | Whether to update the old partition path when the same record key arrives with a different partition path. This must be `true` for `GLOBAL_RECORD_LEVEL_INDEX` and is set to `false` for `RECORD_LEVEL_INDEX`. |
+| index.bootstrap.enabled | false (Optional) | When `index.type=GLOBAL_RECORD_LEVEL_INDEX`, controls whether Flink bootstraps the global index into a local RocksDB backend. If not explicitly set for global RLI, Flink enables bootstrap by default. Set to `false` to force native metadata-table RLI access. |
+| index.bootstrap.rocksdb.path | (Optional) | Local directory path for the RocksDB backend used when `index.bootstrap.enabled=true`. Each task manager creates a unique subdirectory under this path. |
+| index.rli.cache.size | 256 (Optional) | Maximum memory, in MB, allocated for the record-level index cache per bucket-assign task. Applies to native metadata-table RLI access and partitioned RLI caches. |
+| index.rli.cache.concurrent.partitions.num | 2 (Optional) | Expected number of partitions whose partitioned RLI caches are updated concurrently. Used to size each partition cache when historical cache usage is unavailable. |
Review Comment:
🤖 The `index.write.tasks` row shows default `(N/A)`. Since this option resolves to the execution environment parallelism at runtime, it might read more clearly as a default of "execution environment parallelism" (matching how `read.tasks` and similar Flink configs are typically described), rather than `(N/A)` which could read as "unsupported."
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
website/docs/indexes.md:
##########
@@ -211,13 +211,38 @@ for more details. All these, support the index types mentioned [above](#addition
#### Flink based configs
-For Flink DataStream and Flink SQL, Bucket index and Flink state index are supported.
+For Flink DataStream and Flink SQL, Bucket index, Flink state index, and record-level index are supported.
Following are the basic configs that control the indexing behavior. Please refer [the Flink configurations](configurations.md#Flink-Options-advanced-configs) for advanced configs.
-| Config Name | Default | Description |
-|----------------------------|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| index.type | FLINK_STATE (Optional) | Index type of Flink write job, default is using state backed index. Possible values:<br /> <ul><li>FLINK_STATE</li><li>BUCKET</li></ul><br /> `Config Param: INDEX_TYPE` |
-| hoodie.index.bucket.engine | SIMPLE (Optional) | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. Possible Values: <br /> <ul><li>SIMPLE</li><li>CONSISTENT_HASHING</li></ul> |
+| Config Name | Default | Description |
+|-------------------------------------------------------------|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| index.type | FLINK_STATE (Optional) | Index type of Flink write job, default is using state backed index. Possible values:<br /> <ul><li>FLINK_STATE</li><li>BUCKET</li><li>GLOBAL_RECORD_LEVEL_INDEX</li><li>RECORD_LEVEL_INDEX</li></ul><br /> `Config Param: INDEX_TYPE` |
+| hoodie.index.bucket.engine | SIMPLE (Optional) | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. Possible Values: <br /> <ul><li>SIMPLE</li><li>CONSISTENT_HASHING</li></ul> |
+| metadata.enabled | true (Optional) | Enables the metadata table. Required for Flink record-level index lookups. |
+| index.global.enabled | true (Optional) | Whether to update the old partition path when the same record key arrives with a different partition path. This must be `true` for `GLOBAL_RECORD_LEVEL_INDEX` and is set to `false` for `RECORD_LEVEL_INDEX`. |
Review Comment:
🤖 The description says this cache size "Applies to native metadata-table RLI access and partitioned RLI caches." The FlinkOptions Javadoc adds an important detail: the per-checkpoint cache size is dynamically computed based on historical checkpoint averages. It might help to mention this dynamic sizing here so users understand `index.rli.cache.size` is an upper bound rather than a fixed allocation.
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
website/docs/indexes.md:
##########
@@ -211,13 +211,38 @@ for more details. All these, support the index types mentioned [above](#addition
#### Flink based configs
-For Flink DataStream and Flink SQL, Bucket index and Flink state index are supported.
+For Flink DataStream and Flink SQL, Bucket index, Flink state index, and record-level index are supported.
Following are the basic configs that control the indexing behavior. Please refer [the Flink configurations](configurations.md#Flink-Options-advanced-configs) for advanced configs.
-| Config Name | Default | Description |
-|----------------------------|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| index.type | FLINK_STATE (Optional) | Index type of Flink write job, default is using state backed index. Possible values:<br /> <ul><li>FLINK_STATE</li><li>BUCKET</li></ul><br /> `Config Param: INDEX_TYPE` |
-| hoodie.index.bucket.engine | SIMPLE (Optional) | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. Possible Values: <br /> <ul><li>SIMPLE</li><li>CONSISTENT_HASHING</li></ul> |
+| Config Name | Default | Description |
+|-------------------------------------------------------------|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| index.type | FLINK_STATE (Optional) | Index type of Flink write job, default is using state backed index. Possible values:<br /> <ul><li>FLINK_STATE</li><li>BUCKET</li><li>GLOBAL_RECORD_LEVEL_INDEX</li><li>RECORD_LEVEL_INDEX</li></ul><br /> `Config Param: INDEX_TYPE` |
+| hoodie.index.bucket.engine | SIMPLE (Optional) | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. Possible Values: <br /> <ul><li>SIMPLE</li><li>CONSISTENT_HASHING</li></ul> |
+| metadata.enabled | true (Optional) | Enables the metadata table. Required for Flink record-level index lookups. |
+| index.global.enabled | true (Optional) | Whether to update the old partition path when the same record key arrives with a different partition path. This must be `true` for `GLOBAL_RECORD_LEVEL_INDEX` and is set to `false` for `RECORD_LEVEL_INDEX`. |
+| index.bootstrap.enabled | false (Optional) | When `index.type=GLOBAL_RECORD_LEVEL_INDEX`, controls whether Flink bootstraps the global index into a local RocksDB backend. If not explicitly set for global RLI, Flink enables bootstrap by default. Set to `false` to force native metadata-table RLI access. |
+| index.bootstrap.rocksdb.path | (Optional) | Local directory path for the RocksDB backend used when `index.bootstrap.enabled=true`. Each task manager creates a unique subdirectory under this path. |
+| index.rli.cache.size | 256 (Optional) | Maximum memory, in MB, allocated for the record-level index cache per bucket-assign task. Applies to native metadata-table RLI access and partitioned RLI caches. |
+| index.rli.cache.concurrent.partitions.num | 2 (Optional) | Expected number of partitions whose partitioned RLI caches are updated concurrently. Used to size each partition cache when historical cache usage is unavailable. |
+| index.rli.lookup.minibatch.size | 1000 (Optional) | Maximum number of input records buffered for mini-batch record-index lookup. Mini-batching reduces individual metadata-table lookup calls for native global RLI access. |
+| index.rli.write.buffer.size | 100 (Optional) | Maximum memory, in MB, for the index record writer buffer. When the threshold is reached, Flink flushes index records to avoid OOM. |
+| index.write.tasks | (N/A) | Parallelism for tasks that write record-level index records. Defaults to the execution environment parallelism when not set. |
+| metadata.compaction.schedule.enabled | true (Optional) | Schedules metadata table compaction plans. |
Review Comment:
🤖 The doc states "Flink ingestion does not support deferred RLI initialization, so keep this set to `false` for Flink RLI writes." Could you confirm where this restriction is enforced in Flink? If a user sets `hoodie.metadata.record.level.index.defer.init=true` with Flink today, does the write fail loudly, or does it silently fall back? A pointer to the failure mode (or to the relevant validator) would make this guidance more actionable for on-call debugging.
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
website/docs/indexes.md:
##########
@@ -211,13 +211,38 @@ for more details. All these, support the index types mentioned [above](#addition
#### Flink based configs
-For Flink DataStream and Flink SQL, Bucket index and Flink state index are supported.
+For Flink DataStream and Flink SQL, Bucket index, Flink state index, and record-level index are supported.
Following are the basic configs that control the indexing behavior. Please refer [the Flink configurations](configurations.md#Flink-Options-advanced-configs) for advanced configs.
-| Config Name | Default | Description |
-|----------------------------|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| index.type | FLINK_STATE (Optional) | Index type of Flink write job, default is using state backed index. Possible values:<br /> <ul><li>FLINK_STATE</li><li>BUCKET</li></ul><br /> `Config Param: INDEX_TYPE` |
-| hoodie.index.bucket.engine | SIMPLE (Optional) | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. Possible Values: <br /> <ul><li>SIMPLE</li><li>CONSISTENT_HASHING</li></ul> |
+| Config Name | Default | Description |
+|-------------------------------------------------------------|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| index.type | FLINK_STATE (Optional) | Index type of Flink write job, default is using state backed index. Possible values:<br /> <ul><li>FLINK_STATE</li><li>BUCKET</li><li>GLOBAL_RECORD_LEVEL_INDEX</li><li>RECORD_LEVEL_INDEX</li></ul><br /> `Config Param: INDEX_TYPE` |
+| hoodie.index.bucket.engine | SIMPLE (Optional) | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. Possible Values: <br /> <ul><li>SIMPLE</li><li>CONSISTENT_HASHING</li></ul> |
+| metadata.enabled | true (Optional) | Enables the metadata table. Required for Flink record-level index lookups. |
+| index.global.enabled | true (Optional) | Whether to update the old partition path when the same record key arrives with a different partition path. This must be `true` for `GLOBAL_RECORD_LEVEL_INDEX` and is set to `false` for `RECORD_LEVEL_INDEX`. |
+| index.bootstrap.enabled | false (Optional) | When `index.type=GLOBAL_RECORD_LEVEL_INDEX`, controls whether Flink bootstraps the global index into a local RocksDB backend. If not explicitly set for global RLI, Flink enables bootstrap by default. Set to `false` to force native metadata-table RLI access. |
+| index.bootstrap.rocksdb.path | (Optional) | Local directory path for the RocksDB backend used when `index.bootstrap.enabled=true`. Each task manager creates a unique subdirectory under this path. |
Review Comment:
🤖 The FlinkOptions Javadoc for `index.rli.lookup.minibatch.size` notes that 1000 is also the minimum — if a smaller value is configured, the default is used instead. Worth surfacing here so users tuning down the value aren't surprised that small settings are silently ignored.
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: add flink RLI related configurations [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #18869:
URL: https://github.com/apache/hudi/pull/18869#issuecomment-4564629437
@yihua let me lande it first, then you can reconcile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org