You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Rajeev-01 (via GitHub)" <gi...@apache.org> on 2026/01/30 10:00:41 UTC
[PR] [MINOR] Add database name to the metadata table config [hudi]
Rajeev-01 opened a new pull request, #18054:
URL: https://github.com/apache/hudi/pull/18054
### Describe the issue this Pull Request addresses
- This PR is adding the `hoodie.database.name` to the metadata table if main table writer config has been configured with db name.
### Summary and Changelog
- Modified the `HoodieBackedTableMetadataWriter` class to keep the database name in the metadata properties if it is present actual data table properties.
- While initializing the metadata client using `initializeMetaClient()` method, we will look into actual table config if this property is being present and adding it to metadata table config if it's present.
### Impact
- Able to keep the table naming format synced across both data table and metadata table.
### Risk Level
- None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Add database name to the metadata table config [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18054:
URL: https://github.com/apache/hudi/pull/18054#issuecomment-3822899089
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f575c50f0317a2c376e9a3a7ca8620be2339a346",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f575c50f0317a2c376e9a3a7ca8620be2339a346",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f575c50f0317a2c376e9a3a7ca8620be2339a346 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Add database name to the metadata table config [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18054:
URL: https://github.com/apache/hudi/pull/18054#issuecomment-3822916310
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f575c50f0317a2c376e9a3a7ca8620be2339a346",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11543",
"triggerID" : "f575c50f0317a2c376e9a3a7ca8620be2339a346",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f575c50f0317a2c376e9a3a7ca8620be2339a346 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11543)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Add database name to the metadata table config [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18054:
URL: https://github.com/apache/hudi/pull/18054#issuecomment-3823212270
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f575c50f0317a2c376e9a3a7ca8620be2339a346",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11543",
"triggerID" : "f575c50f0317a2c376e9a3a7ca8620be2339a346",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f575c50f0317a2c376e9a3a7ca8620be2339a346 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11543)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Add database name to the metadata table config [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on PR #18054:
URL: https://github.com/apache/hudi/pull/18054#issuecomment-3828574325
cc : @danny0405 @codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Add database name to the metadata table config [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on PR #18054:
URL: https://github.com/apache/hudi/pull/18054#issuecomment-3828789235
> @Rajeev-01 Could you explain the usecase for why database name for metadata table matters? Is it just for tooling/observability?
Yeah we need it for our catalog service to capture all the tables to store specific properties. Since if database name is not present we are missing the metadata tables when data tables has been created with `hoodie.database.name` and `hoodie.table.name` properties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Add database name to the metadata table config [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #18054:
URL: https://github.com/apache/hudi/pull/18054#discussion_r2749721305
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -949,8 +949,16 @@ private HoodieTableMetaClient initializeMetaClient() throws IOException {
.setBaseFileFormat(HoodieFileFormat.HFILE.toString())
.setRecordKeyFields(RECORD_KEY_FIELD_NAME)
.setPopulateMetaFields(DEFAULT_METADATA_POPULATE_META_FIELDS)
- .setKeyGeneratorClassProp(HoodieTableMetadataKeyGenerator.class.getCanonicalName())
- .initTable(storageConf.newInstance(), metadataWriteConfig.getBasePath());
+ .setKeyGeneratorClassProp(HoodieTableMetadataKeyGenerator.class.getCanonicalName());
+
+ // Get the database name from dataTable config
+ String databaseName = dataMetaClient.getTableConfig().getDatabaseName();
+
+ if (!StringUtils.isNullOrEmpty(databaseName)) {
+ builder.setDatabaseName(databaseName);
+ }
Review Comment:
Every data table needs to get to know database name and table name and let say if the data table previously only has `hoodie.table.name` which is of format `db.table` then metadata table should also have created with db.table format which is good. Let say if the data table properties has been updated but the metadata table still points to the older one which is fine since it has db.table format.
But the only miss will be the direct property updates which can't gets propagate to the metadata table. But I guess we can do it by adding only specific whitelist config updates needs to propagate to metadata table. WDYT? 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Add database name to the metadata table config [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #18054:
URL: https://github.com/apache/hudi/pull/18054#discussion_r2749687236
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -949,8 +949,16 @@ private HoodieTableMetaClient initializeMetaClient() throws IOException {
.setBaseFileFormat(HoodieFileFormat.HFILE.toString())
.setRecordKeyFields(RECORD_KEY_FIELD_NAME)
.setPopulateMetaFields(DEFAULT_METADATA_POPULATE_META_FIELDS)
- .setKeyGeneratorClassProp(HoodieTableMetadataKeyGenerator.class.getCanonicalName())
- .initTable(storageConf.newInstance(), metadataWriteConfig.getBasePath());
+ .setKeyGeneratorClassProp(HoodieTableMetadataKeyGenerator.class.getCanonicalName());
+
+ // Get the database name from dataTable config
Review Comment:
```suggestion
// Propagate the database name from the data table to the metadata table, if present
```
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -949,8 +949,16 @@ private HoodieTableMetaClient initializeMetaClient() throws IOException {
.setBaseFileFormat(HoodieFileFormat.HFILE.toString())
.setRecordKeyFields(RECORD_KEY_FIELD_NAME)
.setPopulateMetaFields(DEFAULT_METADATA_POPULATE_META_FIELDS)
- .setKeyGeneratorClassProp(HoodieTableMetadataKeyGenerator.class.getCanonicalName())
- .initTable(storageConf.newInstance(), metadataWriteConfig.getBasePath());
+ .setKeyGeneratorClassProp(HoodieTableMetadataKeyGenerator.class.getCanonicalName());
+
+ // Get the database name from dataTable config
+ String databaseName = dataMetaClient.getTableConfig().getDatabaseName();
+
+ if (!StringUtils.isNullOrEmpty(databaseName)) {
+ builder.setDatabaseName(databaseName);
+ }
Review Comment:
What happens if an existing metadata table was created without the db name, and later the data table is updated with a db name? Will the metadata table db name be updated? It is probably fine in most use cases if it won't be updated, but worth documenting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org