You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vineethNaroju (via GitHub)" <gi...@apache.org> on 2025/06/12 13:36:47 UTC
[PR] Support database and table name for Glue/ Datahub catalog [hudi]
vineethNaroju opened a new pull request, #13424:
URL: https://github.com/apache/hudi/pull/13424
### Change Logs
Added separate configs for glue and datahub to set database/table name in sync client.
### Impact
Hudi database/table name can be configured for glue/datahub catalog separately.
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
```
hoodie.datasource.meta.sync.glue.database_name: "database"
hoodie.datasource.meta.sync.glue.table_name: "table"
hoodie.meta.sync.datahub.database.name: "database"
hoodie.meta.sync.datahub.table.name: "table"
```
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-2967514019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62c0823fa0846054cc29ccf9e06c0324f480d01a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159)
* 505deb9a2de03af50c074dedcf464ae2f81c80b0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "vineethNaroju (via GitHub)" <gi...@apache.org>.
vineethNaroju commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3006925169
> > it always gets created with `hoodie.table.name`
>
> @vinishjail97 I have no access to the link, I see there are already some options like `hoodie.gcp.bigquery.sync.table_name` in the `BigQuerySyncConfig` on master:
>
> https://github.com/apache/hudi/blob/f1faabe2f577d7f33fdb0194a490e7c18b22546c/hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySyncConfig.java#L83
Yes, we want to have similar config for glue and datahub catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2306111697
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -178,6 +179,16 @@ private static GlueAsyncClient buildAsyncClient(HiveSyncConfig config) {
}
}
+ @Override
+ public String getTableName() {
+ return config.getStringOrDefault(GLUE_SYNC_TABLE_NAME, GLUE_SYNC_TABLE_NAME.getInferFunction().get().apply(config).get());
Review Comment:
not sure what do you mean.
yes, the tool will fetch it from syncClient. and this is the getter in the syncClient (AwsGlueCatalogSyncClient) that the tool will be polling into.
So, w/o fixing this, not sure how that will work.
can you help clarify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3225810210
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450)
* 2379b0ff5b0700ea9a326130d5ecf383a2b3d36f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3003546914
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2305928157
##########
hudi-sync/hudi-datahub-sync/src/main/java/org/apache/hudi/sync/datahub/DataHubSyncClient.java:
##########
@@ -97,6 +100,16 @@ public DataHubSyncClient(DataHubSyncConfig config, HoodieTableMetaClient metaCli
this.databaseName = datasetIdentifier.getDatabaseName();
Review Comment:
Same as above, the names are already here which is inferred from the `HoodieDataHubDatasetIdentifier `, we can return them directly for the getters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3233285846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
}, {
"hash" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167",
"triggerID" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"triggerType" : "PUSH"
}, {
"hash" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8180",
"triggerID" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"triggerType" : "PUSH"
}, {
"hash" : "aed83f0fca04a7b4f6f0032724055b34903e9c16",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8199",
"triggerID" : "aed83f0fca04a7b4f6f0032724055b34903e9c16",
"triggerType" : "PUSH"
}, {
"hash" : "680de0da1a0fff4b1313e71dc3a0462402837765",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8208",
"triggerID" : "680de0da1a0fff4b1313e71dc3a0462402837765",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 680de0da1a0fff4b1313e71dc3a0462402837765 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8208)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3231980720
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
}, {
"hash" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167",
"triggerID" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"triggerType" : "PUSH"
}, {
"hash" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8180",
"triggerID" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"triggerType" : "PUSH"
}, {
"hash" : "aed83f0fca04a7b4f6f0032724055b34903e9c16",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8199",
"triggerID" : "aed83f0fca04a7b4f6f0032724055b34903e9c16",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aed83f0fca04a7b4f6f0032724055b34903e9c16 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8199)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3231790537
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
}, {
"hash" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167",
"triggerID" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"triggerType" : "PUSH"
}, {
"hash" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8180",
"triggerID" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"triggerType" : "PUSH"
}, {
"hash" : "aed83f0fca04a7b4f6f0032724055b34903e9c16",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aed83f0fca04a7b4f6f0032724055b34903e9c16",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cd6ee1e49010e3147926e62472943ea322624cd5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8180)
* aed83f0fca04a7b4f6f0032724055b34903e9c16 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2306112467
##########
hudi-sync/hudi-datahub-sync/src/main/java/org/apache/hudi/sync/datahub/DataHubSyncClient.java:
##########
@@ -97,6 +100,16 @@ public DataHubSyncClient(DataHubSyncConfig config, HoodieTableMetaClient metaCli
this.databaseName = datasetIdentifier.getDatabaseName();
Review Comment:
sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3006702635
> Added separate configs for glue and datahub to set database/table name in sync client.
@vineethNaroju can you explain why we need a new options key for the db/table name as long as the existing options already work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3229700689
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
}, {
"hash" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167",
"triggerID" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"triggerType" : "PUSH"
}, {
"hash" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8180",
"triggerID" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 180c5dda39e73234ed483ab51d194fad9ba3eeee Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167)
* cd6ee1e49010e3147926e62472943ea322624cd5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8180)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2306108731
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java:
##########
@@ -124,10 +122,13 @@ public HiveSyncTool(Properties props, Configuration hadoopConf, Option<HoodieTab
}
this.config = new HiveSyncConfig(props, hadoopConfForSync);
- this.databaseName = config.getStringOrDefault(META_SYNC_DATABASE_NAME);
- this.tableName = config.getStringOrDefault(META_SYNC_TABLE_NAME);
HoodieTableMetaClient metaClient = metaClientOption.orElseGet(() -> buildMetaClient(config));
initSyncClient(config, metaClient);
+ // initSyncClient leaves syncClient as null
+ if (!Objects.isNull(this.syncClient)) {
+ this.tableName = this.syncClient.getTableName();
Review Comment:
hey @the-other-tim-brown :
do you happened to know on which case we should hit this? i.e. syncClient being null after calling `initSyncClient(config, metaClient)`. I don't see any inherited sync tools that would result in syncClient being null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3225973481
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2379b0ff5b0700ea9a326130d5ecf383a2b3d36f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2302396356
##########
hudi-aws/src/main/java/org/apache/hudi/config/GlueCatalogSyncClientConfig.java:
##########
@@ -97,4 +103,21 @@ public class GlueCatalogSyncClientConfig extends HoodieConfig {
.markAdvanced()
.withDocumentation("Glue sync may fail if the Glue table exists with partitions differing from the Hoodie table or if schema evolution is not supported by Glue."
+ "Enabling this configuration will drop and create the table to match the Hoodie config");
+
+ public static final ConfigProperty<String> GLUE_SYNC_DATABASE_NAME = ConfigProperty
+ .key(GLUE_CLIENT_PROPERTY_PREFIX + "database_name")
+ .noDefaultValue()
+ .withInferFunction(cfg -> Option.ofNullable(cfg.getString(META_SYNC_DATABASE_NAME.key()))
+ .or(() -> Option.of(cfg.getStringOrDefault(DATABASE_NAME, DATABASE_NAME.defaultValue()))))
Review Comment:
`DATABASE_NAME.defaultValue()`: the default value is null, therefore, will `Option.of()` should throw NPE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3233008054
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
}, {
"hash" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167",
"triggerID" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"triggerType" : "PUSH"
}, {
"hash" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8180",
"triggerID" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"triggerType" : "PUSH"
}, {
"hash" : "aed83f0fca04a7b4f6f0032724055b34903e9c16",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8199",
"triggerID" : "aed83f0fca04a7b4f6f0032724055b34903e9c16",
"triggerType" : "PUSH"
}, {
"hash" : "680de0da1a0fff4b1313e71dc3a0462402837765",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8208",
"triggerID" : "680de0da1a0fff4b1313e71dc3a0462402837765",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aed83f0fca04a7b4f6f0032724055b34903e9c16 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8199)
* 680de0da1a0fff4b1313e71dc3a0462402837765 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8208)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3226257631
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
}, {
"hash" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167",
"triggerID" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2379b0ff5b0700ea9a326130d5ecf383a2b3d36f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163)
* 180c5dda39e73234ed483ab51d194fad9ba3eeee Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2302196931
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -178,6 +179,16 @@ private static GlueAsyncClient buildAsyncClient(HiveSyncConfig config) {
}
}
+ @Override
+ public String getTableName() {
+ return config.getStringOrDefault(GLUE_SYNC_TABLE_NAME, GLUE_SYNC_TABLE_NAME.getInferFunction().get().apply(config).get());
Review Comment:
Note to Reviewer:
We do not instantiate or use the GlueCatalogSyncClientConfig class here and its an instance of HivesyncConfig. So, the infer func automatically does not kick in. so, I had to take this route.
Same applies for DatahubSyncConfig as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2302443247
##########
hudi-aws/src/main/java/org/apache/hudi/config/GlueCatalogSyncClientConfig.java:
##########
@@ -97,4 +103,21 @@ public class GlueCatalogSyncClientConfig extends HoodieConfig {
.markAdvanced()
.withDocumentation("Glue sync may fail if the Glue table exists with partitions differing from the Hoodie table or if schema evolution is not supported by Glue."
+ "Enabling this configuration will drop and create the table to match the Hoodie config");
+
+ public static final ConfigProperty<String> GLUE_SYNC_DATABASE_NAME = ConfigProperty
+ .key(GLUE_CLIENT_PROPERTY_PREFIX + "database_name")
+ .noDefaultValue()
+ .withInferFunction(cfg -> Option.ofNullable(cfg.getString(META_SYNC_DATABASE_NAME.key()))
+ .or(() -> Option.of(cfg.getStringOrDefault(DATABASE_NAME, DATABASE_NAME.defaultValue()))))
Review Comment:
it was supposed to be META_SYNC_DATABASE_NAME.defaultValue.
##########
hudi-sync/hudi-datahub-sync/src/main/java/org/apache/hudi/sync/datahub/config/DataHubSyncConfig.java:
##########
@@ -115,6 +119,22 @@ public class DataHubSyncConfig extends HoodieSyncConfig {
.defaultValue(true)
.markAdvanced()
.withDocumentation("Suppress exceptions during DataHub sync. This is true by default to ensure that when running inline with other jobs, the sync does not fail the job.");
+ public static final ConfigProperty<String> META_SYNC_DATAHUB_DATABASE_NAME = ConfigProperty
+ .key("hoodie.meta.sync.datahub.database.name")
+ .noDefaultValue()
+ .withInferFunction(cfg -> Option.ofNullable(cfg.getString(META_SYNC_DATABASE_NAME.key()))
+ .or(() -> Option.of(cfg.getStringOrDefault(DATABASE_NAME, META_SYNC_DATABASE_NAME.defaultValue()))))
Review Comment:
META_SYNC_DATABASE_NAME. defaultValue()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3005442912
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3006906244
> it always gets created with `hoodie.table.name`
@vinishjail97 I have no access to the link, I see there are already some options like `hoodie.gcp.bigquery.sync.table_name` in the `BigQuerySyncConfig` on master: https://github.com/apache/hudi/blob/f1faabe2f577d7f33fdb0194a490e7c18b22546c/hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySyncConfig.java#L83
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-2967251477
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62c0823fa0846054cc29ccf9e06c0324f480d01a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-2967508316
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62c0823fa0846054cc29ccf9e06c0324f480d01a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159)
* 505deb9a2de03af50c074dedcf464ae2f81c80b0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-2979029128
https://issues.apache.org/jira/browse/HUDI-9528
@vineethNaroju Use this JIRA for your PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3225805839
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450)
* 2379b0ff5b0700ea9a326130d5ecf383a2b3d36f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3229917994
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
}, {
"hash" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167",
"triggerID" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"triggerType" : "PUSH"
}, {
"hash" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8180",
"triggerID" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cd6ee1e49010e3147926e62472943ea322624cd5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8180)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3003408659
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 505deb9a2de03af50c074dedcf464ae2f81c80b0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168)
* 2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3005742776
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "vineethNaroju (via GitHub)" <gi...@apache.org>.
vineethNaroju commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3005439671
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-2967842579
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 505deb9a2de03af50c074dedcf464ae2f81c80b0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2306843762
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -178,6 +179,16 @@ private static GlueAsyncClient buildAsyncClient(HiveSyncConfig config) {
}
}
+ @Override
+ public String getTableName() {
+ return config.getStringOrDefault(GLUE_SYNC_TABLE_NAME, GLUE_SYNC_TABLE_NAME.getInferFunction().get().apply(config).get());
Review Comment:
Can we move the instantiation of the table name to the constructor, just like the databaseName?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2307485193
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java:
##########
@@ -124,10 +122,13 @@ public HiveSyncTool(Properties props, Configuration hadoopConf, Option<HoodieTab
}
this.config = new HiveSyncConfig(props, hadoopConfForSync);
- this.databaseName = config.getStringOrDefault(META_SYNC_DATABASE_NAME);
- this.tableName = config.getStringOrDefault(META_SYNC_TABLE_NAME);
HoodieTableMetaClient metaClient = metaClientOption.orElseGet(() -> buildMetaClient(config));
initSyncClient(config, metaClient);
+ // initSyncClient leaves syncClient as null
+ if (!Objects.isNull(this.syncClient)) {
+ this.tableName = this.syncClient.getTableName();
Review Comment:
This is for the `HIVE_IGNORE_EXCEPTIONS` case. In that case the client is left as null. It is some existing behavior but I don't understand its utility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3229695880
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
}, {
"hash" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167",
"triggerID" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"triggerType" : "PUSH"
}, {
"hash" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 180c5dda39e73234ed483ab51d194fad9ba3eeee Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167)
* cd6ee1e49010e3147926e62472943ea322624cd5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3235159877
Landed as part of https://github.com/apache/hudi/pull/13785
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3003411229
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 505deb9a2de03af50c074dedcf464ae2f81c80b0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168)
* 2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3006784221
> > Added separate configs for glue and datahub to set database/table name in sync client.
>
> @vineethNaroju can you explain why we need a new options key for the db/table name even though the existing options already work?
@danny0405 We support database/table names being different for other catalogs/metastores like BigQuery for example. The restriction for user right now is that for Glue/DataHub, it always gets created with `hoodie.table.name`
https://github.com/onehouseinc/hudi-internal/blob/master/hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySyncConfig.java#L74
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-2966761703
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62c0823fa0846054cc29ccf9e06c0324f480d01a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed pull request #13424: [HUDI-9528] Support database and table name for Glue/ Datahub catalog
URL: https://github.com/apache/hudi/pull/13424
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3226250524
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
}, {
"hash" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2379b0ff5b0700ea9a326130d5ecf383a2b3d36f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163)
* 180c5dda39e73234ed483ab51d194fad9ba3eeee UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2305925681
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -178,6 +179,16 @@ private static GlueAsyncClient buildAsyncClient(HiveSyncConfig config) {
}
}
+ @Override
+ public String getTableName() {
+ return config.getStringOrDefault(GLUE_SYNC_TABLE_NAME, GLUE_SYNC_TABLE_NAME.getInferFunction().get().apply(config).get());
Review Comment:
Looks like we always fetch the db and table name from the sync client from the sync tool, so I guess there is no need to infer the names again in these getters? Just initializes them well in the sync client constructor and return it here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2305930606
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java:
##########
@@ -124,10 +122,13 @@ public HiveSyncTool(Properties props, Configuration hadoopConf, Option<HoodieTab
}
this.config = new HiveSyncConfig(props, hadoopConfForSync);
- this.databaseName = config.getStringOrDefault(META_SYNC_DATABASE_NAME);
- this.tableName = config.getStringOrDefault(META_SYNC_TABLE_NAME);
HoodieTableMetaClient metaClient = metaClientOption.orElseGet(() -> buildMetaClient(config));
initSyncClient(config, metaClient);
+ // initSyncClient leaves syncClient as null
+ if (!Objects.isNull(this.syncClient)) {
+ this.tableName = this.syncClient.getTableName();
Review Comment:
the sync would be abandoned when syncClient is null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3231795168
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
}, {
"hash" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167",
"triggerID" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"triggerType" : "PUSH"
}, {
"hash" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8180",
"triggerID" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"triggerType" : "PUSH"
}, {
"hash" : "aed83f0fca04a7b4f6f0032724055b34903e9c16",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8199",
"triggerID" : "aed83f0fca04a7b4f6f0032724055b34903e9c16",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cd6ee1e49010e3147926e62472943ea322624cd5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8180)
* aed83f0fca04a7b4f6f0032724055b34903e9c16 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8199)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-2966767916
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62c0823fa0846054cc29ccf9e06c0324f480d01a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2302396356
##########
hudi-aws/src/main/java/org/apache/hudi/config/GlueCatalogSyncClientConfig.java:
##########
@@ -97,4 +103,21 @@ public class GlueCatalogSyncClientConfig extends HoodieConfig {
.markAdvanced()
.withDocumentation("Glue sync may fail if the Glue table exists with partitions differing from the Hoodie table or if schema evolution is not supported by Glue."
+ "Enabling this configuration will drop and create the table to match the Hoodie config");
+
+ public static final ConfigProperty<String> GLUE_SYNC_DATABASE_NAME = ConfigProperty
+ .key(GLUE_CLIENT_PROPERTY_PREFIX + "database_name")
+ .noDefaultValue()
+ .withInferFunction(cfg -> Option.ofNullable(cfg.getString(META_SYNC_DATABASE_NAME.key()))
+ .or(() -> Option.of(cfg.getStringOrDefault(DATABASE_NAME, DATABASE_NAME.defaultValue()))))
Review Comment:
`DATABASE_NAME.defaultValue()`: the default value is null, therefore, will `Option.of()` should throw NPE. Will it be expected?
##########
hudi-aws/src/test/java/org/apache/hudi/aws/sync/TestAWSGlueSyncClient.java:
##########
@@ -259,6 +267,50 @@ void testUpdateTableProperties() throws ExecutionException, InterruptedException
assertThrows(HoodieGlueSyncException.class, () -> awsGlueSyncClient.updateTableProperties(tableName, newTableProperties));
}
+ @Test
+ void testTableAndDatabaseName() {
+ assertEquals(GlueTestUtil.DB_NAME, awsGlueSyncClient.getDatabaseName());
+ assertEquals(GlueTestUtil.TABLE_NAME, awsGlueSyncClient.getTableName());
+
+ String dbName = "test_db1";
+ String tableName = "test_table1";
+ Properties properties = new Properties();
+ properties.setProperty(GlueCatalogSyncClientConfig.GLUE_SYNC_DATABASE_NAME.key(), dbName);
+ properties.setProperty(GlueCatalogSyncClientConfig.GLUE_SYNC_TABLE_NAME.key(), tableName);
+
+ HiveSyncConfig hiveSyncConfig = new HiveSyncConfig(properties);
+ awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, hiveSyncConfig, GlueTestUtil.getMetaClient());
+ assertEquals(dbName, awsGlueSyncClient.getDatabaseName());
+ assertEquals(tableName, awsGlueSyncClient.getTableName());
+
+ dbName = "test_db2";
+ tableName = "test_table2";
+ properties = new Properties();
+ properties.setProperty(META_SYNC_DATABASE_NAME.key(), dbName);
+ properties.setProperty(META_SYNC_TABLE_NAME.key(), tableName);
+
+ hiveSyncConfig = new HiveSyncConfig(properties);
+ awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, hiveSyncConfig, GlueTestUtil.getMetaClient());
+ assertEquals(dbName, awsGlueSyncClient.getDatabaseName());
+ assertEquals(tableName, awsGlueSyncClient.getTableName());
+
+ dbName = "test_db3";
+ tableName = "test_table3";
+ properties = new Properties();
+ properties.setProperty(DATABASE_NAME.key(), dbName);
+ properties.setProperty(HOODIE_TABLE_NAME_KEY, tableName);
+
+ hiveSyncConfig = new HiveSyncConfig(properties);
+ awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, hiveSyncConfig, GlueTestUtil.getMetaClient());
+ assertEquals(dbName, awsGlueSyncClient.getDatabaseName());
+ assertEquals(tableName, awsGlueSyncClient.getTableName());
+
+ hiveSyncConfig = new HiveSyncConfig(new Properties());
+ awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, hiveSyncConfig, GlueTestUtil.getMetaClient());
+ assertEquals(META_SYNC_DATABASE_NAME.defaultValue(), awsGlueSyncClient.getDatabaseName());
+ assertEquals(META_SYNC_TABLE_NAME.defaultValue(), awsGlueSyncClient.getTableName());
Review Comment:
for the dabase or table the value could be null. When it happen, should it be safer to throw instead of return null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2302398476
##########
hudi-sync/hudi-datahub-sync/src/main/java/org/apache/hudi/sync/datahub/config/DataHubSyncConfig.java:
##########
@@ -115,6 +119,22 @@ public class DataHubSyncConfig extends HoodieSyncConfig {
.defaultValue(true)
.markAdvanced()
.withDocumentation("Suppress exceptions during DataHub sync. This is true by default to ensure that when running inline with other jobs, the sync does not fail the job.");
+ public static final ConfigProperty<String> META_SYNC_DATAHUB_DATABASE_NAME = ConfigProperty
+ .key("hoodie.meta.sync.datahub.database.name")
+ .noDefaultValue()
+ .withInferFunction(cfg -> Option.ofNullable(cfg.getString(META_SYNC_DATABASE_NAME.key()))
+ .or(() -> Option.of(cfg.getStringOrDefault(DATABASE_NAME, META_SYNC_DATABASE_NAME.defaultValue()))))
Review Comment:
`META_SYNC_DATABASE_NAME.defaultValue()` or `DATABASE_NAME.defaultValue()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2302390133
##########
hudi-aws/src/test/java/org/apache/hudi/aws/sync/TestAWSGlueSyncClient.java:
##########
@@ -259,6 +267,50 @@ void testUpdateTableProperties() throws ExecutionException, InterruptedException
assertThrows(HoodieGlueSyncException.class, () -> awsGlueSyncClient.updateTableProperties(tableName, newTableProperties));
}
+ @Test
+ void testTableAndDatabaseName() {
+ assertEquals(GlueTestUtil.DB_NAME, awsGlueSyncClient.getDatabaseName());
+ assertEquals(GlueTestUtil.TABLE_NAME, awsGlueSyncClient.getTableName());
+
+ String dbName = "test_db1";
+ String tableName = "test_table1";
+ Properties properties = new Properties();
+ properties.setProperty(GlueCatalogSyncClientConfig.GLUE_SYNC_DATABASE_NAME.key(), dbName);
+ properties.setProperty(GlueCatalogSyncClientConfig.GLUE_SYNC_TABLE_NAME.key(), tableName);
+
+ HiveSyncConfig hiveSyncConfig = new HiveSyncConfig(properties);
+ awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, hiveSyncConfig, GlueTestUtil.getMetaClient());
+ assertEquals(dbName, awsGlueSyncClient.getDatabaseName());
+ assertEquals(tableName, awsGlueSyncClient.getTableName());
+
+ dbName = "test_db2";
+ tableName = "test_table2";
+ properties = new Properties();
+ properties.setProperty(META_SYNC_DATABASE_NAME.key(), dbName);
+ properties.setProperty(META_SYNC_TABLE_NAME.key(), tableName);
+
+ hiveSyncConfig = new HiveSyncConfig(properties);
+ awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, hiveSyncConfig, GlueTestUtil.getMetaClient());
+ assertEquals(dbName, awsGlueSyncClient.getDatabaseName());
+ assertEquals(tableName, awsGlueSyncClient.getTableName());
+
+ dbName = "test_db3";
+ tableName = "test_table3";
+ properties = new Properties();
+ properties.setProperty(DATABASE_NAME.key(), dbName);
+ properties.setProperty(HOODIE_TABLE_NAME_KEY, tableName);
+
+ hiveSyncConfig = new HiveSyncConfig(properties);
+ awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, hiveSyncConfig, GlueTestUtil.getMetaClient());
+ assertEquals(dbName, awsGlueSyncClient.getDatabaseName());
+ assertEquals(tableName, awsGlueSyncClient.getTableName());
+
+ hiveSyncConfig = new HiveSyncConfig(new Properties());
+ awsGlueSyncClient = new AWSGlueCatalogSyncClient(mockAwsGlue, hiveSyncConfig, GlueTestUtil.getMetaClient());
+ assertEquals(META_SYNC_DATABASE_NAME.defaultValue(), awsGlueSyncClient.getDatabaseName());
+ assertEquals(META_SYNC_TABLE_NAME.defaultValue(), awsGlueSyncClient.getTableName());
Review Comment:
for the dabase or table the value could be null. When it happen, should it be safer to throw instead of return null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3226445032
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
}, {
"hash" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167",
"triggerID" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 180c5dda39e73234ed483ab51d194fad9ba3eeee Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3228297120
@danny0405 : can you review this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3233001774
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6159",
"triggerID" : "62c0823fa0846054cc29ccf9e06c0324f480d01a",
"triggerType" : "PUSH"
}, {
"hash" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6168",
"triggerID" : "505deb9a2de03af50c074dedcf464ae2f81c80b0",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6439",
"triggerID" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"triggerType" : "PUSH"
}, {
"hash" : "2a3de2c3b62aecb8695f40b2feb4b31f9d6b195e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6450",
"triggerID" : "3005439671",
"triggerType" : "MANUAL"
}, {
"hash" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8163",
"triggerID" : "2379b0ff5b0700ea9a326130d5ecf383a2b3d36f",
"triggerType" : "PUSH"
}, {
"hash" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8167",
"triggerID" : "180c5dda39e73234ed483ab51d194fad9ba3eeee",
"triggerType" : "PUSH"
}, {
"hash" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8180",
"triggerID" : "cd6ee1e49010e3147926e62472943ea322624cd5",
"triggerType" : "PUSH"
}, {
"hash" : "aed83f0fca04a7b4f6f0032724055b34903e9c16",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8199",
"triggerID" : "aed83f0fca04a7b4f6f0032724055b34903e9c16",
"triggerType" : "PUSH"
}, {
"hash" : "680de0da1a0fff4b1313e71dc3a0462402837765",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "680de0da1a0fff4b1313e71dc3a0462402837765",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aed83f0fca04a7b4f6f0032724055b34903e9c16 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8199)
* 680de0da1a0fff4b1313e71dc3a0462402837765 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3225809514
hey @danny0405 : patch is ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13424:
URL: https://github.com/apache/hudi/pull/13424#issuecomment-3225339059
yes, I agree. we should have inference logic. if catalog specific db and table names are overridden, we can take it from there. if not, we should fallback to the generic db and table name.
I will work on addressing the feedback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9528] Support database and table name for Glue/ Datahub catalog [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13424:
URL: https://github.com/apache/hudi/pull/13424#discussion_r2306114729
##########
hudi-sync/hudi-datahub-sync/src/main/java/org/apache/hudi/sync/datahub/DataHubSyncClient.java:
##########
@@ -97,6 +100,16 @@ public DataHubSyncClient(DataHubSyncConfig config, HoodieTableMetaClient metaCli
this.databaseName = datasetIdentifier.getDatabaseName();
Review Comment:
address this comment of yours
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org