You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2023/02/09 01:05:42 UTC
[GitHub] [hudi] nsivabalan opened a new pull request, #7901: [HUDI-5665] Adding support to re-use table configs
nsivabalan opened a new pull request, #7901:
URL: https://github.com/apache/hudi/pull/7901
### Change Logs
- As of now, we expect users to set some of the mandatory fields in every write. For eg, record keys, partition path etc. These cannot change for a given table and gets serialized into table config. In this patch, we are adding support to re-use table configs. So, users can set these configs only in the first commit for a given table. Subsequent writes might re-use from table config if not explicitly set by the user.
### Impact
Improves user experience as users don't need to set N no of configs repeatedly.
### Risk level (write none, low medium or high below)
low.
### Documentation Update
Might need to call it out in our release highlight, that users don't need to set the table configs again and again.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1449141326
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
}, {
"hash" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486)
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
* 05db189b2e1765cf769c019000c2d3aa8829332a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1449135186
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486)
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7901:
URL: https://github.com/apache/hudi/pull/7901#discussion_r1115179259
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -121,15 +122,19 @@ object HoodieSparkSqlWriter {
val fs = basePath.getFileSystem(sparkContext.hadoopConfiguration)
tableExists = fs.exists(new Path(basePath, HoodieTableMetaClient.METAFOLDER_NAME))
var tableConfig = getHoodieTableConfig(sparkContext, path, hoodieTableConfigOpt)
- val (parameters, hoodieConfig) = mergeParamsAndGetHoodieConfig(optParams, tableConfig, mode)
- val originKeyGeneratorClassName = HoodieWriterUtils.getOriginKeyGenerator(parameters)
+ // get initial params and validate configs
+ val initialRawParams = getInitialParams(optParams)
Review Comment:
I have taken a stab at fixing the naming. can you check now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1423872095
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1448591866
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa531e8154c5eec87854ce75c6f3183bbe62a695 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464)
* 31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1447523473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461)
* fa531e8154c5eec87854ce75c6f3183bbe62a695 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1423467799
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1442790708
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ee4a7ebbf09c02b2bd81c425f4656e783f815c7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #7901:
URL: https://github.com/apache/hudi/pull/7901#discussion_r1113876340
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -38,26 +38,23 @@ import scala.collection.JavaConverters._
*/
object HoodieWriterUtils {
- def javaParametersWithWriteDefaults(parameters: java.util.Map[String, String]): java.util.Map[String, String] = {
+ /*def javaParametersWithWriteDefaults(parameters: java.util.Map[String, String]): java.util.Map[String, String] = {
Review Comment:
let's remove this?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -86,6 +83,20 @@ object HoodieWriterUtils {
hoodieConfig.setDefaultValue(RECONCILE_SCHEMA)
hoodieConfig.setDefaultValue(DROP_PARTITION_COLUMNS)
hoodieConfig.setDefaultValue(KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED)
+ if (isFreshTable) { // only set default values for a fresh table. these might be used for config validation in subsequent commits and hence
+ // we should not setting defaults. (infer function could result in setting wrong defaults)
+ hoodieConfig.setDefaultValue(PAYLOAD_CLASS_NAME)
+ hoodieConfig.setDefaultValue(KEYGENERATOR_CLASS_NAME)
Review Comment:
What about payload ordering/precombine field, record key and partitionpath fields? They don't have any defaults, is it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1453672297
CI is green
<img width="1186" alt="image" src="https://user-images.githubusercontent.com/513218/222755326-1e1e9741-df9d-454e-801f-09829c236bd1.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1452517030
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
}, {
"hash" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15495",
"triggerID" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"triggerType" : "PUSH"
}, {
"hash" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493",
"triggerID" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"triggerType" : "PUSH"
}, {
"hash" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15526",
"triggerID" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"triggerType" : "PUSH"
}, {
"hash" : "b0ddb5d1365ec41ce3890fe33a1aecba4a506b40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0ddb5d1365ec41ce3890fe33a1aecba4a506b40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
* 3d36b5c0206e3cab5d1551f22ae41082f3cc7354 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15526)
* b0ddb5d1365ec41ce3890fe33a1aecba4a506b40 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1447482447
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da53ca2480f60edf61bced07304373dae4a34332 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449)
* bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1449084081
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1446918127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ee4a7ebbf09c02b2bd81c425f4656e783f815c7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373)
* ab532211ea0aa3e15604aa60582801c3111246bf UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1447141572
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab532211ea0aa3e15604aa60582801c3111246bf Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447)
* da53ca2480f60edf61bced07304373dae4a34332 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1451149406
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
}, {
"hash" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15495",
"triggerID" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"triggerType" : "PUSH"
}, {
"hash" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493",
"triggerID" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"triggerType" : "PUSH"
}, {
"hash" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
* 172d68e4bbca0cbcd87a734676670884011e42ef Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493)
* 3d36b5c0206e3cab5d1551f22ae41082f3cc7354 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7901:
URL: https://github.com/apache/hudi/pull/7901#discussion_r1115179371
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -86,6 +83,20 @@ object HoodieWriterUtils {
hoodieConfig.setDefaultValue(RECONCILE_SCHEMA)
hoodieConfig.setDefaultValue(DROP_PARTITION_COLUMNS)
hoodieConfig.setDefaultValue(KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED)
+ if (isFreshTable) { // only set default values for a fresh table. these might be used for config validation in subsequent commits and hence
+ // we should not setting defaults. (infer function could result in setting wrong defaults)
+ hoodieConfig.setDefaultValue(PAYLOAD_CLASS_NAME)
+ hoodieConfig.setDefaultValue(KEYGENERATOR_CLASS_NAME)
Review Comment:
I ran into some issues around precombine and hence left it out. Let me try to recollect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1447471156
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da53ca2480f60edf61bced07304373dae4a34332 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1447488211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da53ca2480f60edf61bced07304373dae4a34332 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449)
* bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1440986768
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046)
* c55e96ff2e67739855776b50b3a88dd43bfc2f9d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7901:
URL: https://github.com/apache/hudi/pull/7901#discussion_r1114602192
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -830,6 +830,33 @@ object DataSourceOptionsHelper {
translatedOpt.toMap
}
+ /**
+ * Some config keys differ from what user sets and whats part of table Config. this method assists in fetching the
+ * right table config and populating write configs.
+ * @param tableConfig table config of interest.
+ * @param params incoming write params.
+ * @return missing params that needs to be added to incoming write params
+ */
+ def loadParamsFromTableConfig(tableConfig: HoodieTableConfig, params: Map[String, String]) : Map[String, String] = {
Review Comment:
sure. will take a look and fix them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1440355311
@xushiyan : this is to improve the usability. wrt table configs, they are not allowed to be modified once created. they are deemed to be immutable. hence we thought we can simplify things for user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1440360040
@codope : table configs are immutable in general. thats why. its like database name, tablename or primary key in rdbms world.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #7901:
URL: https://github.com/apache/hudi/pull/7901#discussion_r1120559992
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala:
##########
@@ -75,7 +75,7 @@ trait ProvidesHoodieConfig extends Logging {
HiveSyncConfigHolder.HIVE_SUPPORT_TIMESTAMP_TYPE.key -> hiveSyncConfig.getBoolean(HiveSyncConfigHolder.HIVE_SUPPORT_TIMESTAMP_TYPE).toString
)
- val overridingOpts = Map[String, String](
+ var overridingOpts = Map[String, String](
Review Comment:
can we stick with `val` instead of `var` ? it's safer and less error-prone
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -55,9 +51,6 @@ object HoodieWriterUtils {
val hoodieConfig: HoodieConfig = new HoodieConfig(props)
hoodieConfig.setDefaultValue(OPERATION)
hoodieConfig.setDefaultValue(TABLE_TYPE)
- hoodieConfig.setDefaultValue(PRECOMBINE_FIELD)
- hoodieConfig.setDefaultValue(PAYLOAD_CLASS_NAME)
- hoodieConfig.setDefaultValue(KEYGENERATOR_CLASS_NAME)
Review Comment:
unnecessary move
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -1024,8 +1029,9 @@ object HoodieSparkSqlWriter {
private def getHoodieTableConfig(sparkContext: SparkContext,
Review Comment:
javadoc to explain the usage will help demystify it - it's internally more complicated than the name `getHoodieTableConfig` suggests
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -86,6 +79,17 @@ object HoodieWriterUtils {
hoodieConfig.setDefaultValue(RECONCILE_SCHEMA)
hoodieConfig.setDefaultValue(DROP_PARTITION_COLUMNS)
hoodieConfig.setDefaultValue(KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED)
+ hoodieConfig.setDefaultValue(PAYLOAD_CLASS_NAME)
+ hoodieConfig.setDefaultValue(KEYGENERATOR_CLASS_NAME)
+ Map() ++ hoodieConfig.getProps.asScala ++ globalProps ++ DataSourceOptionsHelper.translateConfigurations(parameters)
+ }
+
+ def getParamsWithAlternatives(parameters: Map[String, String]): Map[String, String] = {
+ val globalProps = DFSPropertiesConfiguration.getGlobalProps.asScala
+ val props = new Properties()
+ props.putAll(parameters)
+ val hoodieConfig: HoodieConfig = new HoodieConfig(props)
+ // do not set any default as this is called before validation.
Review Comment:
rename `parameters` to `overridingParams` makes the logic clearer. this applies to `parametersWithWriteDefaults()` too.
`WithAlternatives` is not clear enough. you meant the datasource's shorter form of option keys? this looks overlapping with `getParamsWithoutDefaults()` which also translate configs. Can we merge these 2?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala:
##########
@@ -117,10 +121,7 @@ trait ProvidesHoodieConfig extends Logging {
val partitionFieldsStr = hoodieCatalogTable.partitionFields.mkString(",")
- // NOTE: Here we fallback to "" to make sure that null value is not overridden with
- // default value ("ts")
- // TODO(HUDI-3456) clean up
- val preCombineField = hoodieCatalogTable.preCombineKey.getOrElse("")
+ val preCombineField = hoodieCatalogTable.preCombineKey.getOrElse(null)
Review Comment:
does this cause a bug? or just a clean up after we make preCombineKey optional?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala:
##########
@@ -197,17 +198,19 @@ trait ProvidesHoodieConfig extends Logging {
HoodieSyncConfig.META_SYNC_PARTITION_EXTRACTOR_CLASS.key -> hiveSyncConfig.getStringOrDefault(HoodieSyncConfig.META_SYNC_PARTITION_EXTRACTOR_CLASS)
)
- val overridingOpts = extraOptions ++ Map(
+ var overridingOpts = extraOptions ++ Map(
"path" -> path,
TABLE_TYPE.key -> tableType,
TBL_NAME.key -> hoodieCatalogTable.tableName,
OPERATION.key -> operation,
HIVE_STYLE_PARTITIONING.key -> hiveStylePartitioningEnable,
URL_ENCODE_PARTITIONING.key -> urlEncodePartitioning,
RECORDKEY_FIELD.key -> hoodieCatalogTable.primaryKeys.mkString(","),
- PRECOMBINE_FIELD.key -> preCombineField,
PARTITIONPATH_FIELD.key -> partitionFieldsStr
)
+ if (preCombineField != null) {
+ overridingOpts = overridingOpts ++ Map(PRECOMBINE_FIELD.key -> preCombineField)
+ }
Review Comment:
this is actually not needed - `combineOptions()` filters out null values
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -86,6 +79,17 @@ object HoodieWriterUtils {
hoodieConfig.setDefaultValue(RECONCILE_SCHEMA)
hoodieConfig.setDefaultValue(DROP_PARTITION_COLUMNS)
hoodieConfig.setDefaultValue(KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED)
+ hoodieConfig.setDefaultValue(PAYLOAD_CLASS_NAME)
+ hoodieConfig.setDefaultValue(KEYGENERATOR_CLASS_NAME)
+ Map() ++ hoodieConfig.getProps.asScala ++ globalProps ++ DataSourceOptionsHelper.translateConfigurations(parameters)
+ }
+
+ def getParamsWithAlternatives(parameters: Map[String, String]): Map[String, String] = {
+ val globalProps = DFSPropertiesConfiguration.getGlobalProps.asScala
+ val props = new Properties()
+ props.putAll(parameters)
+ val hoodieConfig: HoodieConfig = new HoodieConfig(props)
+ // do not set any default as this is called before validation.
Review Comment:
move this to javadoc so it's clear to dev when to use this
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -150,6 +154,10 @@ object HoodieWriterUtils {
&& datasourcePreCombineKey != tableConfigPreCombineKey) {
diffConfigs.append(s"PreCombineKey:\t$datasourcePreCombineKey\t$tableConfigPreCombineKey\n")
}
+ // not set in tableConfig, but incoming has some valid value
+ if (tableConfigPreCombineKey == null && datasourcePreCombineKey != null) {
Review Comment:
this can be merged with above condition check right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #7901:
URL: https://github.com/apache/hudi/pull/7901#discussion_r1120657881
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -1037,17 +1043,23 @@ object HoodieSparkSqlWriter {
private def mergeParamsAndGetHoodieConfig(optParams: Map[String, String],
tableConfig: HoodieTableConfig, mode: SaveMode): (Map[String, String], HoodieConfig) = {
val translatedOptions = DataSourceWriteOptions.translateSqlOptions(optParams)
- val mergedParams = mutable.Map.empty ++ HoodieWriterUtils.parametersWithWriteDefaults(translatedOptions)
+ var translatedOptsWithMappedTableConfig = mutable.Map.empty ++ translatedOptions.toMap
+ if (tableConfig != null && mode != SaveMode.Overwrite) {
+ // for missing write configs corresponding to table configs, fill them up.
+ fetchMissingWriteConfigsFromTableConfig(tableConfig, optParams).foreach((kv) => translatedOptsWithMappedTableConfig += (kv._1 -> kv._2))
+ }
+ val mergedParams = mutable.Map.empty ++ HoodieWriterUtils.parametersWithWriteDefaults(translatedOptsWithMappedTableConfig.toMap)
if (!mergedParams.contains(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME.key)
&& mergedParams.contains(KEYGENERATOR_CLASS_NAME.key)) {
- mergedParams(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME.key) = mergedParams(KEYGENERATOR_CLASS_NAME.key)
+ mergedParams(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME.key) = mergedParams(DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME.key)
Review Comment:
should just keep using the static import `KEYGENERATOR_CLASS_NAME`
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestColumnStatsIndex.scala:
##########
@@ -95,6 +96,7 @@ class TestColumnStatsIndex extends HoodieSparkClientTestBase {
DataSourceWriteOptions.TABLE_TYPE.key -> testCase.tableType.toString,
RECORDKEY_FIELD.key -> "c1",
PRECOMBINE_FIELD.key -> "c1",
+ DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME.key() -> classOf[NonpartitionedKeyGenerator].getCanonicalName,
Review Comment:
not getting this additionally set config - we don't expect any user-side changes, as in we only re-use table configs if not set. am i missing sth here?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/cdc/TestCDCDataFrameSuite.scala:
##########
@@ -433,6 +434,7 @@ class TestCDCDataFrameSuite extends HoodieCDCTestBase {
def testDataSourceWriteWithPartitionField(tableType: String, loggingMode: String): Unit = {
val options = commonOpts ++ Map(
DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "partition",
+ DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME.key() -> classOf[SimpleKeyGenerator].getCanonicalName,
Review Comment:
ditto
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -202,7 +206,7 @@ object HoodieSparkSqlWriter {
.setRecordKeyFields(hoodieConfig.getString(RECORDKEY_FIELD))
.setCDCEnabled(hoodieConfig.getBooleanOrDefault(HoodieTableConfig.CDC_ENABLED))
.setCDCSupplementalLoggingMode(hoodieConfig.getStringOrDefault(HoodieTableConfig.CDC_SUPPLEMENTAL_LOGGING_MODE))
- .setKeyGeneratorClassProp(originKeyGeneratorClassName)
+ .setKeyGeneratorClassProp(hoodieConfig.getString(DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME.key))
Review Comment:
static import to reduce verbosity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1447508000
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1447557668
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461)
* fa531e8154c5eec87854ce75c6f3183bbe62a695 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #7901:
URL: https://github.com/apache/hudi/pull/7901#discussion_r1113711166
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -830,6 +830,33 @@ object DataSourceOptionsHelper {
translatedOpt.toMap
}
+ /**
+ * Some config keys differ from what user sets and whats part of table Config. this method assists in fetching the
+ * right table config and populating write configs.
+ * @param tableConfig table config of interest.
+ * @param params incoming write params.
+ * @return missing params that needs to be added to incoming write params
+ */
+ def loadParamsFromTableConfig(tableConfig: HoodieTableConfig, params: Map[String, String]) : Map[String, String] = {
Review Comment:
the method reads like loading straight from table config but in fact it's conditioned on input params. we should separate the loading from merging, or name the method properly
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -121,15 +122,19 @@ object HoodieSparkSqlWriter {
val fs = basePath.getFileSystem(sparkContext.hadoopConfiguration)
tableExists = fs.exists(new Path(basePath, HoodieTableMetaClient.METAFOLDER_NAME))
var tableConfig = getHoodieTableConfig(sparkContext, path, hoodieTableConfigOpt)
- val (parameters, hoodieConfig) = mergeParamsAndGetHoodieConfig(optParams, tableConfig, mode)
- val originKeyGeneratorClassName = HoodieWriterUtils.getOriginKeyGenerator(parameters)
+ // get initial params and validate configs
+ val initialRawParams = getInitialParams(optParams)
Review Comment:
I found names like `getInitialParams()`, `initialRawParams` and `getRawInitialParams()` pretty unclear to explain what the "stage" the params are. i suggest look for existing examples in the codebase and stick with the same naming patterns. on top of my head, i've seen `optParams` is the original user-input params, there is `finalOptParams`. Variables in this part worth spending some time crafting good names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1447127710
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ee4a7ebbf09c02b2bd81c425f4656e783f815c7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373)
* ab532211ea0aa3e15604aa60582801c3111246bf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447)
* da53ca2480f60edf61bced07304373dae4a34332 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1452686051
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
}, {
"hash" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15495",
"triggerID" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"triggerType" : "PUSH"
}, {
"hash" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493",
"triggerID" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"triggerType" : "PUSH"
}, {
"hash" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15526",
"triggerID" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"triggerType" : "PUSH"
}, {
"hash" : "b0ddb5d1365ec41ce3890fe33a1aecba4a506b40",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15538",
"triggerID" : "b0ddb5d1365ec41ce3890fe33a1aecba4a506b40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
* b0ddb5d1365ec41ce3890fe33a1aecba4a506b40 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15538)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1449191510
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
}, {
"hash" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15495",
"triggerID" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"triggerType" : "PUSH"
}, {
"hash" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493",
"triggerID" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486)
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
* 05db189b2e1765cf769c019000c2d3aa8829332a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15495)
* 172d68e4bbca0cbcd87a734676670884011e42ef Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1452915990
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
}, {
"hash" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15495",
"triggerID" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"triggerType" : "PUSH"
}, {
"hash" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493",
"triggerID" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"triggerType" : "PUSH"
}, {
"hash" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15526",
"triggerID" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"triggerType" : "PUSH"
}, {
"hash" : "b0ddb5d1365ec41ce3890fe33a1aecba4a506b40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15538",
"triggerID" : "b0ddb5d1365ec41ce3890fe33a1aecba4a506b40",
"triggerType" : "PUSH"
}, {
"hash" : "d86666046498c1811e9e0a446cef557514007f47",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15540",
"triggerID" : "d86666046498c1811e9e0a446cef557514007f47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
* d86666046498c1811e9e0a446cef557514007f47 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15540)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1447527872
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461)
* fa531e8154c5eec87854ce75c6f3183bbe62a695 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1449326033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
}, {
"hash" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15495",
"triggerID" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"triggerType" : "PUSH"
}, {
"hash" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493",
"triggerID" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
* 05db189b2e1765cf769c019000c2d3aa8829332a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15495)
* 172d68e4bbca0cbcd87a734676670884011e42ef Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1448602788
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa531e8154c5eec87854ce75c6f3183bbe62a695 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464)
* 31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1442703792
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c55e96ff2e67739855776b50b3a88dd43bfc2f9d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347)
* 0ee4a7ebbf09c02b2bd81c425f4656e783f815c7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1441098686
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c55e96ff2e67739855776b50b3a88dd43bfc2f9d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7901:
URL: https://github.com/apache/hudi/pull/7901#discussion_r1114603509
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -121,15 +122,19 @@ object HoodieSparkSqlWriter {
val fs = basePath.getFileSystem(sparkContext.hadoopConfiguration)
tableExists = fs.exists(new Path(basePath, HoodieTableMetaClient.METAFOLDER_NAME))
var tableConfig = getHoodieTableConfig(sparkContext, path, hoodieTableConfigOpt)
- val (parameters, hoodieConfig) = mergeParamsAndGetHoodieConfig(optParams, tableConfig, mode)
- val originKeyGeneratorClassName = HoodieWriterUtils.getOriginKeyGenerator(parameters)
+ // get initial params and validate configs
+ val initialRawParams = getInitialParams(optParams)
Review Comment:
yes, I too felt that. let me take a look at all of them and fix the naming properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1449186706
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
}, {
"hash" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"triggerType" : "PUSH"
}, {
"hash" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486)
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
* 05db189b2e1765cf769c019000c2d3aa8829332a UNKNOWN
* 172d68e4bbca0cbcd87a734676670884011e42ef UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1447002383
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ee4a7ebbf09c02b2bd81c425f4656e783f815c7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373)
* ab532211ea0aa3e15604aa60582801c3111246bf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1447518311
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* da53ca2480f60edf61bced07304373dae4a34332 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449)
* bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1451182596
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
}, {
"hash" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15495",
"triggerID" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"triggerType" : "PUSH"
}, {
"hash" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493",
"triggerID" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"triggerType" : "PUSH"
}, {
"hash" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15526",
"triggerID" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
* 172d68e4bbca0cbcd87a734676670884011e42ef Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493)
* 3d36b5c0206e3cab5d1551f22ae41082f3cc7354 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15526)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1451326574
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
}, {
"hash" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15495",
"triggerID" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"triggerType" : "PUSH"
}, {
"hash" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493",
"triggerID" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"triggerType" : "PUSH"
}, {
"hash" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15526",
"triggerID" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
* 3d36b5c0206e3cab5d1551f22ae41082f3cc7354 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15526)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1452525298
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
}, {
"hash" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15495",
"triggerID" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"triggerType" : "PUSH"
}, {
"hash" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493",
"triggerID" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"triggerType" : "PUSH"
}, {
"hash" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15526",
"triggerID" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"triggerType" : "PUSH"
}, {
"hash" : "b0ddb5d1365ec41ce3890fe33a1aecba4a506b40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15538",
"triggerID" : "b0ddb5d1365ec41ce3890fe33a1aecba4a506b40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
* 3d36b5c0206e3cab5d1551f22ae41082f3cc7354 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15526)
* b0ddb5d1365ec41ce3890fe33a1aecba4a506b40 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15538)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1452769509
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
}, {
"hash" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15495",
"triggerID" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"triggerType" : "PUSH"
}, {
"hash" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493",
"triggerID" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"triggerType" : "PUSH"
}, {
"hash" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15526",
"triggerID" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"triggerType" : "PUSH"
}, {
"hash" : "b0ddb5d1365ec41ce3890fe33a1aecba4a506b40",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15538",
"triggerID" : "b0ddb5d1365ec41ce3890fe33a1aecba4a506b40",
"triggerType" : "PUSH"
}, {
"hash" : "d86666046498c1811e9e0a446cef557514007f47",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15540",
"triggerID" : "d86666046498c1811e9e0a446cef557514007f47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
* b0ddb5d1365ec41ce3890fe33a1aecba4a506b40 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15538)
* d86666046498c1811e9e0a446cef557514007f47 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15540)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1442698689
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c55e96ff2e67739855776b50b3a88dd43bfc2f9d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347)
* 0ee4a7ebbf09c02b2bd81c425f4656e783f815c7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1440980804
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046)
* c55e96ff2e67739855776b50b3a88dd43bfc2f9d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1423472994
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #7901:
URL: https://github.com/apache/hudi/pull/7901#discussion_r1118241518
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -120,16 +121,19 @@ object HoodieSparkSqlWriter {
val fs = basePath.getFileSystem(sparkContext.hadoopConfiguration)
tableExists = fs.exists(new Path(basePath, HoodieTableMetaClient.METAFOLDER_NAME))
- var tableConfig = getHoodieTableConfig(sparkContext, path, hoodieTableConfigOpt)
- val (parameters, hoodieConfig) = mergeParamsAndGetHoodieConfig(optParams, tableConfig, mode)
- val originKeyGeneratorClassName = HoodieWriterUtils.getOriginKeyGenerator(parameters)
+ var tableConfig = if (mode == SaveMode.Overwrite || !tableExists) null else getHoodieTableConfig(sparkContext, path, hoodieTableConfigOpt)
Review Comment:
`tableExists` is handled in `getHoodieTableConfig` already. Also it's too early to check `SaveMode` here, which leads to confusion about why tableConfig being null while on storage it's non-null.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -86,6 +79,20 @@ object HoodieWriterUtils {
hoodieConfig.setDefaultValue(RECONCILE_SCHEMA)
hoodieConfig.setDefaultValue(DROP_PARTITION_COLUMNS)
hoodieConfig.setDefaultValue(KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED)
+ if (isFreshTable) { // only set default values for a fresh table. these might be used for config validation in subsequent commits and hence
+ // we should not be setting any defaults. (infer function could result in setting wrong defaults)
Review Comment:
> should not be setting any defaults.
confused by this comment. aren't you setting defaults below?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -658,7 +662,8 @@ object HoodieSparkSqlWriter {
val sparkContext = sqlContext.sparkContext
val fs = basePath.getFileSystem(sparkContext.hadoopConfiguration)
tableExists = fs.exists(new Path(basePath, HoodieTableMetaClient.METAFOLDER_NAME))
- val tableConfig = getHoodieTableConfig(sparkContext, path, hoodieTableConfigOpt)
+ if (mode == SaveMode.Overwrite || !tableExists) null else getHoodieTableConfig(sparkContext, path, hoodieTableConfigOpt)
+ val tableConfig = if (mode == SaveMode.Overwrite || !tableExists) null else getHoodieTableConfig(sparkContext, path, hoodieTableConfigOpt)
Review Comment:
some repeated logic, should try some refactoring here.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -1021,17 +1026,20 @@ object HoodieSparkSqlWriter {
private def mergeParamsAndGetHoodieConfig(optParams: Map[String, String],
tableConfig: HoodieTableConfig, mode: SaveMode): (Map[String, String], HoodieConfig) = {
val translatedOptions = DataSourceWriteOptions.translateSqlOptions(optParams)
- val mergedParams = mutable.Map.empty ++ HoodieWriterUtils.parametersWithWriteDefaults(translatedOptions)
+ val mergedParams = mutable.Map.empty ++ HoodieWriterUtils.parametersWithWriteDefaults(translatedOptions, tableConfig == null)
Review Comment:
it looks unintuitive to grasp tableConfig == null indicates fresh table or table to be init/re-written.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1447878562
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa531e8154c5eec87854ce75c6f3183bbe62a695 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1449431572
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
}, {
"hash" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15495",
"triggerID" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"triggerType" : "PUSH"
}, {
"hash" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493",
"triggerID" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
* 172d68e4bbca0cbcd87a734676670884011e42ef Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7901:
URL: https://github.com/apache/hudi/pull/7901#discussion_r1120961044
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -86,6 +79,17 @@ object HoodieWriterUtils {
hoodieConfig.setDefaultValue(RECONCILE_SCHEMA)
hoodieConfig.setDefaultValue(DROP_PARTITION_COLUMNS)
hoodieConfig.setDefaultValue(KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED)
+ hoodieConfig.setDefaultValue(PAYLOAD_CLASS_NAME)
+ hoodieConfig.setDefaultValue(KEYGENERATOR_CLASS_NAME)
+ Map() ++ hoodieConfig.getProps.asScala ++ globalProps ++ DataSourceOptionsHelper.translateConfigurations(parameters)
+ }
+
+ def getParamsWithAlternatives(parameters: Map[String, String]): Map[String, String] = {
+ val globalProps = DFSPropertiesConfiguration.getGlobalProps.asScala
+ val props = new Properties()
+ props.putAll(parameters)
+ val hoodieConfig: HoodieConfig = new HoodieConfig(props)
+ // do not set any default as this is called before validation.
Review Comment:
yes, removed the other methed. we just have 1 method now.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala:
##########
@@ -117,10 +121,7 @@ trait ProvidesHoodieConfig extends Logging {
val partitionFieldsStr = hoodieCatalogTable.partitionFields.mkString(",")
- // NOTE: Here we fallback to "" to make sure that null value is not overridden with
- // default value ("ts")
- // TODO(HUDI-3456) clean up
- val preCombineField = hoodieCatalogTable.preCombineKey.getOrElse("")
+ val preCombineField = hoodieCatalogTable.preCombineKey.getOrElse(null)
Review Comment:
its a clean up. preCombine wasn't strictly optional before.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -202,7 +206,7 @@ object HoodieSparkSqlWriter {
.setRecordKeyFields(hoodieConfig.getString(RECORDKEY_FIELD))
.setCDCEnabled(hoodieConfig.getBooleanOrDefault(HoodieTableConfig.CDC_ENABLED))
.setCDCSupplementalLoggingMode(hoodieConfig.getStringOrDefault(HoodieTableConfig.CDC_SUPPLEMENTAL_LOGGING_MODE))
- .setKeyGeneratorClassProp(originKeyGeneratorClassName)
+ .setKeyGeneratorClassProp(hoodieConfig.getString(DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME.key))
Review Comment:
I intentionally kept it this way. its confusing whether we are referring to data source key gen or table config key gen config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7901:
URL: https://github.com/apache/hudi/pull/7901#issuecomment-1452733368
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15046",
"triggerID" : "4f81cc10efc5863beb8f9656c05fc2e03ce6c7ee",
"triggerType" : "PUSH"
}, {
"hash" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15347",
"triggerID" : "c55e96ff2e67739855776b50b3a88dd43bfc2f9d",
"triggerType" : "PUSH"
}, {
"hash" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15373",
"triggerID" : "0ee4a7ebbf09c02b2bd81c425f4656e783f815c7",
"triggerType" : "PUSH"
}, {
"hash" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15447",
"triggerID" : "ab532211ea0aa3e15604aa60582801c3111246bf",
"triggerType" : "PUSH"
}, {
"hash" : "da53ca2480f60edf61bced07304373dae4a34332",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15449",
"triggerID" : "da53ca2480f60edf61bced07304373dae4a34332",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15459",
"triggerID" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"triggerType" : "PUSH"
}, {
"hash" : "bb4e42cc4991ae497b3d6df9fd9b77d02a65ef3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15461",
"triggerID" : "1447508000",
"triggerType" : "MANUAL"
}, {
"hash" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15464",
"triggerID" : "fa531e8154c5eec87854ce75c6f3183bbe62a695",
"triggerType" : "PUSH"
}, {
"hash" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15486",
"triggerID" : "31c2cf8d62b8f35653f7fc613e4dbf4f86ac5b50",
"triggerType" : "PUSH"
}, {
"hash" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c",
"triggerType" : "PUSH"
}, {
"hash" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15495",
"triggerID" : "05db189b2e1765cf769c019000c2d3aa8829332a",
"triggerType" : "PUSH"
}, {
"hash" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15493",
"triggerID" : "172d68e4bbca0cbcd87a734676670884011e42ef",
"triggerType" : "PUSH"
}, {
"hash" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15526",
"triggerID" : "3d36b5c0206e3cab5d1551f22ae41082f3cc7354",
"triggerType" : "PUSH"
}, {
"hash" : "b0ddb5d1365ec41ce3890fe33a1aecba4a506b40",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15538",
"triggerID" : "b0ddb5d1365ec41ce3890fe33a1aecba4a506b40",
"triggerType" : "PUSH"
}, {
"hash" : "d86666046498c1811e9e0a446cef557514007f47",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d86666046498c1811e9e0a446cef557514007f47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN
* b0ddb5d1365ec41ce3890fe33a1aecba4a506b40 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15538)
* d86666046498c1811e9e0a446cef557514007f47 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #7901: [HUDI-5665] Adding support to re-use table configs
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #7901:
URL: https://github.com/apache/hudi/pull/7901
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org