You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/16 08:21:32 UTC
[GitHub] [hudi] yihua opened a new pull request, #6694: [DO NOT MERGE][HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
yihua opened a new pull request, #6694:
URL: https://github.com/apache/hudi/pull/6694
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
**Risk level: none | low | medium | high**
_Choose one. If medium or high, explain what verification was done to mitigate the risks._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1279115963
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5a6eed936fc08b943370db12c258ea6e75430912",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422",
"triggerID" : "5a6eed936fc08b943370db12c258ea6e75430912",
"triggerType" : "PUSH"
}, {
"hash" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11838",
"triggerID" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"triggerType" : "PUSH"
}, {
"hash" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11855",
"triggerID" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"triggerType" : "PUSH"
}, {
"hash" : "048886383d8a4b017cfa8db6a882a677342014c2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "048886383d8a4b017cfa8db6a882a677342014c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9d1849308eda33f9b8e4667aae0cdbad60185bca Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11855)
* 048886383d8a4b017cfa8db6a882a677342014c2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6694: [DO NOT MERGE][HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1249094331
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5a6eed936fc08b943370db12c258ea6e75430912",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422",
"triggerID" : "5a6eed936fc08b943370db12c258ea6e75430912",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5a6eed936fc08b943370db12c258ea6e75430912 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6694: [DO NOT MERGE][HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1249088011
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5a6eed936fc08b943370db12c258ea6e75430912",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5a6eed936fc08b943370db12c258ea6e75430912",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5a6eed936fc08b943370db12c258ea6e75430912 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6694: [DO NOT MERGE][HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1249869916
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5a6eed936fc08b943370db12c258ea6e75430912",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422",
"triggerID" : "5a6eed936fc08b943370db12c258ea6e75430912",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5a6eed936fc08b943370db12c258ea6e75430912 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
codope merged PR #6694:
URL: https://github.com/apache/hudi/pull/6694
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1261286012
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5a6eed936fc08b943370db12c258ea6e75430912",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422",
"triggerID" : "5a6eed936fc08b943370db12c258ea6e75430912",
"triggerType" : "PUSH"
}, {
"hash" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11838",
"triggerID" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"triggerType" : "PUSH"
}, {
"hash" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b054f56340368ab5db1ff8d7b30415cbf273e729 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11838)
* 9d1849308eda33f9b8e4667aae0cdbad60185bca UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6694:
URL: https://github.com/apache/hudi/pull/6694#discussion_r993376698
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java:
##########
@@ -639,11 +639,14 @@ public void testBulkInsertsAndUpsertsWithBootstrap() throws Exception {
Dataset<Row> sourceDf = sqlContext.read()
.format("org.apache.hudi")
.load(tableBasePath);
- sourceDf.write().format("parquet").save(bootstrapSourcePath);
+ // TODO(HUDI-4944): fix the test to use a partition column with slashes (`/`) included
+ // in the value. Currently it fails the tests due to slash encoding.
+ sourceDf.write().format("parquet").partitionBy("rider").save(bootstrapSourcePath);
Review Comment:
should we fix it in this PR? trying to understand what's the effort here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1280434088
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5a6eed936fc08b943370db12c258ea6e75430912",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422",
"triggerID" : "5a6eed936fc08b943370db12c258ea6e75430912",
"triggerType" : "PUSH"
}, {
"hash" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11838",
"triggerID" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"triggerType" : "PUSH"
}, {
"hash" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11855",
"triggerID" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"triggerType" : "PUSH"
}, {
"hash" : "048886383d8a4b017cfa8db6a882a677342014c2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12213",
"triggerID" : "048886383d8a4b017cfa8db6a882a677342014c2",
"triggerType" : "PUSH"
}, {
"hash" : "df1330e38535c748c322be6642b17cc43c081f04",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12246",
"triggerID" : "df1330e38535c748c322be6642b17cc43c081f04",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 048886383d8a4b017cfa8db6a882a677342014c2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12213)
* df1330e38535c748c322be6642b17cc43c081f04 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12246)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
yihua commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1261233705
@codope I fixed the tests. Let's see if the CI can pass now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1261279707
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5a6eed936fc08b943370db12c258ea6e75430912",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422",
"triggerID" : "5a6eed936fc08b943370db12c258ea6e75430912",
"triggerType" : "PUSH"
}, {
"hash" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11838",
"triggerID" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b054f56340368ab5db1ff8d7b30415cbf273e729 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11838)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6694: [DO NOT MERGE][HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1260713996
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5a6eed936fc08b943370db12c258ea6e75430912",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422",
"triggerID" : "5a6eed936fc08b943370db12c258ea6e75430912",
"triggerType" : "PUSH"
}, {
"hash" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5a6eed936fc08b943370db12c258ea6e75430912 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422)
* b054f56340368ab5db1ff8d7b30415cbf273e729 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6694:
URL: https://github.com/apache/hudi/pull/6694#discussion_r982685165
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/BootstrapExecutor.java:
##########
@@ -191,11 +194,32 @@ private void initializeTable() throws IOException {
.fromProperties(props)
.setTableType(cfg.tableType)
.setTableName(cfg.targetTableName)
+ .setRecordKeyFields(props.getString(KeyGeneratorOptions.RECORDKEY_FIELD_NAME.key()))
+ .setPartitionFields(props.getString(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(), null))
+ .setPreCombineField(props.getString(HoodieWriteConfig.PRECOMBINE_FIELD_NAME.key()))
Review Comment:
For the precombine config, the logic now falls back to the default value if not provided by the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6694:
URL: https://github.com/apache/hudi/pull/6694#discussion_r993376698
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java:
##########
@@ -639,11 +639,14 @@ public void testBulkInsertsAndUpsertsWithBootstrap() throws Exception {
Dataset<Row> sourceDf = sqlContext.read()
.format("org.apache.hudi")
.load(tableBasePath);
- sourceDf.write().format("parquet").save(bootstrapSourcePath);
+ // TODO(HUDI-4944): fix the test to use a partition column with slashes (`/`) included
+ // in the value. Currently it fails the tests due to slash encoding.
+ sourceDf.write().format("parquet").partitionBy("rider").save(bootstrapSourcePath);
Review Comment:
should we fix it in this PR? trying to understand what's effort here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1279179321
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5a6eed936fc08b943370db12c258ea6e75430912",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422",
"triggerID" : "5a6eed936fc08b943370db12c258ea6e75430912",
"triggerType" : "PUSH"
}, {
"hash" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11838",
"triggerID" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"triggerType" : "PUSH"
}, {
"hash" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11855",
"triggerID" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"triggerType" : "PUSH"
}, {
"hash" : "048886383d8a4b017cfa8db6a882a677342014c2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12213",
"triggerID" : "048886383d8a4b017cfa8db6a882a677342014c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 048886383d8a4b017cfa8db6a882a677342014c2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12213)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1280428200
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5a6eed936fc08b943370db12c258ea6e75430912",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422",
"triggerID" : "5a6eed936fc08b943370db12c258ea6e75430912",
"triggerType" : "PUSH"
}, {
"hash" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11838",
"triggerID" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"triggerType" : "PUSH"
}, {
"hash" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11855",
"triggerID" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"triggerType" : "PUSH"
}, {
"hash" : "048886383d8a4b017cfa8db6a882a677342014c2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12213",
"triggerID" : "048886383d8a4b017cfa8db6a882a677342014c2",
"triggerType" : "PUSH"
}, {
"hash" : "df1330e38535c748c322be6642b17cc43c081f04",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "df1330e38535c748c322be6642b17cc43c081f04",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 048886383d8a4b017cfa8db6a882a677342014c2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12213)
* df1330e38535c748c322be6642b17cc43c081f04 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1280630024
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5a6eed936fc08b943370db12c258ea6e75430912",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422",
"triggerID" : "5a6eed936fc08b943370db12c258ea6e75430912",
"triggerType" : "PUSH"
}, {
"hash" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11838",
"triggerID" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"triggerType" : "PUSH"
}, {
"hash" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11855",
"triggerID" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"triggerType" : "PUSH"
}, {
"hash" : "048886383d8a4b017cfa8db6a882a677342014c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12213",
"triggerID" : "048886383d8a4b017cfa8db6a882a677342014c2",
"triggerType" : "PUSH"
}, {
"hash" : "df1330e38535c748c322be6642b17cc43c081f04",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12246",
"triggerID" : "df1330e38535c748c322be6642b17cc43c081f04",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df1330e38535c748c322be6642b17cc43c081f04 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12246)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #6694: [DO NOT MERGE][HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
codope commented on code in PR #6694:
URL: https://github.com/apache/hudi/pull/6694#discussion_r974236009
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/BootstrapExecutor.java:
##########
@@ -191,11 +194,32 @@ private void initializeTable() throws IOException {
.fromProperties(props)
.setTableType(cfg.tableType)
.setTableName(cfg.targetTableName)
+ .setRecordKeyFields(props.getString(KeyGeneratorOptions.RECORDKEY_FIELD_NAME.key()))
+ .setPartitionFields(props.getString(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(), null))
+ .setPreCombineField(props.getString(HoodieWriteConfig.PRECOMBINE_FIELD_NAME.key()))
Review Comment:
Probably, we need to set this in the test as well. I see some CI test failure due to precombine config not set in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6694:
URL: https://github.com/apache/hudi/pull/6694#discussion_r993976464
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java:
##########
@@ -639,11 +639,14 @@ public void testBulkInsertsAndUpsertsWithBootstrap() throws Exception {
Dataset<Row> sourceDf = sqlContext.read()
.format("org.apache.hudi")
.load(tableBasePath);
- sourceDf.write().format("parquet").save(bootstrapSourcePath);
+ // TODO(HUDI-4944): fix the test to use a partition column with slashes (`/`) included
+ // in the value. Currently it fails the tests due to slash encoding.
+ sourceDf.write().format("parquet").partitionBy("rider").save(bootstrapSourcePath);
Review Comment:
This is to prepare the parquet dataset for bootstrapping. For non-partitioned table, it's failing, along with IT failures, due to schema mismatch. I need to dig deeper into how this can be fixed properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1261495654
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5a6eed936fc08b943370db12c258ea6e75430912",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422",
"triggerID" : "5a6eed936fc08b943370db12c258ea6e75430912",
"triggerType" : "PUSH"
}, {
"hash" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11838",
"triggerID" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"triggerType" : "PUSH"
}, {
"hash" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11855",
"triggerID" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9d1849308eda33f9b8e4667aae0cdbad60185bca Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11855)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6694: [HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1261291616
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5a6eed936fc08b943370db12c258ea6e75430912",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422",
"triggerID" : "5a6eed936fc08b943370db12c258ea6e75430912",
"triggerType" : "PUSH"
}, {
"hash" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11838",
"triggerID" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"triggerType" : "PUSH"
}, {
"hash" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11855",
"triggerID" : "9d1849308eda33f9b8e4667aae0cdbad60185bca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b054f56340368ab5db1ff8d7b30415cbf273e729 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11838)
* 9d1849308eda33f9b8e4667aae0cdbad60185bca Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11855)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6694: [DO NOT MERGE][HUDI-4855] Add missing table configs for bootstrap in Deltastreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6694:
URL: https://github.com/apache/hudi/pull/6694#issuecomment-1260719819
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5a6eed936fc08b943370db12c258ea6e75430912",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422",
"triggerID" : "5a6eed936fc08b943370db12c258ea6e75430912",
"triggerType" : "PUSH"
}, {
"hash" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11838",
"triggerID" : "b054f56340368ab5db1ff8d7b30415cbf273e729",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5a6eed936fc08b943370db12c258ea6e75430912 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11422)
* b054f56340368ab5db1ff8d7b30415cbf273e729 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11838)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org