You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/07/22 12:19:54 UTC
[GitHub] [hudi] pengzhiwei2018 opened a new pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
pengzhiwei2018 opened a new pull request #3328:
URL: https://github.com/apache/hudi/pull/3328
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-894355629
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* 80aeae998fc99dd3f31662a78c8dc5a9dce57252 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179)
* a427b11ca3c1146c57cdd0ce502ae56da01e66fc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* 123f88802eb116939544de462d3fa372b8eb1684 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406)
* f4e8be72b0140410daeb4eb01879047eba074751 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* e88244d233d323364916c4fc240083566ddc4e56 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272)
* b3e8a6d36161d5da60a1429e518253e1bff92a9d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* f4e8be72b0140410daeb4eb01879047eba074751 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459)
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
* ebd3310059d27544648a31a7c3fb3cb1febcea60 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* 8d35aa66c41c3dc087a8b1514aa12274c4b661ca Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144)
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r682137257
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
+ // insert overwrite table
+ case (_, _, true, _) if !isPartitionedTable => INSERT_OVERWRITE_TABLE_OPERATION_OPT_VAL
+ // if the table has primaryKey and the dropDuplicate has disable, use the upsert operation
+ case (true, false, false, false) => UPSERT_OPERATION_OPT_VAL
+ // if enableBulkInsert is true and the table is non-primaryKeyed, use the bulk insert operation
+ case (false, true, _, _) => BULK_INSERT_OPERATION_OPT_VAL
+ // for the rest case, use the insert operation
+ case (_, _, _, _) => INSERT_OPERATION_OPT_VAL
Review comment:
actually I came across [INSERT OVERWRITE DIRECTORY](https://spark.apache.org/docs/latest/sql-ref-syntax-dml-insert-overwrite-directory.html) which can be mapped to insert_overwrite.
Here is a suggestion w/o using any additional configs:
CTAS -> bulk_insert
Insert into -> insert
INSERT OVERWRITE -> insert overwrite table
INSERT OVERWRITE DIRECTORY -> insert overwrite (partitions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-894613109
Hi @nsivabalan , The PR has updated with the follow chanes:
add "upsert" mode for insert.mode. Currently we have 3 insert mode:
- upsert
In upsert mode for insert into, duplicate record on primary key will be updated.This is the default insert mode for pk-table.
- strict
In strict mode for insert into, we do the pk uniqueness guarantee for COW pk-table. For MOR pk-table, it has the same behavior with "upsert" mode.
- non-strict
In non-strict mode for insert into, we use insert operation to write data which allow writing the duplicate record.
The default insert mode is `upsert` for pk-table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r682101452
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
+ // insert overwrite table
+ case (_, _, true, _) if !isPartitionedTable => INSERT_OVERWRITE_TABLE_OPERATION_OPT_VAL
+ // if the table has primaryKey and the dropDuplicate has disable, use the upsert operation
+ case (true, false, false, false) => UPSERT_OPERATION_OPT_VAL
+ // if enableBulkInsert is true and the table is non-primaryKeyed, use the bulk insert operation
+ case (false, true, _, _) => BULK_INSERT_OPERATION_OPT_VAL
+ // for the rest case, use the insert operation
+ case (_, _, _, _) => INSERT_OPERATION_OPT_VAL
Review comment:
@vinothchandar : do check this out before reviewing other feedbacks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-894571028
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098)
* 1a30511e4a029d862d22864659a5be0fac7a4fd0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101)
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 52f852a534a7d1106ed25dcb48374e5c9947380c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103)
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* 8087f44f7c591e0b50f113fd415a76136d376a8c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"triggerType" : "PUSH"
}, {
"hash" : "cf12550810f530b84896fb904f2feb60eb440ac5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1543",
"triggerID" : "cf12550810f530b84896fb904f2feb60eb440ac5",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "abed45a1e858e7bd16b40e203c9aa88302e67921",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "abed45a1e858e7bd16b40e203c9aa88302e67921",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
* 0fddc583afb6b8eb460eb3dcff57ba355db5b7a8 UNKNOWN
* cf12550810f530b84896fb904f2feb60eb440ac5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1543)
* abed45a1e858e7bd16b40e203c9aa88302e67921 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-894613109
Hi @nsivabalan , The PR has updated with the follow chanes:
add "upsert" mode for insert.mode. Currently we have 3 insert mode:
- upsert
In upsert mode for insert into, duplicate record on primary key will be updated.This is the default insert mode for pk-table.
- strict
In strict mode for insert into, we do the pk uniqueness guarantee for COW pk-table. For MOR pk-table, it has the same behavior with "upsert" mode.
- non-strict
In non-strict mode for insert into, we use insert operation to write data which allow writing the duplicate record.
The default insert mode is `upsert` for pk-table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r684803078
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala
##########
@@ -303,5 +304,184 @@ class TestInsertTable extends TestHoodieSqlBase {
"assertion failed: Required select columns count: 4, Current select columns(including static partition column)" +
" count: 3，columns: (1,a1,10)"
)
+ spark.sql("set hoodie.sql.bulk.insert.enable = true")
+ spark.sql("set hoodie.sql.insert.mode= strict")
+
+ val tableName2 = generateTableName
Review comment:
Can we also enhance the test w/ both type of partitions(single level and multi-level).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r682137257
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
+ // insert overwrite table
+ case (_, _, true, _) if !isPartitionedTable => INSERT_OVERWRITE_TABLE_OPERATION_OPT_VAL
+ // if the table has primaryKey and the dropDuplicate has disable, use the upsert operation
+ case (true, false, false, false) => UPSERT_OPERATION_OPT_VAL
+ // if enableBulkInsert is true and the table is non-primaryKeyed, use the bulk insert operation
+ case (false, true, _, _) => BULK_INSERT_OPERATION_OPT_VAL
+ // for the rest case, use the insert operation
+ case (_, _, _, _) => INSERT_OPERATION_OPT_VAL
Review comment:
actually I came across [INSERT OVERWRITE DIRECTORY](https://spark.apache.org/docs/latest/sql-ref-syntax-dml-insert-overwrite-directory.html) which can be mapped to insert_overwrite.
Here is a suggestion w/o using any additional configs:
CTAS -> bulk_insert
Insert into -> insert
INSERT OVERWRITE -> insert overwrite table
INSERT OVERWRITE DIRECTORY -> insert overwrite
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r679153080
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
Review comment:
also, if there are any valid optimizations, then probably we should move it to HoodiesparkSqlSwriter so that both spark datasource and sql dml benefits :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r680212711
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
+ // insert overwrite table
+ case (_, _, true, _) if !isPartitionedTable => INSERT_OVERWRITE_TABLE_OPERATION_OPT_VAL
+ // if the table has primaryKey and the dropDuplicate has disable, use the upsert operation
+ case (true, false, false, false) => UPSERT_OPERATION_OPT_VAL
+ // if enableBulkInsert is true and the table is non-primaryKeyed, use the bulk insert operation
+ case (false, true, _, _) => BULK_INSERT_OPERATION_OPT_VAL
+ // for the rest case, use the insert operation
+ case (_, _, _, _) => INSERT_OPERATION_OPT_VAL
Review comment:
Here is my thought on choosing the right operation. Having too many case statements might complicate things and is error prone too. As I mentioned earlier, we should try to do any valid conversions in HoodiesSparkSqlWriter. Only those thats applicable just to sql dml, we should keep it here.
Anyways, here is one simplified approach. Ignoring the primary, non primary key table for now. We can come back to that later once we have consensus on this.
We need just two configs.
hoodie.sql.enable.bulk_insert (default false)
hoodie.sql.overwrite.entire.table (default true)
From sql syntax, there are two commands allowed.
"INSERT" into and "INSERT OVERWRITE". And these need to map to 4 operations on the hudi end (insert, bulk_insert, insert over write and insert overwrite table)
"INSERT" with no other configs set -> insert operation
"INSERT" with enable bulk insert set -> bulk_insert
"INSERT OVERWRITE" with no other configs set -> insert_overwrite_table operation
"INSERT OVERWRITE" with hoodie.sql.overwrite.entire.table = false -> insert_overwrite operation.
"INSERT OVERWRITE" with enable bulk_insert set -> bulk_insert. pass the right save mode to HoodieSparkSqlWriter
"INSERT OVERWRITE" with enable bulk_insert set and hoodie.sql.overwrite.entire.table = false -> bulk_insert. pass the right save mode to HoodieSparkSqlWriter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* a427b11ca3c1146c57cdd0ce502ae56da01e66fc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r674866640
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -159,7 +159,10 @@ object HoodieSparkSqlWriter {
// Convert to RDD[HoodieRecord]
val genericRecords: RDD[GenericRecord] = HoodieSparkUtils.createRdd(df, schema, structName, nameSpace)
- val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean || operation.equals(WriteOperationType.UPSERT);
+ val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean ||
+ operation.equals(WriteOperationType.UPSERT) ||
+ parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
Review comment:
here is a bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098)
* 1a30511e4a029d862d22864659a5be0fac7a4fd0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* a427b11ca3c1146c57cdd0ce502ae56da01e66fc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186)
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* 46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* a427b11ca3c1146c57cdd0ce502ae56da01e66fc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186)
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* 46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-886327353
Hi @leesf , I have introduced another config key to enable the bulk insert for sql. Please take a review when you have time~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* 0f565dae9227a4653aed57ee74937d697b6a583a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165)
* 80aeae998fc99dd3f31662a78c8dc5a9dce57252 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r680693753
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -243,6 +256,8 @@ object InsertIntoHoodieTableCommand {
RECORDKEY_FIELD_OPT_KEY.key -> primaryColumns.mkString(","),
PARTITIONPATH_FIELD_OPT_KEY.key -> partitionFields,
PAYLOAD_CLASS_OPT_KEY.key -> payloadClassName,
+ ENABLE_ROW_WRITER_OPT_KEY.key -> enableBulkInsert.toString,
+ HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key -> isPrimaryKeyTable.toString, // if the table has primaryKey, enable the combine
Review comment:
Just like the upsert operation, Hudi do the combine automatic， we can do this for the user too which is much friendly for our users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* d14438fa452f1f9be2235ba6baffd8059b655a15 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131)
* 8d35aa66c41c3dc087a8b1514aa12274c4b661ca UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r677405475
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
Review comment:
may I know why do we have this constraint?
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -159,7 +159,10 @@ object HoodieSparkSqlWriter {
// Convert to RDD[HoodieRecord]
val genericRecords: RDD[GenericRecord] = HoodieSparkUtils.createRdd(df, schema, structName, nameSpace)
- val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean || operation.equals(WriteOperationType.UPSERT);
+ val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean ||
+ operation.equals(WriteOperationType.UPSERT) ||
+ parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
Review comment:
sorry I don't get you. Precombine is just one field as is right. Not sure what do you mean by "not compute the preCombine field value"? can you throw some more light please.
In general, for inserts we don't do any precombine. But if this config (COMBINE_BEFORE_INSERT_PROP) is enabled, we need to do preCombine.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -243,6 +256,8 @@ object InsertIntoHoodieTableCommand {
RECORDKEY_FIELD_OPT_KEY.key -> primaryColumns.mkString(","),
PARTITIONPATH_FIELD_OPT_KEY.key -> partitionFields,
PAYLOAD_CLASS_OPT_KEY.key -> payloadClassName,
+ ENABLE_ROW_WRITER_OPT_KEY.key -> enableBulkInsert.toString,
Review comment:
you can add one, but make the default as true.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
Review comment:
Am just trying to understand the sql dml here. We already handle save modes within HoodieSparkSqlWriter. So, trying to understand whats required in addition to that? Trying to avoid duplication if possible. I mean, for some of the cases listed here, its just about overWrite mode.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -159,7 +159,10 @@ object HoodieSparkSqlWriter {
// Convert to RDD[HoodieRecord]
val genericRecords: RDD[GenericRecord] = HoodieSparkUtils.createRdd(df, schema, structName, nameSpace)
- val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean || operation.equals(WriteOperationType.UPSERT);
+ val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean ||
+ operation.equals(WriteOperationType.UPSERT) ||
+ parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
Review comment:
also, if its something that required just for sql dml, then we should probably think about adding it to sql dml classes and not here which spans spark datasource as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* f4e8be72b0140410daeb4eb01879047eba074751 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r676542368
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -243,6 +256,8 @@ object InsertIntoHoodieTableCommand {
RECORDKEY_FIELD_OPT_KEY.key -> primaryColumns.mkString(","),
PARTITIONPATH_FIELD_OPT_KEY.key -> partitionFields,
PAYLOAD_CLASS_OPT_KEY.key -> payloadClassName,
+ ENABLE_ROW_WRITER_OPT_KEY.key -> enableBulkInsert.toString,
Review comment:
here enableBulkInsert means using ROW_WRITER, should we also introduce another config to control the ROW_WRITER?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #3328:
URL: https://github.com/apache/hudi/pull/3328
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r675960618
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala
##########
@@ -144,7 +145,7 @@ class TestInsertTable extends TestHoodieSqlBase {
| partitioned by (dt)
| location '${tmp.getCanonicalPath}/$tableName'
""".stripMargin)
-
+ spark.sql("set hoodie.datasource.write.row.writer.enable = false")
Review comment:
remove this now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r676621769
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -247,6 +247,13 @@ object DataSourceWriteOptions {
.defaultValue("false")
.withDocumentation("When set to true, will perform write operations directly using the spark native " +
"`Row` representation, avoiding any additional conversion costs.")
+ /**
+ * Enable the bulk insert for sql insert statement.
+ */
+ val SQL_ENABLE_BULK_INSERT:ConfigProperty[String] = ConfigProperty
+ .key("hoodie.sq.bulk.insert.enable")
Review comment:
Good catch!
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -243,6 +256,8 @@ object InsertIntoHoodieTableCommand {
RECORDKEY_FIELD_OPT_KEY.key -> primaryColumns.mkString(","),
PARTITIONPATH_FIELD_OPT_KEY.key -> partitionFields,
PAYLOAD_CLASS_OPT_KEY.key -> payloadClassName,
+ ENABLE_ROW_WRITER_OPT_KEY.key -> enableBulkInsert.toString,
Review comment:
Currently we already have the ENABLE_ROW_WRITER_OPT_KEY to config the Row write. The enableBulkInsert is only used to control the row write for bulk insert.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -243,6 +256,8 @@ object InsertIntoHoodieTableCommand {
RECORDKEY_FIELD_OPT_KEY.key -> primaryColumns.mkString(","),
PARTITIONPATH_FIELD_OPT_KEY.key -> partitionFields,
PAYLOAD_CLASS_OPT_KEY.key -> payloadClassName,
+ ENABLE_ROW_WRITER_OPT_KEY.key -> enableBulkInsert.toString,
+ HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key -> isPrimaryKeyTable.toString, // if the table has primaryKey, enable the combine
Review comment:
Because we must guarantee the uniqueness for p-k table. So we should combine the input for pk-table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r683207185
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -248,6 +248,14 @@ object DataSourceWriteOptions {
.withDocumentation("When set to true, will perform write operations directly using the spark native " +
"`Row` representation, avoiding any additional conversion costs.")
+ /**
+ * Enable the bulk insert for sql insert statement.
+ */
+ val SQL_ENABLE_BULK_INSERT:ConfigProperty[String] = ConfigProperty
Review comment:
Sound reasonable about this. CTAS use the bulk_insert by default, and regular insert for insert into by default.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
Review comment:
For CTAS, we can relax this. Because there is no data exist in the target table. We can just combine the input by pk before bulk insert to reach the same goal.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -159,7 +159,10 @@ object HoodieSparkSqlWriter {
// Convert to RDD[HoodieRecord]
val genericRecords: RDD[GenericRecord] = HoodieSparkUtils.createRdd(df, schema, structName, nameSpace)
- val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean || operation.equals(WriteOperationType.UPSERT);
+ val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean ||
+ operation.equals(WriteOperationType.UPSERT) ||
+ parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
Review comment:
@vinothchandar well I think INSERT_DROP_DUPS_OPT_KEY is some different from COMBINE_BEFORE_INSERT_PROP.
**INSERT_DROP_DUPS_OPT_KEY**: is used to drop the duplicate record in the target table.
`COMBINE_BEFORE_INSERT_PROP`: is used to combine the duplicate record in the input.
So they are not total the same config. IMO.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
Review comment:
Well, in spark-sql , `insert overwrite partitioned table ` do not has the meaning of insert overwrite all the table. It is only overwrite the affected partitions.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
Review comment:
Yes, currently bulk insert has not support INSERT_OVERWRITE operation. So we cannot convert the` insert overwrite partitioned table ` statement to bulk insert.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
+ // insert overwrite table
+ case (_, _, true, _) if !isPartitionedTable => INSERT_OVERWRITE_TABLE_OPERATION_OPT_VAL
Review comment:
Yes, but is cannot convert the write operation type for sql's insert statement. Here we must do the convert and pass the write type to HoodieSparkSqlWriter.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
Review comment:
Yes, the validate is some duplication here. But to keep the logic intact, I think we should keep this.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
+ // insert overwrite table
+ case (_, _, true, _) if !isPartitionedTable => INSERT_OVERWRITE_TABLE_OPERATION_OPT_VAL
+ // if the table has primaryKey and the dropDuplicate has disable, use the upsert operation
+ case (true, false, false, false) => UPSERT_OPERATION_OPT_VAL
Review comment:
As we said above， we need do the pk uniqueness check. So we need use the upsert to reach this goal by a custom Playload class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r683207185
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -248,6 +248,14 @@ object DataSourceWriteOptions {
.withDocumentation("When set to true, will perform write operations directly using the spark native " +
"`Row` representation, avoiding any additional conversion costs.")
+ /**
+ * Enable the bulk insert for sql insert statement.
+ */
+ val SQL_ENABLE_BULK_INSERT:ConfigProperty[String] = ConfigProperty
Review comment:
Sound reasonable about this. CTAS use the bulk_insert by default, and regular insert for insert into by default.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
Review comment:
For CTAS, we can relax this. Because there is no data exist in the target table. We can just combine the input by pk before bulk insert to reach the same goal.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -159,7 +159,10 @@ object HoodieSparkSqlWriter {
// Convert to RDD[HoodieRecord]
val genericRecords: RDD[GenericRecord] = HoodieSparkUtils.createRdd(df, schema, structName, nameSpace)
- val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean || operation.equals(WriteOperationType.UPSERT);
+ val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean ||
+ operation.equals(WriteOperationType.UPSERT) ||
+ parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
Review comment:
@vinothchandar well I think INSERT_DROP_DUPS_OPT_KEY is some different from COMBINE_BEFORE_INSERT_PROP.
**INSERT_DROP_DUPS_OPT_KEY**: is used to drop the duplicate record in the target table.
`COMBINE_BEFORE_INSERT_PROP`: is used to combine the duplicate record in the input.
So they are not total the same config. IMO.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
Review comment:
Well, in spark-sql , `insert overwrite partitioned table ` do not has the meaning of insert overwrite all the table. It is only overwrite the affected partitions.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
Review comment:
Yes, currently bulk insert has not support INSERT_OVERWRITE operation. So we cannot convert the` insert overwrite partitioned table ` statement to bulk insert.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
+ // insert overwrite table
+ case (_, _, true, _) if !isPartitionedTable => INSERT_OVERWRITE_TABLE_OPERATION_OPT_VAL
Review comment:
Yes, but is cannot convert the write operation type for sql's insert statement. Here we must do the convert and pass the write type to HoodieSparkSqlWriter.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
Review comment:
Yes, the validate is some duplication here. But to keep the logic intact, I think we should keep this.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
+ // insert overwrite table
+ case (_, _, true, _) if !isPartitionedTable => INSERT_OVERWRITE_TABLE_OPERATION_OPT_VAL
+ // if the table has primaryKey and the dropDuplicate has disable, use the upsert operation
+ case (true, false, false, false) => UPSERT_OPERATION_OPT_VAL
Review comment:
As we said above， we need do the pk uniqueness check. So we need use the upsert to reach this goal by a custom Playload class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* 123f88802eb116939544de462d3fa372b8eb1684 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-893396472
@vinothchandar @nsivabalan The PR has updated with the follow changes:
1、Allow bulk insert for pk-table.
I introduce a config: `hoodie.sql.insert.mode` . If set to "strict", we will do the pk uniqueness guarantee. If set to "non-strict", we will ignore the uniqueness guarantee for pk table. The bulk insert is support in the case. By default the value is "non-strict".
2、CTAS use bulk insert by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* 80aeae998fc99dd3f31662a78c8dc5a9dce57252 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179)
* a427b11ca3c1146c57cdd0ce502ae56da01e66fc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-894939778
thanks for your contribution! Good job 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r679789141
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
Review comment:
HoodieSparkSqlWriter cannot handle the mode for` insert overwrite partitioned table`, we should translate the write type to INSERT_OVERWRITE_TABLE for such case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r684870787
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala
##########
@@ -303,5 +304,184 @@ class TestInsertTable extends TestHoodieSqlBase {
"assertion failed: Required select columns count: 4, Current select columns(including static partition column)" +
" count: 3，columns: (1,a1,10)"
)
+ spark.sql("set hoodie.sql.bulk.insert.enable = true")
+ spark.sql("set hoodie.sql.insert.mode= strict")
+
+ val tableName2 = generateTableName
Review comment:
Yes, will add the case for multi-level partition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* f4e8be72b0140410daeb4eb01879047eba074751 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447)
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* 8d35aa66c41c3dc087a8b1514aa12274c4b661ca Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144)
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* ceeb9a317a103002e7f6cd198e19310955ab7572 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* e88244d233d323364916c4fc240083566ddc4e56 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-894613109
Hi @nsivabalan , The PR has updated with the follow chanes:
add "upsert" mode for insert.mode. Currently we have 3 insert mode:
- upsert
In upsert mode for insert into, duplicate record on primary key will be updated.This is the default insert mode for pk-table.
- strict
In strict mode for insert into, we do the pk uniqueness guarantee for COW pk-table. For MOR pk-table, it has the same behavior with "upsert" mode.
- non-strict
In non-strict mode for insert into, we use insert operation to write data which allow writing the duplicate record.
The default insert mode is `upsert`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* 8d35aa66c41c3dc087a8b1514aa12274c4b661ca Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144)
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* ceeb9a317a103002e7f6cd198e19310955ab7572 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162)
* 0f565dae9227a4653aed57ee74937d697b6a583a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-894613775
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r683843426
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
+ // insert overwrite table
+ case (_, _, true, _) if !isPartitionedTable => INSERT_OVERWRITE_TABLE_OPERATION_OPT_VAL
+ // if the table has primaryKey and the dropDuplicate has disable, use the upsert operation
+ case (true, false, false, false) => UPSERT_OPERATION_OPT_VAL
+ // if enableBulkInsert is true and the table is non-primaryKeyed, use the bulk insert operation
+ case (false, true, _, _) => BULK_INSERT_OPERATION_OPT_VAL
+ // for the rest case, use the insert operation
+ case (_, _, _, _) => INSERT_OPERATION_OPT_VAL
Review comment:
I did go through every case here and have 2 suggestions. rest of the cases looks good. You don't need to consider my proposal above. But would like you to consider below feedback.
1.
```
case (true, true, _, _) if !isNonStrictMode => throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert in strict mode.")
```
Can we enable preCombine here and proceed with Bulk_Insert operation. Within hudi, we can do preCombine/dedup. As we agreed on using bulk_insert as default with CTAS, this will be a very common use-case.
2.
```
case (_, true, true, _) if isPartitionedTable =>
throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
```
since we agreed on enabling Bulk_insert as default for CTAS, this will be very common use-case as well. Can you help me understand why do we fail this call? why can't we let it proceed. This is basically, CTAS for a partitioned table.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -159,7 +159,10 @@ object HoodieSparkSqlWriter {
// Convert to RDD[HoodieRecord]
val genericRecords: RDD[GenericRecord] = HoodieSparkUtils.createRdd(df, schema, structName, nameSpace)
- val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean || operation.equals(WriteOperationType.UPSERT);
+ val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean ||
+ operation.equals(WriteOperationType.UPSERT) ||
+ parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
Review comment:
my bad. I get it now.
If InsertDropDups is set, we automatically set combine.before.insert. but if a user has set just "combine.before.insert", we need to do PreCombine here.
But I am not sure why this wasn't reported by anyone until now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"triggerType" : "PUSH"
}, {
"hash" : "cf12550810f530b84896fb904f2feb60eb440ac5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1543",
"triggerID" : "cf12550810f530b84896fb904f2feb60eb440ac5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
* 0fddc583afb6b8eb460eb3dcff57ba355db5b7a8 UNKNOWN
* Unknown: [CANCELED](TBD)
* cf12550810f530b84896fb904f2feb60eb440ac5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1543)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r682108786
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
Review comment:
insert overwrite partition should be using the `INSERT_OVERWRITE` operation.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -159,7 +159,10 @@ object HoodieSparkSqlWriter {
// Convert to RDD[HoodieRecord]
val genericRecords: RDD[GenericRecord] = HoodieSparkUtils.createRdd(df, schema, structName, nameSpace)
- val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean || operation.equals(WriteOperationType.UPSERT);
+ val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean ||
+ operation.equals(WriteOperationType.UPSERT) ||
+ parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
Review comment:
@pengzhiwei2018 do you agree with siva's analysis above?
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
Review comment:
+1 users might have another hudi table for e.g to CTAS from. So if we disallow bulk insert with a pk, then there is no good way to do a full bootstrap. Can we relax this?
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
+ // insert overwrite table
+ case (_, _, true, _) if !isPartitionedTable => INSERT_OVERWRITE_TABLE_OPERATION_OPT_VAL
+ // if the table has primaryKey and the dropDuplicate has disable, use the upsert operation
+ case (true, false, false, false) => UPSERT_OPERATION_OPT_VAL
+ // if enableBulkInsert is true and the table is non-primaryKeyed, use the bulk insert operation
+ case (false, true, _, _) => BULK_INSERT_OPERATION_OPT_VAL
+ // for the rest case, use the insert operation
+ case (_, _, _, _) => INSERT_OPERATION_OPT_VAL
Review comment:
having configs to control sql behavior sounds pretty complex to me. Can we just pick the right operation for each type.
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -248,6 +248,14 @@ object DataSourceWriteOptions {
.withDocumentation("When set to true, will perform write operations directly using the spark native " +
"`Row` representation, avoiding any additional conversion costs.")
+ /**
+ * Enable the bulk insert for sql insert statement.
+ */
+ val SQL_ENABLE_BULK_INSERT:ConfigProperty[String] = ConfigProperty
Review comment:
CTAS should use bulk_insert and insert into should use regular inserts IMO.
lets please not add more configs like this proactively. We can get user feedback and iterate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
* Unknown: [CANCELED](TBD)
* 0fddc583afb6b8eb460eb3dcff57ba355db5b7a8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
* ebd3310059d27544648a31a7c3fb3cb1febcea60 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a30511e4a029d862d22864659a5be0fac7a4fd0 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101)
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 52f852a534a7d1106ed25dcb48374e5c9947380c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* f4e8be72b0140410daeb4eb01879047eba074751 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* 46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194)
* e88244d233d323364916c4fc240083566ddc4e56 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"triggerType" : "PUSH"
}, {
"hash" : "cf12550810f530b84896fb904f2feb60eb440ac5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1543",
"triggerID" : "cf12550810f530b84896fb904f2feb60eb440ac5",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
* 0fddc583afb6b8eb460eb3dcff57ba355db5b7a8 UNKNOWN
* cf12550810f530b84896fb904f2feb60eb440ac5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1543)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* 8d35aa66c41c3dc087a8b1514aa12274c4b661ca Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098)
* 1a30511e4a029d862d22864659a5be0fac7a4fd0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* f4e8be72b0140410daeb4eb01879047eba074751 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459)
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
* ebd3310059d27544648a31a7c3fb3cb1febcea60 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r677534597
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -243,6 +256,8 @@ object InsertIntoHoodieTableCommand {
RECORDKEY_FIELD_OPT_KEY.key -> primaryColumns.mkString(","),
PARTITIONPATH_FIELD_OPT_KEY.key -> partitionFields,
PAYLOAD_CLASS_OPT_KEY.key -> payloadClassName,
+ ENABLE_ROW_WRITER_OPT_KEY.key -> enableBulkInsert.toString,
+ HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key -> isPrimaryKeyTable.toString, // if the table has primaryKey, enable the combine
Review comment:
even for pk-table, in insert mode, there is no need to combine the input, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* 0f565dae9227a4653aed57ee74937d697b6a583a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165)
* 80aeae998fc99dd3f31662a78c8dc5a9dce57252 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 52f852a534a7d1106ed25dcb48374e5c9947380c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103)
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* 8087f44f7c591e0b50f113fd415a76136d376a8c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"triggerType" : "PUSH"
}, {
"hash" : "cf12550810f530b84896fb904f2feb60eb440ac5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1543",
"triggerID" : "cf12550810f530b84896fb904f2feb60eb440ac5",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "abed45a1e858e7bd16b40e203c9aa88302e67921",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1558",
"triggerID" : "abed45a1e858e7bd16b40e203c9aa88302e67921",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
* 0fddc583afb6b8eb460eb3dcff57ba355db5b7a8 UNKNOWN
* cf12550810f530b84896fb904f2feb60eb440ac5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1543)
* abed45a1e858e7bd16b40e203c9aa88302e67921 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1558)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* d14438fa452f1f9be2235ba6baffd8059b655a15 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131)
* 8d35aa66c41c3dc087a8b1514aa12274c4b661ca Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-894651032
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* 8087f44f7c591e0b50f113fd415a76136d376a8c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* 0f565dae9227a4653aed57ee74937d697b6a583a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* ceeb9a317a103002e7f6cd198e19310955ab7572 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162)
* 0f565dae9227a4653aed57ee74937d697b6a583a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* ceeb9a317a103002e7f6cd198e19310955ab7572 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162)
* 0f565dae9227a4653aed57ee74937d697b6a583a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r684894126
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala
##########
@@ -378,6 +379,41 @@ class TestInsertTable extends TestHoodieSqlBase {
Seq(1, "a1", 10.0, "2021-07-18"),
Seq(2, "a2", 10.0, "2021-07-18")
)
+
+ // Test bulk insert for multi-level partition
+ val tableMultiPartition = generateTableName
+ spark.sql(
+ s"""
+ |create table $tableMultiPartition (
+ | id int,
+ | name string,
+ | price double,
+ | dt string,
+ | hh string
+ |) using hudi
+ | options (
+ | type = '$tableType'
+ | )
+ | partitioned by (dt, hh)
+ | location '${tmp.getCanonicalPath}/$tableMultiPartition'
+ """.stripMargin)
+
+ // Enable the bulk insert
+ spark.sql("set hoodie.sql.bulk.insert.enable = true")
+ spark.sql(s"insert into $tableMultiPartition values(1, 'a1', 10, '2021-07-18', '12')")
+
+ checkAnswer(s"select id, name, price, dt, hh from $tableMultiPartition")(
Review comment:
lets verify meta fields as well as suggested in other patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"triggerType" : "PUSH"
}, {
"hash" : "cf12550810f530b84896fb904f2feb60eb440ac5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1543",
"triggerID" : "cf12550810f530b84896fb904f2feb60eb440ac5",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "abed45a1e858e7bd16b40e203c9aa88302e67921",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1558",
"triggerID" : "abed45a1e858e7bd16b40e203c9aa88302e67921",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
* 0fddc583afb6b8eb460eb3dcff57ba355db5b7a8 UNKNOWN
* abed45a1e858e7bd16b40e203c9aa88302e67921 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1558)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r674869873
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/HoodieOptionConfig.scala
##########
@@ -172,6 +178,15 @@ object HoodieOptionConfig {
params.get(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY.key)
}
+ /**
+ * Whether enable the bulk insert for sql insert statement when there is no primaryKey in the table.
+ */
+ def enableBulkInsert(options: Map[String, String]): Boolean = {
Review comment:
ENABLE_ROW_WRITER_OPT_KEY is a different type of bulk_insert, spark sql support both types?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r678031341
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -159,7 +159,10 @@ object HoodieSparkSqlWriter {
// Convert to RDD[HoodieRecord]
val genericRecords: RDD[GenericRecord] = HoodieSparkUtils.createRdd(df, schema, structName, nameSpace)
- val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean || operation.equals(WriteOperationType.UPSERT);
+ val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean ||
+ operation.equals(WriteOperationType.UPSERT) ||
+ parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
Review comment:
If the `COMBINE_BEFORE_INSERT_PROP` has enabled, `SparkInsertCommitActionExecutor` will do the `combineOnCondition` in `AbstractWriteHelper` which will need the `precombine` value .
But here we have ignored the case of `COMBINE_BEFORE_INSERT_PROP` which will not extract the `precombine` value to the HoodiePayload.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
Review comment:
Because currently we will do the primary key uniqueness check when inserting data to the pk-table, just like what database does. Bulk insert currently cannot do such things.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -243,6 +256,8 @@ object InsertIntoHoodieTableCommand {
RECORDKEY_FIELD_OPT_KEY.key -> primaryColumns.mkString(","),
PARTITIONPATH_FIELD_OPT_KEY.key -> partitionFields,
PAYLOAD_CLASS_OPT_KEY.key -> payloadClassName,
+ ENABLE_ROW_WRITER_OPT_KEY.key -> enableBulkInsert.toString,
+ HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key -> isPrimaryKeyTable.toString, // if the table has primaryKey, enable the combine
Review comment:
We need to ensure that data is unique for pk-table just like database, so I do combine for the input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
* 0fddc583afb6b8eb460eb3dcff57ba355db5b7a8 UNKNOWN
* Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r680197209
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
Review comment:
anyways, we can call it out that its responsibility of the user to ensure there are uniqueness. Also, IIUC, hudi can handle duplicates. Incase of updates, both records will be updated. but bulk_insert is very performant compared to regular Insert especially w/ row wirter. So, we should not keep it too restrictive for use. I know from the community msgs, that lot of users leverage bulk_insert. I would vote to relax this constraint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* f4e8be72b0140410daeb4eb01879047eba074751 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* 8087f44f7c591e0b50f113fd415a76136d376a8c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123)
* d14438fa452f1f9be2235ba6baffd8059b655a15 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* d14438fa452f1f9be2235ba6baffd8059b655a15 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a30511e4a029d862d22864659a5be0fac7a4fd0 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101)
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 52f852a534a7d1106ed25dcb48374e5c9947380c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* 46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* a427b11ca3c1146c57cdd0ce502ae56da01e66fc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186)
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* f4e8be72b0140410daeb4eb01879047eba074751 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459)
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
* ebd3310059d27544648a31a7c3fb3cb1febcea60 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* f4e8be72b0140410daeb4eb01879047eba074751 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r674875373
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/HoodieOptionConfig.scala
##########
@@ -172,6 +178,15 @@ object HoodieOptionConfig {
params.get(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY.key)
}
+ /**
+ * Whether enable the bulk insert for sql insert statement when there is no primaryKey in the table.
+ */
+ def enableBulkInsert(options: Map[String, String]): Boolean = {
Review comment:
I saw that currently ENABLE_ROW_WRITER_OPT_KEY is only used for bulk insert, so i reused this config.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -159,7 +159,10 @@ object HoodieSparkSqlWriter {
// Convert to RDD[HoodieRecord]
val genericRecords: RDD[GenericRecord] = HoodieSparkUtils.createRdd(df, schema, structName, nameSpace)
- val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean || operation.equals(WriteOperationType.UPSERT);
+ val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean ||
+ operation.equals(WriteOperationType.UPSERT) ||
+ parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
Review comment:
Yes, if we have enable the COMBINE_BEFORE_INSERT_PROP for insert, it has not compute the pre combine field value which will result incorrect result for insert with duplicate records.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* 80aeae998fc99dd3f31662a78c8dc5a9dce57252 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* f4e8be72b0140410daeb4eb01879047eba074751 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 52f852a534a7d1106ed25dcb48374e5c9947380c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103)
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r677534597
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -243,6 +256,8 @@ object InsertIntoHoodieTableCommand {
RECORDKEY_FIELD_OPT_KEY.key -> primaryColumns.mkString(","),
PARTITIONPATH_FIELD_OPT_KEY.key -> partitionFields,
PAYLOAD_CLASS_OPT_KEY.key -> payloadClassName,
+ ENABLE_ROW_WRITER_OPT_KEY.key -> enableBulkInsert.toString,
+ HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key -> isPrimaryKeyTable.toString, // if the table has primaryKey, enable the combine
Review comment:
for pk-table and in insert mode, there is no need to combine the input, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* 123f88802eb116939544de462d3fa372b8eb1684 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406)
* f4e8be72b0140410daeb4eb01879047eba074751 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* f4e8be72b0140410daeb4eb01879047eba074751 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447)
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
* ebd3310059d27544648a31a7c3fb3cb1febcea60 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-894613109
Hi @nsivabalan , The PR has updated with the follow chanes:
add "upsert" mode for insert.mode. Currently we have 3 insert mode:
- upsert
In upsert mode for insert into, duplicate record on primary key will be updated.This is the default insert mode for pk-table.
- strict
In strict mode for insert into, we do the pk uniqueness guarantee for COW pk-table. For MOR pk-table, it has the same behavior with "upsert" mode.
- non-strict
In non-strict mode for insert into, we use insert operation to write data which allow writing the duplicate record.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-894753661
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r674872620
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala
##########
@@ -144,7 +145,7 @@ class TestInsertTable extends TestHoodieSqlBase {
| partitioned by (dt)
| location '${tmp.getCanonicalPath}/$tableName'
""".stripMargin)
-
+ spark.sql("set hoodie.datasource.write.row.writer.enable = false")
Review comment:
all set to false?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r680186620
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -248,6 +248,14 @@ object DataSourceWriteOptions {
.withDocumentation("When set to true, will perform write operations directly using the spark native " +
"`Row` representation, avoiding any additional conversion costs.")
+ /**
+ * Enable the bulk insert for sql insert statement.
+ */
+ val SQL_ENABLE_BULK_INSERT:ConfigProperty[String] = ConfigProperty
Review comment:
@vinothchandar : In sql, we don't have two separate commands like INSERT into and BULK_INSERT into. so, guess we are going this route. But default CTAS choose INSERT operation. I am thinking users may not use bulk_insert only since they have to set the property explicitly. any thoughts.
There are two things to discuss.
1. Which operation to use with CTAS
2. which operation to use with INSERT into.
State as of now, is "Insert". And user has to explicitly set operation type to bulk_insert before calling any of this commands.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
+ // insert overwrite table
+ case (_, _, true, _) if !isPartitionedTable => INSERT_OVERWRITE_TABLE_OPERATION_OPT_VAL
Review comment:
HoodieSparkSqlWriter will handle this save mode.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
+ // insert overwrite table
+ case (_, _, true, _) if !isPartitionedTable => INSERT_OVERWRITE_TABLE_OPERATION_OPT_VAL
+ // if the table has primaryKey and the dropDuplicate has disable, use the upsert operation
+ case (true, false, false, false) => UPSERT_OPERATION_OPT_VAL
+ // if enableBulkInsert is true and the table is non-primaryKeyed, use the bulk insert operation
+ case (false, true, _, _) => BULK_INSERT_OPERATION_OPT_VAL
+ // for the rest case, use the insert operation
+ case (_, _, _, _) => INSERT_OPERATION_OPT_VAL
Review comment:
Here is my thought on choosing the right operation. Having too many case statements might complicate things and is error prone too. As I mentioned earlier, we should try to do any valid conversions in HoodiesSparkSqlWriter. Only those thats applicable just to sql dml, we should keep it here.
Anyways, here is one simplified approach. Ignoring the primary, non primary key table for now. We can come back to that later once we have consensus on this.
We need just two configs.
hoodie.sql.enable.bulk_insert (default false)
hoodie.sql.overwrite.entire.table (default true)
From sql syntax, there are two commands allowed.
"INSERT" into and "INSERT OVERWRITE".
"INSERT" with no other configs set -> insert operation
"INSERT" with enable bulk insert set -> bulk_insert
"INSERT OVERWRITE" with no other configs set -> insert_overwrite_table operation
"INSERT OVERWRITE" with hoodie.sql.overwrite.entire.table = false -> insert_overwrite operation.
"INSERT OVERWRITE" with enable bulk_insert set -> bulk_insert. pass the right save mode to HoodieSparkSqlWriter
"INSERT OVERWRITE" with enable bulk_insert set and hoodie.sql.overwrite.entire.table = false -> bulk_insert. pass the right save mode to HoodieSparkSqlWriter.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
Review comment:
lets say a user is doing something like this.
- create hudi table w/ primary key and partition col set.
- does bulk_insert with overwrite.
Do we fail this command? This is very common use-case. Not sure how we can fail this.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
+ // insert overwrite table
+ case (_, _, true, _) if !isPartitionedTable => INSERT_OVERWRITE_TABLE_OPERATION_OPT_VAL
+ // if the table has primaryKey and the dropDuplicate has disable, use the upsert operation
+ case (true, false, false, false) => UPSERT_OPERATION_OPT_VAL
Review comment:
again trying to understand why we do this. If someone is explicitly issuing "INSERT" into, we should try to use "insert" operation. Why switching to "upsert" just for primary keyed table ? can you please clarify
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
Review comment:
This is already taken care in HoodieSparkSqlWriter. Don't think we need to validate this here.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -243,6 +256,8 @@ object InsertIntoHoodieTableCommand {
RECORDKEY_FIELD_OPT_KEY.key -> primaryColumns.mkString(","),
PARTITIONPATH_FIELD_OPT_KEY.key -> partitionFields,
PAYLOAD_CLASS_OPT_KEY.key -> payloadClassName,
+ ENABLE_ROW_WRITER_OPT_KEY.key -> enableBulkInsert.toString,
+ HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key -> isPrimaryKeyTable.toString, // if the table has primaryKey, enable the combine
Review comment:
yeah, I am also not sure on this. We will do a preCombine step for any INSERT in general. wondering why not let users take up the responsibility.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
+ // insert overwrite partition
+ case (_, _, true, _) if isPartitionedTable => INSERT_OVERWRITE_OPERATION_OPT_VAL
Review comment:
not sure if we can do this.
If user has set "overwrite" for a partitioned table, here we are overwriting only the interested partition and not the entire table. Is that a common sql expectation? What incase a user has a partitioned table and wants to overwrite an entire table?
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
Review comment:
anyways, we can all it out that its responsibility of the user to ensure there are uniqueness. Also, IIUC, hudi can handle duplicates. Incase of updates, both records will be updated. but bulk_insert is very performant compared to regular Insert especially w/ row wirter. So, we should not keep it too restrictive for use. I know from the community msgs, that lot of users leverage bulk_insert. I would vote to relax this constraint.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -159,7 +159,10 @@ object HoodieSparkSqlWriter {
// Convert to RDD[HoodieRecord]
val genericRecords: RDD[GenericRecord] = HoodieSparkUtils.createRdd(df, schema, structName, nameSpace)
- val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean || operation.equals(WriteOperationType.UPSERT);
+ val shouldCombine = parameters(INSERT_DROP_DUPS_OPT_KEY.key()).toBoolean ||
+ operation.equals(WriteOperationType.UPSERT) ||
+ parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
Review comment:
I digged in to understand this. Actually INSERT_DROP_DUPS_OPT_KEY is used at the Dataource layer and COMBINE_BEFORE_INSERT_PROP is used in writeClient layer. In other words, both are referring to same config only. [DataSourceUtils](https://github.com/apache/hudi/blob/6353fc865f43854e0e185af33e8ad091c8870d78/hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java#L178) converts INSERT_DROP_DUPS_OPT_KEY's value as COMBINE_BEFORE_INSERT_PROP.
So, don't think we need to check both INSERT_DROP_DUPS_OPT_KEY and COMBINE_BEFORE_INSERT_PROP.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
Review comment:
hmmm, interesting. I didn't know we do uniqueness check for inserts with primary keyed table.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -209,19 +209,32 @@ object InsertIntoHoodieTableCommand {
.getOrElse(INSERT_DROP_DUPS_OPT_KEY.defaultValue)
.toBoolean
- val operation = if (isOverwrite) {
- if (table.partitionColumnNames.nonEmpty) {
- INSERT_OVERWRITE_OPERATION_OPT_VAL // overwrite partition
- } else {
- INSERT_OPERATION_OPT_VAL
+ val enableBulkInsert = parameters.getOrElse(DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.key,
+ DataSourceWriteOptions.SQL_ENABLE_BULK_INSERT.defaultValue()).toBoolean
+ val isPartitionedTable = table.partitionColumnNames.nonEmpty
+ val isPrimaryKeyTable = primaryColumns.nonEmpty
+ val operation =
+ (isPrimaryKeyTable, enableBulkInsert, isOverwrite, dropDuplicate) match {
+ case (true, true, _, _) =>
+ throw new IllegalArgumentException(s"Table with primaryKey can not use bulk insert.")
+ case (_, true, true, _) if isPartitionedTable =>
+ throw new IllegalArgumentException(s"Insert Overwrite Partition can not use bulk insert.")
+ case (_, true, _, true) =>
+ throw new IllegalArgumentException(s"Bulk insert cannot support drop duplication." +
+ s" Please disable $INSERT_DROP_DUPS_OPT_KEY and try again.")
+ // if enableBulkInsert is true, use bulk insert for the insert overwrite non-partitioned table.
+ case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
Review comment:
```
case (_, true, true, _) if !isPartitionedTable => BULK_INSERT_OPERATION_OPT_VAL
```
Already the operation is bulk_insert right(2nd arg). Not sure why do we need this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r676541180
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -247,6 +247,13 @@ object DataSourceWriteOptions {
.defaultValue("false")
.withDocumentation("When set to true, will perform write operations directly using the spark native " +
"`Row` representation, avoiding any additional conversion costs.")
+ /**
+ * Enable the bulk insert for sql insert statement.
+ */
+ val SQL_ENABLE_BULK_INSERT:ConfigProperty[String] = ConfigProperty
+ .key("hoodie.sq.bulk.insert.enable")
Review comment:
nit: sql
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r676542684
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -243,6 +256,8 @@ object InsertIntoHoodieTableCommand {
RECORDKEY_FIELD_OPT_KEY.key -> primaryColumns.mkString(","),
PARTITIONPATH_FIELD_OPT_KEY.key -> partitionFields,
PAYLOAD_CLASS_OPT_KEY.key -> payloadClassName,
+ ENABLE_ROW_WRITER_OPT_KEY.key -> enableBulkInsert.toString,
+ HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key -> isPrimaryKeyTable.toString, // if the table has primaryKey, enable the combine
Review comment:
why the table has primaryKey, should enable the combine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"triggerType" : "PUSH"
}, {
"hash" : "cf12550810f530b84896fb904f2feb60eb440ac5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cf12550810f530b84896fb904f2feb60eb440ac5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
* 0fddc583afb6b8eb460eb3dcff57ba355db5b7a8 UNKNOWN
* Unknown: [CANCELED](TBD)
* cf12550810f530b84896fb904f2feb60eb440ac5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 52f852a534a7d1106ed25dcb48374e5c9947380c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* 8087f44f7c591e0b50f113fd415a76136d376a8c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123)
* d14438fa452f1f9be2235ba6baffd8059b655a15 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* b3e8a6d36161d5da60a1429e518253e1bff92a9d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402)
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* 123f88802eb116939544de462d3fa372b8eb1684 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#discussion_r677536750
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala
##########
@@ -243,6 +256,8 @@ object InsertIntoHoodieTableCommand {
RECORDKEY_FIELD_OPT_KEY.key -> primaryColumns.mkString(","),
PARTITIONPATH_FIELD_OPT_KEY.key -> partitionFields,
PAYLOAD_CLASS_OPT_KEY.key -> payloadClassName,
+ ENABLE_ROW_WRITER_OPT_KEY.key -> enableBulkInsert.toString,
Review comment:
so here why there is no option for bulk_insert without row_writer? must in row_writer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* 46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194)
* e88244d233d323364916c4fc240083566ddc4e56 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-884869427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1098",
"triggerID" : "834080ee7aa7c5ad2fba139d7dd8f66ddaf75b0d",
"triggerType" : "PUSH"
}, {
"hash" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1101",
"triggerID" : "1a30511e4a029d862d22864659a5be0fac7a4fd0",
"triggerType" : "PUSH"
}, {
"hash" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9f804618dd0275abdae10673c21bf1f5737caf",
"triggerType" : "PUSH"
}, {
"hash" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1103",
"triggerID" : "52f852a534a7d1106ed25dcb48374e5c9947380c",
"triggerType" : "PUSH"
}, {
"hash" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50539ec543951e7a4442798ac7c66e5dc3d3705a",
"triggerType" : "PUSH"
}, {
"hash" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1123",
"triggerID" : "8087f44f7c591e0b50f113fd415a76136d376a8c",
"triggerType" : "PUSH"
}, {
"hash" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1131",
"triggerID" : "d14438fa452f1f9be2235ba6baffd8059b655a15",
"triggerType" : "PUSH"
}, {
"hash" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1144",
"triggerID" : "8d35aa66c41c3dc087a8b1514aa12274c4b661ca",
"triggerType" : "PUSH"
}, {
"hash" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8b449c31ee8601542f00e3cc15fbcab77da7787",
"triggerType" : "PUSH"
}, {
"hash" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1162",
"triggerID" : "ceeb9a317a103002e7f6cd198e19310955ab7572",
"triggerType" : "PUSH"
}, {
"hash" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1165",
"triggerID" : "0f565dae9227a4653aed57ee74937d697b6a583a",
"triggerType" : "PUSH"
}, {
"hash" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1179",
"triggerID" : "80aeae998fc99dd3f31662a78c8dc5a9dce57252",
"triggerType" : "PUSH"
}, {
"hash" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1186",
"triggerID" : "a427b11ca3c1146c57cdd0ce502ae56da01e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb9a6d83361f3a652b2c902b1b3dc846de617d93",
"triggerType" : "PUSH"
}, {
"hash" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1194",
"triggerID" : "46f6c46dd03ca0d342ad0b8e774f2c8b77429c0d",
"triggerType" : "PUSH"
}, {
"hash" : "e88244d233d323364916c4fc240083566ddc4e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272",
"triggerID" : "e88244d233d323364916c4fc240083566ddc4e56",
"triggerType" : "PUSH"
}, {
"hash" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1402",
"triggerID" : "b3e8a6d36161d5da60a1429e518253e1bff92a9d",
"triggerType" : "PUSH"
}, {
"hash" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb7a1eae66a8d9ea6a730e5206da9b4434a50e",
"triggerType" : "PUSH"
}, {
"hash" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d1608a58c08664f19ca8439162fece22d11e3f",
"triggerType" : "PUSH"
}, {
"hash" : "123f88802eb116939544de462d3fa372b8eb1684",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1406",
"triggerID" : "123f88802eb116939544de462d3fa372b8eb1684",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1426",
"triggerID" : "f4e8be72b0140410daeb4eb01879047eba074751",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443",
"triggerID" : "894355629",
"triggerType" : "MANUAL"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1447",
"triggerID" : "894571028",
"triggerType" : "MANUAL"
}, {
"hash" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b986aa4cc25d36da24a8ea926e3ecbe8912f1f17",
"triggerType" : "PUSH"
}, {
"hash" : "f4e8be72b0140410daeb4eb01879047eba074751",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1459",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"triggerType" : "PUSH"
}, {
"hash" : "ebd3310059d27544648a31a7c3fb3cb1febcea60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1460",
"triggerID" : "894613775",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
}, {
"hash" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fddc583afb6b8eb460eb3dcff57ba355db5b7a8",
"triggerType" : "PUSH"
}, {
"hash" : "cf12550810f530b84896fb904f2feb60eb440ac5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1543",
"triggerID" : "cf12550810f530b84896fb904f2feb60eb440ac5",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "894651032",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c9f804618dd0275abdae10673c21bf1f5737caf UNKNOWN
* 50539ec543951e7a4442798ac7c66e5dc3d3705a UNKNOWN
* f8b449c31ee8601542f00e3cc15fbcab77da7787 UNKNOWN
* bb9a6d83361f3a652b2c902b1b3dc846de617d93 UNKNOWN
* becb7a1eae66a8d9ea6a730e5206da9b4434a50e UNKNOWN
* 37d1608a58c08664f19ca8439162fece22d11e3f UNKNOWN
* b986aa4cc25d36da24a8ea926e3ecbe8912f1f17 UNKNOWN
* 0fddc583afb6b8eb460eb3dcff57ba355db5b7a8 UNKNOWN
* cf12550810f530b84896fb904f2feb60eb440ac5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1543)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 edited a comment on pull request #3328: [HUDI-2208] Support Bulk Insert For Spark Sql
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 edited a comment on pull request #3328:
URL: https://github.com/apache/hudi/pull/3328#issuecomment-894613109
Hi @nsivabalan , The PR has updated with the follow chanes:
add "upsert" mode for insert.mode. Currently we have 3 insert mode:
- upsert
In upsert mode for insert into, duplicate record on primary key will be updated.This is the default insert mode for pk-table.
- strict
In strict mode for insert into, we do the pk uniqueness guarantee for COW pk-table. For MOR pk-table, it has the same behavior with "upsert" mode.
- non-strict
In non-strict mode for insert into, we use insert operation to write data which allow writing the duplicate record.
The default insert mode is `upsert` for pk-table. And these config is only used to control the behavior of pk-table. For non pk-table, the insert operation is always `insert` or `bulkinsert`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org