You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/18 07:19:46 UTC
[GitHub] [hudi] weimingdiit opened a new pull request, #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
weimingdiit opened a new pull request, #6983:
URL: https://github.com/apache/hudi/pull/6983
…rce table has partitions and the target table does not
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1367183737
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
}, {
"hash" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906",
"triggerID" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"triggerType" : "PUSH"
}, {
"hash" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14035",
"triggerID" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d1694b05083fee989a712a00f01c771bb0f7f95f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906)
* d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14035)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1324979375
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 862be37daa8c19a5bcfd8f602ea2fedc94b556cd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147)
* 48ef299dc5efceb79aec32ff69266887885e7172 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1294408694
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 20c8a6f31ed6f507475129170fc0e40ae6c6a89f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289)
* f5a8b04cb184f9c9f00961884c479856594f57f2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1767760342
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
}, {
"hash" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906",
"triggerID" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"triggerType" : "PUSH"
}, {
"hash" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14035",
"triggerID" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"triggerType" : "PUSH"
}, {
"hash" : "0593cd212628684db658d7a8bdd8fc320069d090",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20225",
"triggerID" : "0593cd212628684db658d7a8bdd8fc320069d090",
"triggerType" : "PUSH"
}, {
"hash" : "003721a9e975415951aed2725a744b29f87cacc1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20377",
"triggerID" : "003721a9e975415951aed2725a744b29f87cacc1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 003721a9e975415951aed2725a744b29f87cacc1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20377)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] weimingdiit closed pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
weimingdiit closed pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
URL: https://github.com/apache/hudi/pull/6983
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1327381608
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 139a4b2aa768aa8195c2ed88028c902ea62fc77f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1294503898
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5a8b04cb184f9c9f00961884c479856594f57f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] weimingdiit commented on a diff in pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
weimingdiit commented on code in PR #6983:
URL: https://github.com/apache/hudi/pull/6983#discussion_r1006794252
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/CopyOnWriteInsertHandler.java:
##########
@@ -69,9 +73,16 @@ public CopyOnWriteInsertHandler(HoodieWriteConfig config, String instantTime,
@Override
public void consumeOneRecord(HoodieInsertValueGenResult<HoodieRecord> payload) {
- final HoodieRecord insertPayload = payload.record;
+ final HoodieRecord<T> insertPayload = payload.record;
String partitionPath = insertPayload.getPartitionPath();
HoodieWriteHandle<?,?,?,?> handle = handles.get(partitionPath);
+
+ Option<IndexedRecord> insertRecord = payload.insertValue;
+ // just skip the ignored record，do not make partitions on fs
+ if (insertRecord.isPresent() && insertRecord.get().equals(IGNORE_RECORD)) {
Review Comment:
@KnightChess The partition path is created in the writeHandleFactory.create method. If the handle was created, an empty file has been created
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] weimingdiit commented on a diff in pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
weimingdiit commented on code in PR #6983:
URL: https://github.com/apache/hudi/pull/6983#discussion_r1007675272
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/CopyOnWriteInsertHandler.java:
##########
@@ -69,28 +73,29 @@ public CopyOnWriteInsertHandler(HoodieWriteConfig config, String instantTime,
@Override
public void consumeOneRecord(HoodieInsertValueGenResult<HoodieRecord> payload) {
- final HoodieRecord insertPayload = payload.record;
+ final HoodieRecord<T> insertPayload = payload.record;
String partitionPath = insertPayload.getPartitionPath();
HoodieWriteHandle<?,?,?,?> handle = handles.get(partitionPath);
- if (handle == null) {
- // If the records are sorted, this means that we encounter a new partition path
- // and the records for the previous partition path are all written,
- // so we can safely closely existing open handle to reduce memory footprint.
- if (areRecordsSorted) {
- closeOpenHandles();
+ if (handle == null || !handle.canWrite(payload.record)) {
+ if (handle == null) {
+ // If the records are sorted, this means that we encounter a new partition path
+ // and the records for the previous partition path are all written,
+ // so we can safely closely existing open handle to reduce memory footprint.
+ if (areRecordsSorted) {
+ closeOpenHandles();
+ }
+ } else {
+ // Handle is full. Close the handle and add the WriteStatus
+ statuses.addAll(handle.close());
}
- // Lazily initialize the handle, for the first time
- handle = writeHandleFactory.create(config, instantTime, hoodieTable,
- insertPayload.getPartitionPath(), idPrefix, taskContextSupplier);
- handles.put(partitionPath, handle);
- }
-
- if (!handle.canWrite(payload.record)) {
- // Handle is full. Close the handle and add the WriteStatus
- statuses.addAll(handle.close());
- // Open new handle
+ Option<IndexedRecord> insertRecord = payload.insertValue;
+ // just skip the ignored record，do not make partitions on fs
+ if (insertRecord.isPresent() && insertRecord.get().equals(IGNORE_RECORD)) {
+ return;
+ }
Review Comment:
after judge handle, so you don't have to judge every Record, Just judge when the if condition is met，Most records do not need to be judged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1363566468
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
}, {
"hash" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906",
"triggerID" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d1694b05083fee989a712a00f01c771bb0f7f95f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1324969755
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 862be37daa8c19a5bcfd8f602ea2fedc94b556cd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147)
* 48ef299dc5efceb79aec32ff69266887885e7172 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] weimingdiit commented on a diff in pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
weimingdiit commented on code in PR #6983:
URL: https://github.com/apache/hudi/pull/6983#discussion_r1023417893
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/CopyOnWriteInsertHandler.java:
##########
@@ -69,28 +73,29 @@ public CopyOnWriteInsertHandler(HoodieWriteConfig config, String instantTime,
@Override
public void consumeOneRecord(HoodieInsertValueGenResult<HoodieRecord> payload) {
- final HoodieRecord insertPayload = payload.record;
+ final HoodieRecord<T> insertPayload = payload.record;
String partitionPath = insertPayload.getPartitionPath();
HoodieWriteHandle<?,?,?,?> handle = handles.get(partitionPath);
- if (handle == null) {
- // If the records are sorted, this means that we encounter a new partition path
- // and the records for the previous partition path are all written,
- // so we can safely closely existing open handle to reduce memory footprint.
- if (areRecordsSorted) {
- closeOpenHandles();
+ if (handle == null || !handle.canWrite(payload.record)) {
+ if (handle == null) {
+ // If the records are sorted, this means that we encounter a new partition path
+ // and the records for the previous partition path are all written,
+ // so we can safely closely existing open handle to reduce memory footprint.
+ if (areRecordsSorted) {
+ closeOpenHandles();
+ }
+ } else {
+ // Handle is full. Close the handle and add the WriteStatus
+ statuses.addAll(handle.close());
}
- // Lazily initialize the handle, for the first time
- handle = writeHandleFactory.create(config, instantTime, hoodieTable,
- insertPayload.getPartitionPath(), idPrefix, taskContextSupplier);
- handles.put(partitionPath, handle);
- }
-
- if (!handle.canWrite(payload.record)) {
- // Handle is full. Close the handle and add the WriteStatus
- statuses.addAll(handle.close());
- // Open new handle
+ Option<IndexedRecord> insertRecord = payload.insertValue;
+ // just skip the ignored record，do not make partitions on fs
+ if (insertRecord.isPresent() && insertRecord.get().equals(IGNORE_RECORD)) {
+ return;
+ }
Review Comment:
@xushiyan how about that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1750173382
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
}, {
"hash" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906",
"triggerID" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"triggerType" : "PUSH"
}, {
"hash" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14035",
"triggerID" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"triggerType" : "PUSH"
}, {
"hash" : "0593cd212628684db658d7a8bdd8fc320069d090",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20225",
"triggerID" : "0593cd212628684db658d7a8bdd8fc320069d090",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0593cd212628684db658d7a8bdd8fc320069d090 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20225)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1769501500
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
}, {
"hash" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906",
"triggerID" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"triggerType" : "PUSH"
}, {
"hash" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14035",
"triggerID" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"triggerType" : "PUSH"
}, {
"hash" : "0593cd212628684db658d7a8bdd8fc320069d090",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20225",
"triggerID" : "0593cd212628684db658d7a8bdd8fc320069d090",
"triggerType" : "PUSH"
}, {
"hash" : "003721a9e975415951aed2725a744b29f87cacc1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20377",
"triggerID" : "003721a9e975415951aed2725a744b29f87cacc1",
"triggerType" : "PUSH"
}, {
"hash" : "0066bfa306a0ca43aa7d3fa0070cd5c4a8aeb295",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20388",
"triggerID" : "0066bfa306a0ca43aa7d3fa0070cd5c4a8aeb295",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 003721a9e975415951aed2725a744b29f87cacc1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20377)
* 0066bfa306a0ca43aa7d3fa0070cd5c4a8aeb295 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20388)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1321690667
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d6e723cc2365690f99ee59a008988394e03f232 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774)
* 862be37daa8c19a5bcfd8f602ea2fedc94b556cd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1303053023
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d6e723cc2365690f99ee59a008988394e03f232 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1302963014
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5a8b04cb184f9c9f00961884c479856594f57f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632)
* 5d6e723cc2365690f99ee59a008988394e03f232 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1367320447
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
}, {
"hash" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906",
"triggerID" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"triggerType" : "PUSH"
}, {
"hash" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14035",
"triggerID" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14035)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1750003159
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
}, {
"hash" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906",
"triggerID" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"triggerType" : "PUSH"
}, {
"hash" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14035",
"triggerID" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"triggerType" : "PUSH"
}, {
"hash" : "0593cd212628684db658d7a8bdd8fc320069d090",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0593cd212628684db658d7a8bdd8fc320069d090",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14035)
* 0593cd212628684db658d7a8bdd8fc320069d090 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1767631150
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
}, {
"hash" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906",
"triggerID" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"triggerType" : "PUSH"
}, {
"hash" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14035",
"triggerID" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"triggerType" : "PUSH"
}, {
"hash" : "0593cd212628684db658d7a8bdd8fc320069d090",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20225",
"triggerID" : "0593cd212628684db658d7a8bdd8fc320069d090",
"triggerType" : "PUSH"
}, {
"hash" : "003721a9e975415951aed2725a744b29f87cacc1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20377",
"triggerID" : "003721a9e975415951aed2725a744b29f87cacc1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0593cd212628684db658d7a8bdd8fc320069d090 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20225)
* 003721a9e975415951aed2725a744b29f87cacc1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20377)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1282270124
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 20c8a6f31ed6f507475129170fc0e40ae6c6a89f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6983:
URL: https://github.com/apache/hudi/pull/6983#discussion_r1007622645
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/CopyOnWriteInsertHandler.java:
##########
@@ -69,28 +73,29 @@ public CopyOnWriteInsertHandler(HoodieWriteConfig config, String instantTime,
@Override
public void consumeOneRecord(HoodieInsertValueGenResult<HoodieRecord> payload) {
- final HoodieRecord insertPayload = payload.record;
+ final HoodieRecord<T> insertPayload = payload.record;
String partitionPath = insertPayload.getPartitionPath();
HoodieWriteHandle<?,?,?,?> handle = handles.get(partitionPath);
- if (handle == null) {
- // If the records are sorted, this means that we encounter a new partition path
- // and the records for the previous partition path are all written,
- // so we can safely closely existing open handle to reduce memory footprint.
- if (areRecordsSorted) {
- closeOpenHandles();
+ if (handle == null || !handle.canWrite(payload.record)) {
+ if (handle == null) {
+ // If the records are sorted, this means that we encounter a new partition path
+ // and the records for the previous partition path are all written,
+ // so we can safely closely existing open handle to reduce memory footprint.
+ if (areRecordsSorted) {
+ closeOpenHandles();
+ }
+ } else {
+ // Handle is full. Close the handle and add the WriteStatus
+ statuses.addAll(handle.close());
}
- // Lazily initialize the handle, for the first time
- handle = writeHandleFactory.create(config, instantTime, hoodieTable,
- insertPayload.getPartitionPath(), idPrefix, taskContextSupplier);
- handles.put(partitionPath, handle);
- }
-
- if (!handle.canWrite(payload.record)) {
- // Handle is full. Close the handle and add the WriteStatus
- statuses.addAll(handle.close());
- // Open new handle
+ Option<IndexedRecord> insertRecord = payload.insertValue;
+ // just skip the ignored record，do not make partitions on fs
+ if (insertRecord.isPresent() && insertRecord.get().equals(IGNORE_RECORD)) {
+ return;
+ }
+ // Lazily initialize the handle for the first time or Open new handle
handle = writeHandleFactory.create(config, instantTime, hoodieTable,
- insertPayload.getPartitionPath(), idPrefix, taskContextSupplier);
+ insertPayload.getPartitionPath(), idPrefix, taskContextSupplier);
Review Comment:
unnecessary change
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/CopyOnWriteInsertHandler.java:
##########
@@ -69,28 +73,29 @@ public CopyOnWriteInsertHandler(HoodieWriteConfig config, String instantTime,
@Override
public void consumeOneRecord(HoodieInsertValueGenResult<HoodieRecord> payload) {
- final HoodieRecord insertPayload = payload.record;
+ final HoodieRecord<T> insertPayload = payload.record;
String partitionPath = insertPayload.getPartitionPath();
HoodieWriteHandle<?,?,?,?> handle = handles.get(partitionPath);
- if (handle == null) {
- // If the records are sorted, this means that we encounter a new partition path
- // and the records for the previous partition path are all written,
- // so we can safely closely existing open handle to reduce memory footprint.
- if (areRecordsSorted) {
- closeOpenHandles();
+ if (handle == null || !handle.canWrite(payload.record)) {
+ if (handle == null) {
Review Comment:
why change this logic flow? pls revert unnecessary changes
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/CopyOnWriteInsertHandler.java:
##########
@@ -69,28 +73,29 @@ public CopyOnWriteInsertHandler(HoodieWriteConfig config, String instantTime,
@Override
public void consumeOneRecord(HoodieInsertValueGenResult<HoodieRecord> payload) {
- final HoodieRecord insertPayload = payload.record;
+ final HoodieRecord<T> insertPayload = payload.record;
String partitionPath = insertPayload.getPartitionPath();
HoodieWriteHandle<?,?,?,?> handle = handles.get(partitionPath);
- if (handle == null) {
- // If the records are sorted, this means that we encounter a new partition path
- // and the records for the previous partition path are all written,
- // so we can safely closely existing open handle to reduce memory footprint.
- if (areRecordsSorted) {
- closeOpenHandles();
+ if (handle == null || !handle.canWrite(payload.record)) {
+ if (handle == null) {
+ // If the records are sorted, this means that we encounter a new partition path
+ // and the records for the previous partition path are all written,
+ // so we can safely closely existing open handle to reduce memory footprint.
+ if (areRecordsSorted) {
+ closeOpenHandles();
+ }
+ } else {
+ // Handle is full. Close the handle and add the WriteStatus
+ statuses.addAll(handle.close());
}
- // Lazily initialize the handle, for the first time
- handle = writeHandleFactory.create(config, instantTime, hoodieTable,
- insertPayload.getPartitionPath(), idPrefix, taskContextSupplier);
- handles.put(partitionPath, handle);
- }
-
- if (!handle.canWrite(payload.record)) {
- // Handle is full. Close the handle and add the WriteStatus
- statuses.addAll(handle.close());
- // Open new handle
+ Option<IndexedRecord> insertRecord = payload.insertValue;
+ // just skip the ignored record，do not make partitions on fs
+ if (insertRecord.isPresent() && insertRecord.get().equals(IGNORE_RECORD)) {
+ return;
+ }
Review Comment:
if you want to return based on these conditions, why not return early?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] weimingdiit commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
weimingdiit commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1291322183
> @weimingdiit i think only `merge into` can cause this issue, right?
yes, it is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
YannByron commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1290716531
@weimingdiit i think only `merge into` can cause this issue, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1294404873
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 20c8a6f31ed6f507475129170fc0e40ae6c6a89f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289)
* f5a8b04cb184f9c9f00961884c479856594f57f2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1321890893
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 862be37daa8c19a5bcfd8f602ea2fedc94b556cd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1767623459
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
}, {
"hash" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906",
"triggerID" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"triggerType" : "PUSH"
}, {
"hash" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14035",
"triggerID" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"triggerType" : "PUSH"
}, {
"hash" : "0593cd212628684db658d7a8bdd8fc320069d090",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20225",
"triggerID" : "0593cd212628684db658d7a8bdd8fc320069d090",
"triggerType" : "PUSH"
}, {
"hash" : "003721a9e975415951aed2725a744b29f87cacc1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003721a9e975415951aed2725a744b29f87cacc1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0593cd212628684db658d7a8bdd8fc320069d090 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20225)
* 003721a9e975415951aed2725a744b29f87cacc1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] weimingdiit commented on a diff in pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
weimingdiit commented on code in PR #6983:
URL: https://github.com/apache/hudi/pull/6983#discussion_r1007675272
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/CopyOnWriteInsertHandler.java:
##########
@@ -69,28 +73,29 @@ public CopyOnWriteInsertHandler(HoodieWriteConfig config, String instantTime,
@Override
public void consumeOneRecord(HoodieInsertValueGenResult<HoodieRecord> payload) {
- final HoodieRecord insertPayload = payload.record;
+ final HoodieRecord<T> insertPayload = payload.record;
String partitionPath = insertPayload.getPartitionPath();
HoodieWriteHandle<?,?,?,?> handle = handles.get(partitionPath);
- if (handle == null) {
- // If the records are sorted, this means that we encounter a new partition path
- // and the records for the previous partition path are all written,
- // so we can safely closely existing open handle to reduce memory footprint.
- if (areRecordsSorted) {
- closeOpenHandles();
+ if (handle == null || !handle.canWrite(payload.record)) {
+ if (handle == null) {
+ // If the records are sorted, this means that we encounter a new partition path
+ // and the records for the previous partition path are all written,
+ // so we can safely closely existing open handle to reduce memory footprint.
+ if (areRecordsSorted) {
+ closeOpenHandles();
+ }
+ } else {
+ // Handle is full. Close the handle and add the WriteStatus
+ statuses.addAll(handle.close());
}
- // Lazily initialize the handle, for the first time
- handle = writeHandleFactory.create(config, instantTime, hoodieTable,
- insertPayload.getPartitionPath(), idPrefix, taskContextSupplier);
- handles.put(partitionPath, handle);
- }
-
- if (!handle.canWrite(payload.record)) {
- // Handle is full. Close the handle and add the WriteStatus
- statuses.addAll(handle.close());
- // Open new handle
+ Option<IndexedRecord> insertRecord = payload.insertValue;
+ // just skip the ignored record，do not make partitions on fs
+ if (insertRecord.isPresent() && insertRecord.get().equals(IGNORE_RECORD)) {
+ return;
+ }
Review Comment:
after judge handle, so you don't have to judge every Record, Just judge when the if condition is met
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/CopyOnWriteInsertHandler.java:
##########
@@ -69,28 +73,29 @@ public CopyOnWriteInsertHandler(HoodieWriteConfig config, String instantTime,
@Override
public void consumeOneRecord(HoodieInsertValueGenResult<HoodieRecord> payload) {
- final HoodieRecord insertPayload = payload.record;
+ final HoodieRecord<T> insertPayload = payload.record;
String partitionPath = insertPayload.getPartitionPath();
HoodieWriteHandle<?,?,?,?> handle = handles.get(partitionPath);
- if (handle == null) {
- // If the records are sorted, this means that we encounter a new partition path
- // and the records for the previous partition path are all written,
- // so we can safely closely existing open handle to reduce memory footprint.
- if (areRecordsSorted) {
- closeOpenHandles();
+ if (handle == null || !handle.canWrite(payload.record)) {
+ if (handle == null) {
Review Comment:
if no change this logic，Some code is duplicate:
in my first version :
first judge ignored record, other code no change，but every record need to judge,it is unnecessary
second version:
i move ignored record logic into if condition, but it need twice，if(handle == null) {...} and if(!handle.canWrite(payload.record)) {...}, and writeHandleFactory.create() also call twice, it is duplicate.
so i change this logic flow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] weimingdiit commented on a diff in pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
weimingdiit commented on code in PR #6983:
URL: https://github.com/apache/hudi/pull/6983#discussion_r1008954486
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/CopyOnWriteInsertHandler.java:
##########
@@ -69,28 +73,29 @@ public CopyOnWriteInsertHandler(HoodieWriteConfig config, String instantTime,
@Override
public void consumeOneRecord(HoodieInsertValueGenResult<HoodieRecord> payload) {
- final HoodieRecord insertPayload = payload.record;
+ final HoodieRecord<T> insertPayload = payload.record;
String partitionPath = insertPayload.getPartitionPath();
HoodieWriteHandle<?,?,?,?> handle = handles.get(partitionPath);
- if (handle == null) {
- // If the records are sorted, this means that we encounter a new partition path
- // and the records for the previous partition path are all written,
- // so we can safely closely existing open handle to reduce memory footprint.
- if (areRecordsSorted) {
- closeOpenHandles();
+ if (handle == null || !handle.canWrite(payload.record)) {
+ if (handle == null) {
Review Comment:
@xushiyan how about this? If worried about causing other hidden problems, I can change the code , revert logic flow and return early
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1281992874
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 20c8a6f31ed6f507475129170fc0e40ae6c6a89f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1321697736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d6e723cc2365690f99ee59a008988394e03f232 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774)
* 862be37daa8c19a5bcfd8f602ea2fedc94b556cd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1750009750
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
}, {
"hash" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906",
"triggerID" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"triggerType" : "PUSH"
}, {
"hash" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14035",
"triggerID" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"triggerType" : "PUSH"
}, {
"hash" : "0593cd212628684db658d7a8bdd8fc320069d090",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20225",
"triggerID" : "0593cd212628684db658d7a8bdd8fc320069d090",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14035)
* 0593cd212628684db658d7a8bdd8fc320069d090 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20225)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1769371501
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
}, {
"hash" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906",
"triggerID" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"triggerType" : "PUSH"
}, {
"hash" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14035",
"triggerID" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"triggerType" : "PUSH"
}, {
"hash" : "0593cd212628684db658d7a8bdd8fc320069d090",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20225",
"triggerID" : "0593cd212628684db658d7a8bdd8fc320069d090",
"triggerType" : "PUSH"
}, {
"hash" : "003721a9e975415951aed2725a744b29f87cacc1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20377",
"triggerID" : "003721a9e975415951aed2725a744b29f87cacc1",
"triggerType" : "PUSH"
}, {
"hash" : "0066bfa306a0ca43aa7d3fa0070cd5c4a8aeb295",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0066bfa306a0ca43aa7d3fa0070cd5c4a8aeb295",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 003721a9e975415951aed2725a744b29f87cacc1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20377)
* 0066bfa306a0ca43aa7d3fa0070cd5c4a8aeb295 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1281985425
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 20c8a6f31ed6f507475129170fc0e40ae6c6a89f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1327200902
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48ef299dc5efceb79aec32ff69266887885e7172 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217)
* 139a4b2aa768aa8195c2ed88028c902ea62fc77f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1327193162
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48ef299dc5efceb79aec32ff69266887885e7172 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217)
* 139a4b2aa768aa8195c2ed88028c902ea62fc77f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] KnightChess commented on a diff in pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
KnightChess commented on code in PR #6983:
URL: https://github.com/apache/hudi/pull/6983#discussion_r1004352078
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/CopyOnWriteInsertHandler.java:
##########
@@ -69,9 +73,16 @@ public CopyOnWriteInsertHandler(HoodieWriteConfig config, String instantTime,
@Override
public void consumeOneRecord(HoodieInsertValueGenResult<HoodieRecord> payload) {
- final HoodieRecord insertPayload = payload.record;
+ final HoodieRecord<T> insertPayload = payload.record;
String partitionPath = insertPayload.getPartitionPath();
HoodieWriteHandle<?,?,?,?> handle = handles.get(partitionPath);
+
+ Option<IndexedRecord> insertRecord = payload.insertValue;
+ // just skip the ignored record，do not make partitions on fs
+ if (insertRecord.isPresent() && insertRecord.get().equals(IGNORE_RECORD)) {
Review Comment:
may be only need to jude when the wirte handle be created
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1302960236
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5a8b04cb184f9c9f00961884c479856594f57f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632)
* 5d6e723cc2365690f99ee59a008988394e03f232 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar merged PR #6983:
URL: https://github.com/apache/hudi/pull/6983
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1749996160
@weimingdiit : FYI: Have rebased and made slight changes. Will help take this to completion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] weimingdiit commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
weimingdiit commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1321695044
@xushiyan @yihua @XuQianJin-Stars hi, please take a review for this pr. thks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1325433598
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48ef299dc5efceb79aec32ff69266887885e7172 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] weimingdiit commented on a diff in pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
weimingdiit commented on code in PR #6983:
URL: https://github.com/apache/hudi/pull/6983#discussion_r1007556383
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/CopyOnWriteInsertHandler.java:
##########
@@ -69,9 +73,16 @@ public CopyOnWriteInsertHandler(HoodieWriteConfig config, String instantTime,
@Override
public void consumeOneRecord(HoodieInsertValueGenResult<HoodieRecord> payload) {
- final HoodieRecord insertPayload = payload.record;
+ final HoodieRecord<T> insertPayload = payload.record;
String partitionPath = insertPayload.getPartitionPath();
HoodieWriteHandle<?,?,?,?> handle = handles.get(partitionPath);
+
+ Option<IndexedRecord> insertRecord = payload.insertValue;
+ // just skip the ignored record，do not make partitions on fs
+ if (insertRecord.isPresent() && insertRecord.get().equals(IGNORE_RECORD)) {
Review Comment:
@KnightChess ok，get
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6983:
URL: https://github.com/apache/hudi/pull/6983#discussion_r1004604859
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/CopyOnWriteInsertHandler.java:
##########
@@ -69,9 +73,16 @@ public CopyOnWriteInsertHandler(HoodieWriteConfig config, String instantTime,
@Override
public void consumeOneRecord(HoodieInsertValueGenResult<HoodieRecord> payload) {
- final HoodieRecord insertPayload = payload.record;
+ final HoodieRecord<T> insertPayload = payload.record;
String partitionPath = insertPayload.getPartitionPath();
HoodieWriteHandle<?,?,?,?> handle = handles.get(partitionPath);
+
+ Option<IndexedRecord> insertRecord = payload.insertValue;
+ // just skip the ignored record，do not make partitions on fs
+ if (insertRecord.isPresent() && insertRecord.get().equals(IGNORE_RECORD)) {
Review Comment:
another suggestion: For all `HoodieWriteHandle`'s sub-classes, improve their `close` method to return an empty List<WriteStatus> if no records are written.
@xushiyan WDYH?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] KnightChess commented on a diff in pull request #6983: [HUDI-5031]Hudi merge into creates empty partition files when the sou…
Posted by GitBox <gi...@apache.org>.
KnightChess commented on code in PR #6983:
URL: https://github.com/apache/hudi/pull/6983#discussion_r1006826893
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/CopyOnWriteInsertHandler.java:
##########
@@ -69,9 +73,16 @@ public CopyOnWriteInsertHandler(HoodieWriteConfig config, String instantTime,
@Override
public void consumeOneRecord(HoodieInsertValueGenResult<HoodieRecord> payload) {
- final HoodieRecord insertPayload = payload.record;
+ final HoodieRecord<T> insertPayload = payload.record;
String partitionPath = insertPayload.getPartitionPath();
HoodieWriteHandle<?,?,?,?> handle = handles.get(partitionPath);
+
+ Option<IndexedRecord> insertRecord = payload.insertValue;
+ // just skip the ignored record，do not make partitions on fs
+ if (insertRecord.isPresent() && insertRecord.get().equals(IGNORE_RECORD)) {
Review Comment:
@weimingdiit after handle judge, before `weriteHandleFactory.create`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1361087418
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
}, {
"hash" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 139a4b2aa768aa8195c2ed88028c902ea62fc77f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240)
* d1694b05083fee989a712a00f01c771bb0f7f95f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6983: [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6983:
URL: https://github.com/apache/hudi/pull/6983#issuecomment-1367179839
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12289",
"triggerID" : "20c8a6f31ed6f507475129170fc0e40ae6c6a89f",
"triggerType" : "PUSH"
}, {
"hash" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12632",
"triggerID" : "f5a8b04cb184f9c9f00961884c479856594f57f2",
"triggerType" : "PUSH"
}, {
"hash" : "5d6e723cc2365690f99ee59a008988394e03f232",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12774",
"triggerID" : "5d6e723cc2365690f99ee59a008988394e03f232",
"triggerType" : "PUSH"
}, {
"hash" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13147",
"triggerID" : "862be37daa8c19a5bcfd8f602ea2fedc94b556cd",
"triggerType" : "PUSH"
}, {
"hash" : "48ef299dc5efceb79aec32ff69266887885e7172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13217",
"triggerID" : "48ef299dc5efceb79aec32ff69266887885e7172",
"triggerType" : "PUSH"
}, {
"hash" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13240",
"triggerID" : "139a4b2aa768aa8195c2ed88028c902ea62fc77f",
"triggerType" : "PUSH"
}, {
"hash" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906",
"triggerID" : "d1694b05083fee989a712a00f01c771bb0f7f95f",
"triggerType" : "PUSH"
}, {
"hash" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d1694b05083fee989a712a00f01c771bb0f7f95f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13906)
* d2f4ce7779a835a6f524aabd8fa16c7c5dcc8c6e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org