You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/18 10:25:26 UTC
[GitHub] [hudi] aliceyyan opened a new pull request, #5626: the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
aliceyyan opened a new pull request, #5626:
URL: https://github.com/apache/hudi/pull/5626
## What is the purpose of the pull request
the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
## Brief change log
- add tableDataExists method in class StreamerUtil;
- modify metaClientForReader method in class StreamerUtil;
## Committer checklist
- [4119] Has a corresponding JIRA in PR title & commit
the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi .
ETL path: flink upsert-kafka connector -> hudi table (MOR table,query by stream)
Here is the case:
1. the first time: write two records with the same primary key into kafka, and insert them into hudi table. the query result should be three records: +I first record, -U first record, +U second record; But the first time I query hudi table, I found that all the data operation were +I: +I first record,+I first record and +I second record, and there was no update operation;
Three times +I has affected hudi's subsequent ETL process-the data of groupBy is inaccurate;
2. Second time: Exit the first query, restart the query job of hudi table, and the query results are normal: +I first data, -U first data, +U second data.
Reason:
Reason:There is a bug in the program. When no data log file is generated, the Schema does not include the column' _ hoodie _ operation'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1129842650
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 422c808939ad98051f4b83bb2353191167b6edb0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1129937468
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 422c808939ad98051f4b83bb2353191167b6edb0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aliceyyan commented on a diff in pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
aliceyyan commented on code in PR #5626:
URL: https://github.com/apache/hudi/pull/5626#discussion_r877018237
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/format/TestInputFormat.java:
##########
@@ -481,7 +489,7 @@ void testReadMORWithCompactionPlanScheduled() throws Exception {
List<RowData> dataset = TestData.dataSetInsert(i + 1, i + 2);
TestData.writeData(dataset, conf);
}
-
+ createTableSource();
Review Comment:
If getTableSource method is called before data is written, metaClient will be null, and a call to tablesource.reset () in the unit test method generate null pointer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1131566982
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
}, {
"hash" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756",
"triggerID" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"triggerType" : "PUSH"
}, {
"hash" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8765",
"triggerID" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 21cde89ce0f1b094454c789636e3c0dad33c061a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8765)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1131576627
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
}, {
"hash" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756",
"triggerID" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"triggerType" : "PUSH"
}, {
"hash" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8765",
"triggerID" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"triggerType" : "PUSH"
}, {
"hash" : "781e5c94b698f987dbf46fbb5b57a813aa0175c9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8766",
"triggerID" : "781e5c94b698f987dbf46fbb5b57a813aa0175c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 21cde89ce0f1b094454c789636e3c0dad33c061a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8765)
* 781e5c94b698f987dbf46fbb5b57a813aa0175c9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8766)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1131058558
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
}, {
"hash" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756",
"triggerID" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 422c808939ad98051f4b83bb2353191167b6edb0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740)
* 284ce7503ad459a635148b0761bb3a5ebc9b9de6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1131379858
Thanks for the contribution, please fix the test failure first ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aliceyyan commented on a diff in pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
aliceyyan commented on code in PR #5626:
URL: https://github.com/apache/hudi/pull/5626#discussion_r877683731
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/format/TestInputFormat.java:
##########
@@ -481,7 +489,7 @@ void testReadMORWithCompactionPlanScheduled() throws Exception {
List<RowData> dataset = TestData.dataSetInsert(i + 1, i + 2);
TestData.writeData(dataset, conf);
}
-
+ createTableSource();
Review Comment:
@danny0405 ok,You mean change the logic below? This will not affect batch reading：😄
<img width="927" alt="image" src="https://user-images.githubusercontent.com/104287562/169437477-bad7944e-0812-41c8-97e5-a4ee7b5249f0.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1132589104
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
}, {
"hash" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756",
"triggerID" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"triggerType" : "PUSH"
}, {
"hash" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8765",
"triggerID" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"triggerType" : "PUSH"
}, {
"hash" : "781e5c94b698f987dbf46fbb5b57a813aa0175c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8766",
"triggerID" : "781e5c94b698f987dbf46fbb5b57a813aa0175c9",
"triggerType" : "PUSH"
}, {
"hash" : "78ce0d4f18f3d6a415b5b3f87d09ca41efaf0406",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8778",
"triggerID" : "78ce0d4f18f3d6a415b5b3f87d09ca41efaf0406",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 781e5c94b698f987dbf46fbb5b57a813aa0175c9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8766)
* 78ce0d4f18f3d6a415b5b3f87d09ca41efaf0406 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8778)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars merged pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars merged PR #5626:
URL: https://github.com/apache/hudi/pull/5626
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1132586144
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
}, {
"hash" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756",
"triggerID" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"triggerType" : "PUSH"
}, {
"hash" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8765",
"triggerID" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"triggerType" : "PUSH"
}, {
"hash" : "781e5c94b698f987dbf46fbb5b57a813aa0175c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8766",
"triggerID" : "781e5c94b698f987dbf46fbb5b57a813aa0175c9",
"triggerType" : "PUSH"
}, {
"hash" : "78ce0d4f18f3d6a415b5b3f87d09ca41efaf0406",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "78ce0d4f18f3d6a415b5b3f87d09ca41efaf0406",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 781e5c94b698f987dbf46fbb5b57a813aa0175c9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8766)
* 78ce0d4f18f3d6a415b5b3f87d09ca41efaf0406 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aliceyyan commented on a diff in pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
aliceyyan commented on code in PR #5626:
URL: https://github.com/apache/hudi/pull/5626#discussion_r877683731
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/format/TestInputFormat.java:
##########
@@ -481,7 +489,7 @@ void testReadMORWithCompactionPlanScheduled() throws Exception {
List<RowData> dataset = TestData.dataSetInsert(i + 1, i + 2);
TestData.writeData(dataset, conf);
}
-
+ createTableSource();
Review Comment:
@danny0405 ok,You mean change the logic below? This will not affect batch reading：😄
<img width="927" alt="image" src="https://user-images.githubusercontent.com/104287562/169437477-bad7944e-0812-41c8-97e5-a4ee7b5249f0.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aliceyyan commented on a diff in pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
aliceyyan commented on code in PR #5626:
URL: https://github.com/apache/hudi/pull/5626#discussion_r877687552
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/format/TestInputFormat.java:
##########
@@ -481,7 +489,7 @@ void testReadMORWithCompactionPlanScheduled() throws Exception {
List<RowData> dataset = TestData.dataSetInsert(i + 1, i + 2);
TestData.writeData(dataset, conf);
}
-
+ createTableSource();
Review Comment:
ok,get it ! ~ 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1129845821
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 422c808939ad98051f4b83bb2353191167b6edb0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5626:
URL: https://github.com/apache/hudi/pull/5626#discussion_r877662449
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/format/TestInputFormat.java:
##########
@@ -481,7 +489,7 @@ void testReadMORWithCompactionPlanScheduled() throws Exception {
List<RowData> dataset = TestData.dataSetInsert(i + 1, i + 2);
TestData.writeData(dataset, conf);
}
-
+ createTableSource();
Review Comment:
Thanks, i think the change is risky, because the null value is only handled in `HoodieTableSource#getStreamInputFormat`, original i put a reader meta client for the case where streaming reader starts but there is no data for the table (which is a common case for streaming read), so let's limit the change only to streaming read too and then i think we do not need to change the test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1131570158
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
}, {
"hash" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756",
"triggerID" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"triggerType" : "PUSH"
}, {
"hash" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8765",
"triggerID" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"triggerType" : "PUSH"
}, {
"hash" : "781e5c94b698f987dbf46fbb5b57a813aa0175c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "781e5c94b698f987dbf46fbb5b57a813aa0175c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 21cde89ce0f1b094454c789636e3c0dad33c061a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8765)
* 781e5c94b698f987dbf46fbb5b57a813aa0175c9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1131519160
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
}, {
"hash" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756",
"triggerID" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"triggerType" : "PUSH"
}, {
"hash" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 284ce7503ad459a635148b0761bb3a5ebc9b9de6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756)
* 21cde89ce0f1b094454c789636e3c0dad33c061a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5626:
URL: https://github.com/apache/hudi/pull/5626#discussion_r876956983
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/format/TestInputFormat.java:
##########
@@ -481,7 +489,7 @@ void testReadMORWithCompactionPlanScheduled() throws Exception {
List<RowData> dataset = TestData.dataSetInsert(i + 1, i + 2);
TestData.writeData(dataset, conf);
}
-
+ createTableSource();
Review Comment:
Can we explain a little more why this test has been affected by your change ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1131684791
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
}, {
"hash" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756",
"triggerID" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"triggerType" : "PUSH"
}, {
"hash" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8765",
"triggerID" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"triggerType" : "PUSH"
}, {
"hash" : "781e5c94b698f987dbf46fbb5b57a813aa0175c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8766",
"triggerID" : "781e5c94b698f987dbf46fbb5b57a813aa0175c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 781e5c94b698f987dbf46fbb5b57a813aa0175c9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8766)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1131053372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
}, {
"hash" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 422c808939ad98051f4b83bb2353191167b6edb0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740)
* 284ce7503ad459a635148b0761bb3a5ebc9b9de6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aliceyyan commented on a diff in pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
aliceyyan commented on code in PR #5626:
URL: https://github.com/apache/hudi/pull/5626#discussion_r877931303
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/format/TestInputFormat.java:
##########
@@ -481,7 +489,7 @@ void testReadMORWithCompactionPlanScheduled() throws Exception {
List<RowData> dataset = TestData.dataSetInsert(i + 1, i + 2);
TestData.writeData(dataset, conf);
}
-
+ createTableSource();
Review Comment:
@danny0405 hi danny,I have upgraded the program, please help me to CR,thank you !~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1131186033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
}, {
"hash" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756",
"triggerID" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 284ce7503ad459a635148b0761bb3a5ebc9b9de6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1131522346
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
}, {
"hash" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756",
"triggerID" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"triggerType" : "PUSH"
}, {
"hash" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8765",
"triggerID" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 284ce7503ad459a635148b0761bb3a5ebc9b9de6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756)
* 21cde89ce0f1b094454c789636e3c0dad33c061a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8765)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1132690022
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "422c808939ad98051f4b83bb2353191167b6edb0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8740",
"triggerID" : "422c808939ad98051f4b83bb2353191167b6edb0",
"triggerType" : "PUSH"
}, {
"hash" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8756",
"triggerID" : "284ce7503ad459a635148b0761bb3a5ebc9b9de6",
"triggerType" : "PUSH"
}, {
"hash" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8765",
"triggerID" : "21cde89ce0f1b094454c789636e3c0dad33c061a",
"triggerType" : "PUSH"
}, {
"hash" : "781e5c94b698f987dbf46fbb5b57a813aa0175c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8766",
"triggerID" : "781e5c94b698f987dbf46fbb5b57a813aa0175c9",
"triggerType" : "PUSH"
}, {
"hash" : "78ce0d4f18f3d6a415b5b3f87d09ca41efaf0406",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8778",
"triggerID" : "78ce0d4f18f3d6a415b5b3f87d09ca41efaf0406",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 78ce0d4f18f3d6a415b5b3f87d09ca41efaf0406 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8778)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #5626: [HUDI-4119] the first read result is incorrect when Flink upsert- Kafka connector is used in HUDi
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #5626:
URL: https://github.com/apache/hudi/pull/5626#issuecomment-1132706263
+1, @XuQianJin-Stars , please remember to refactor the code when `TableSchemaResolver` is refactored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org