You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "voonhous (via GitHub)" <gi...@apache.org> on 2023/04/26 06:57:52 UTC
[GitHub] [hudi] voonhous opened a new pull request, #8579: [MINOR] Added docs on gotchas when using PartialUpdateAvroPayload
voonhous opened a new pull request, #8579:
URL: https://github.com/apache/hudi/pull/8579
### Change Logs
Updated java docs and added tests to demonstrate gotchas when using PartialUpdateAvroPayload.
### Impact
None, changes made are docs and tests to clarify gotchas on how PartialUpdateAvroPayload handles records that are arriving late (unordered data)
### Risk level (write none, low medium or high below)
None
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #8579: [MINOR] Added docs for gotchas when using PartialUpdateAvroPayload
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #8579:
URL: https://github.com/apache/hudi/pull/8579
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8579: [MINOR] Added docs for gotchas when using PartialUpdateAvroPayload
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8579:
URL: https://github.com/apache/hudi/pull/8579#discussion_r1177638616
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestPartialUpdateAvroPayload.scala:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi
+
+class TestPartialUpdateAvroPayload extends HoodieSparkSqlTestBase {
+
+ val partialUpdatePayloadClass = "org.apache.hudi.common.model.PartialUpdateAvroPayload"
+
Review Comment:
Can we add UT instead of ITs in existing class `TestPartialUpdateAvroPayload` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] voonhous commented on pull request #8579: [MINOR] Added docs for gotchas when using PartialUpdateAvroPayload
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #8579:
URL: https://github.com/apache/hudi/pull/8579#issuecomment-1523138078
> > Results will be different if combineAndUpdateValue is invoked in order without invoking preCombine.
>
> What is exactly lost here?
# PreCombine + combineAndGetUpdateValue
```
Table schema: {id: int , name: string, price: double, _ts: int}
recordKey: id
precombineField: _ts
Table initial state:
[1 a1_0 10.0 1001]
Table performs an update with an incoming batch that has the following results (2):
(precombine + combineAndGetUpdateValue)
[
[1 a1_0 11.0 999],
[1 a1_0 null 1001]
]
End state of the table:
[1 a1_0 11.0 1001]
```
This is so as the incoming batch at (2) will be `preCombine` before performing a `combineAndUpdateValue`.
# combineAndGetUpdateValue ONLY
Results will be different if `combineAndUpdateValue` is invoked in order without invoking `preCombine`.
Example:
```
Table initial state:
[1 a1_0 10.0 1001]
Table performs an update:
(combineAndGetUpdateValue)
[1 a1_0 11.0 999]
Table performs an update again:
(combineAndGetUpdateValue)
[1 a1_0 null 1001]
End state of the table:
[1 a1_0 10.0 1001]
```
[
[1 a1_0 11.0 999],
[1 a1_0 null 1001]
]
End state of the table:
[1 a1_0 11.0 1001]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] voonhous commented on pull request #8579: [MINOR] Added docs for gotchas when using PartialUpdateAvroPayload
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #8579:
URL: https://github.com/apache/hudi/pull/8579#issuecomment-1523770585
@danny0405 Yeap, this is not a bug, but will affect the query snapshot results.
Also MOR tables are more susceptible of encountering such issues (when performing a query on the _rt table/compaction).
The goal of this MR is not to fix any bugs, rather, it's to document and make known such behaviours so that users understand that this is by design. Or rather, a limitation of how Hudi performs merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8579: [MINOR] Added docs for gotchas when using PartialUpdateAvroPayload
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8579:
URL: https://github.com/apache/hudi/pull/8579#issuecomment-1542294565
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa50b514ec994cde256ae1f85778648dc94e5ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16671",
"triggerID" : "fa50b514ec994cde256ae1f85778648dc94e5ef6",
"triggerType" : "PUSH"
}, {
"hash" : "97fd8fb26924eecfce57c58168018dbd333b9be2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16996",
"triggerID" : "97fd8fb26924eecfce57c58168018dbd333b9be2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 97fd8fb26924eecfce57c58168018dbd333b9be2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16996)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8579: [MINOR] Added docs for gotchas when using PartialUpdateAvroPayload
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8579:
URL: https://github.com/apache/hudi/pull/8579#issuecomment-1541896282
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa50b514ec994cde256ae1f85778648dc94e5ef6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16671",
"triggerID" : "fa50b514ec994cde256ae1f85778648dc94e5ef6",
"triggerType" : "PUSH"
}, {
"hash" : "97fd8fb26924eecfce57c58168018dbd333b9be2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16996",
"triggerID" : "97fd8fb26924eecfce57c58168018dbd333b9be2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa50b514ec994cde256ae1f85778648dc94e5ef6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16671)
* 97fd8fb26924eecfce57c58168018dbd333b9be2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16996)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8579: [MINOR] Added docs for gotchas when using PartialUpdateAvroPayload
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8579:
URL: https://github.com/apache/hudi/pull/8579#issuecomment-1523541516
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa50b514ec994cde256ae1f85778648dc94e5ef6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16671",
"triggerID" : "fa50b514ec994cde256ae1f85778648dc94e5ef6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa50b514ec994cde256ae1f85778648dc94e5ef6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16671)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8579: [MINOR] Added docs on gotchas when using PartialUpdateAvroPayload
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8579:
URL: https://github.com/apache/hudi/pull/8579#issuecomment-1522918267
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa50b514ec994cde256ae1f85778648dc94e5ef6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fa50b514ec994cde256ae1f85778648dc94e5ef6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa50b514ec994cde256ae1f85778648dc94e5ef6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8579: [MINOR] Added docs for gotchas when using PartialUpdateAvroPayload
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8579:
URL: https://github.com/apache/hudi/pull/8579#issuecomment-1541884910
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa50b514ec994cde256ae1f85778648dc94e5ef6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16671",
"triggerID" : "fa50b514ec994cde256ae1f85778648dc94e5ef6",
"triggerType" : "PUSH"
}, {
"hash" : "97fd8fb26924eecfce57c58168018dbd333b9be2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "97fd8fb26924eecfce57c58168018dbd333b9be2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa50b514ec994cde256ae1f85778648dc94e5ef6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16671)
* 97fd8fb26924eecfce57c58168018dbd333b9be2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8579: [MINOR] Added docs for gotchas when using PartialUpdateAvroPayload
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8579:
URL: https://github.com/apache/hudi/pull/8579#issuecomment-1523173556
> If preCombine is invoked with the same key when an old data {price: 11.00, _ts:999} is received together with a new data {price: null, _ts: 1001}, the old data's column value might overwrite the existing newer data {price: 10.0, _ts: 1000}.
This is expected right ? We always ignore the nulls while merging, shouldn't the `#combineAndGetUpdateValue` follow the same convention?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8579: [MINOR] Added docs for gotchas when using PartialUpdateAvroPayload
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8579:
URL: https://github.com/apache/hudi/pull/8579#issuecomment-1523133981
> Results will be different if combineAndUpdateValue is invoked in order without invoking preCombine.
What is exactly lost here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8579: [MINOR] Added docs on gotchas when using PartialUpdateAvroPayload
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8579:
URL: https://github.com/apache/hudi/pull/8579#issuecomment-1522928842
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa50b514ec994cde256ae1f85778648dc94e5ef6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16671",
"triggerID" : "fa50b514ec994cde256ae1f85778648dc94e5ef6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa50b514ec994cde256ae1f85778648dc94e5ef6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16671)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8579: [MINOR] Added docs for gotchas when using PartialUpdateAvroPayload
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8579:
URL: https://github.com/apache/hudi/pull/8579#issuecomment-1523481553
Okay, got the idea, you are addressing that the always merging behavior can be variable with different sequence of inputs, that's true, and it is not a bug I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] voonhous commented on pull request #8579: [MINOR] Added docs for gotchas when using PartialUpdateAvroPayload
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #8579:
URL: https://github.com/apache/hudi/pull/8579#issuecomment-1523206446
> > If preCombine is invoked with the same key when an old data {price: 11.00, _ts:999} is received together with a new data {price: null, _ts: 1001}, the old data's column value might overwrite the existing newer data {price: 10.0, _ts: 1000}.
>
> This is expected right ? We always ignore the nulls while merging, shouldn't the `#combineAndGetUpdateValue` follow the same convention?
`#combineAndGetUpdate` does follow the same convention. The crux of the gotcha here is that if a batch contains multiple records of the same key, it will produce different results when `#combineAndGetUpdateValue` individually.
**NOTE:**
My bad, the initial precombineField value of the table's initial state is wrong. I've edited the previous examples.
Let me provide an example again:
# preCombine + combineAndGetUpdateValue
```
Table initial state (1):
[1 a1_0 10.0 1000]
Table performs an update with an incoming batch that has the following results (2):
(preCombine + combineAndGetUpdateValue)
[
[1 a1_0 11.0 999],
[1 a1_0 null 1001]
]
After preCombine results from (2), we will get (3):
[1 a1_0 11.0 1001]
This will be combineAndGetUpdateValue with (1) to produce:
> (1) + (3)
[1 a1_0 11.0 1001]
```
# combineAndGetUpdateValue only
```
Table initial state (1):
[1 a1_0 10.0 1000]
Table performs an update (2):
(combineAndGetUpdateValue)
[1 a1_0 11.0 999]
to produce (3) [NO CHANGE]:
[1 a1_0 10.0 1000]
Table performs an update again (3):
(combineAndGetUpdateValue)
[1 a1_0 null 1001]
End state of the table:
> (2) + (3):
[1 a1_0 10.0 1001]
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] voonhous commented on a diff in pull request #8579: [MINOR] Added docs for gotchas when using PartialUpdateAvroPayload
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #8579:
URL: https://github.com/apache/hudi/pull/8579#discussion_r1189661223
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestPartialUpdateAvroPayload.scala:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi
+
+class TestPartialUpdateAvroPayload extends HoodieSparkSqlTestBase {
+
+ val partialUpdatePayloadClass = "org.apache.hudi.common.model.PartialUpdateAvroPayload"
+
Review Comment:
Done, can you please take a look at this PR again? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org