You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2023/09/13 23:14:10 UTC
[GitHub] [hudi] yihua opened a new pull request, #9711: [HUDI-6858] Fix checkpoint reading in Spark structured streaming
yihua opened a new pull request, #9711:
URL: https://github.com/apache/hudi/pull/9711
### Change Logs
This PR fixes the checkpoint reading in Hudi streaming sink in Spark structured streaming. The fix avoids reading the requested compaction or clustering plan which is serialized in Avro, causing the deserialization to fail and only scan the commit metadata of completed commits, which are serialized in JSON. Without the fix, the Avro serialized plan throws exceptions (see below). A new test is added to validate the correctness of reading checkpoint for Hudi streaming sink in Spark structured streaming.
```
org.apache.hudi.exception.HoodieIOException: Failed to parse HoodieCommitMetadata for [==>20230913003800000__compaction__REQUESTED__20230913155321000]
at org.apache.hudi.common.util.CommitUtils.lambda$getValidCheckpointForCurrentWriter$3(CommitUtils.java:180)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
...
Caused by: java.io.IOException: unable to read commit metadata
at org.apache.hudi.common.model.HoodieCommitMetadata.fromBytes(HoodieCommitMetadata.java:514)
at org.apache.hudi.common.util.CommitUtils.lambda$getValidCheckpointForCurrentWriter$3(CommitUtils.java:170)
... 77 more
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Objavro': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (String)"Obj\u0001\u0002\u0016avro.schema� {"type":"record","name":"HoodieCompactionPlan","namespace":"org.apache.hudi.avro.model","fields":[{"name":"operations","type":["null",{"type":"array","items":{"type":"record","name":"HoodieCompactionOperation","fields":[{"name":"baseInstantTime","type":["null",{"type":"string","avro.java.string":"String"}]},{"name":"deltaFilePaths","type":["null",{"type":"array","items":{"type":"string","avro.java.string":"String"}}],"default":null},{"name":"dataFilePath","type":["null",{"type"[truncated 1614 chars]; line: 1, column: 11]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:745)
```
```
org.apache.hudi.exception.HoodieIOException: Failed to parse HoodieCommitMetadata for [==>20230913004800000__replacecommit__REQUESTED__20230913155245000]
at org.apache.hudi.common.util.CommitUtils.lambda$getValidCheckpointForCurrentWriter$3(CommitUtils.java:180)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
Caused by: java.io.IOException: unable to read commit metadata
at org.apache.hudi.common.model.HoodieCommitMetadata.fromBytes(HoodieCommitMetadata.java:514)
at org.apache.hudi.common.util.CommitUtils.lambda$getValidCheckpointForCurrentWriter$3(CommitUtils.java:170)
... 77 more
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Objavro': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (String)"Obj\u0001\u0002\u0016avro.schema�&{"type":"record","name":"HoodieRequestedReplaceMetadata","namespace":"org.apache.hudi.avro.model","fields":[{"name":"operationType","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"clusteringPlan","type":["null",{"type":"record","name":"HoodieClusteringPlan","fields":[{"name":"inputGroups","type":["null",{"type":"array","items":{"type":"record","name":"HoodieClusteringGroup","fields":[{"name":"slices","type":["null",{"type":"array","items""[truncated 2037 chars]; line: 1, column: 11]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:745)
```
### Impact
Bug fix
### Risk level
low
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9711: [HUDI-6858] Fix checkpoint reading in Spark structured streaming
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9711:
URL: https://github.com/apache/hudi/pull/9711#issuecomment-1718450561
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc131a0b9e11bc3670456b0763b85ddd17b1c2e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cc131a0b9e11bc3670456b0763b85ddd17b1c2e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc131a0b9e11bc3670456b0763b85ddd17b1c2e5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9711: [HUDI-6858] Fix checkpoint reading in Spark structured streaming
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9711:
URL: https://github.com/apache/hudi/pull/9711#issuecomment-1718554842
So by default, spark structure streaming would skip all the compaction and clustering commits?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9711: [HUDI-6858] Fix checkpoint reading in Spark structured streaming
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9711:
URL: https://github.com/apache/hudi/pull/9711#issuecomment-1718484335
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc131a0b9e11bc3670456b0763b85ddd17b1c2e5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19895",
"triggerID" : "cc131a0b9e11bc3670456b0763b85ddd17b1c2e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc131a0b9e11bc3670456b0763b85ddd17b1c2e5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19895)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #9711: [HUDI-6858] Fix checkpoint reading in Spark structured streaming
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #9711:
URL: https://github.com/apache/hudi/pull/9711
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9711: [HUDI-6858] Fix checkpoint reading in Spark structured streaming
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9711:
URL: https://github.com/apache/hudi/pull/9711#issuecomment-1718711190
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc131a0b9e11bc3670456b0763b85ddd17b1c2e5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19895",
"triggerID" : "cc131a0b9e11bc3670456b0763b85ddd17b1c2e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc131a0b9e11bc3670456b0763b85ddd17b1c2e5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19895)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org