You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "ad1happy2go (via GitHub)" <gi...@apache.org> on 2023/08/23 14:53:57 UTC
[GitHub] [hudi] ad1happy2go opened a new pull request, #9510: [HUDI-6739] Skip checking Active Timeline to improve performance
ad1happy2go opened a new pull request, #9510:
URL: https://github.com/apache/hudi/pull/9510
### Change Logs
Improved the logic for Fixing checkpoint management for multiple streaming writers, so that it dont check the active timeline if offset is 0 as that will mean it is a new writer.
### Impact
none
### Risk level (write none, low medium or high below)
low
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9510: [HUDI-6739] Skip checking Active Timeline to improve performance for multiple streaming writers
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9510:
URL: https://github.com/apache/hudi/pull/9510#issuecomment-1690681912
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "caf735fd3d8cba7d7e05ee7c8a08fcc6d810fb3f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19428",
"triggerID" : "caf735fd3d8cba7d7e05ee7c8a08fcc6d810fb3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* caf735fd3d8cba7d7e05ee7c8a08fcc6d810fb3f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19428)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on a diff in pull request #9510: [HUDI-6739] Skip checking Active Timeline to improve performance for multiple streaming writers
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on code in PR #9510:
URL: https://github.com/apache/hudi/pull/9510#discussion_r1309787047
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieStreamingSink.scala:
##########
@@ -318,14 +318,18 @@ class HoodieStreamingSink(sqlContext: SQLContext,
private def canSkipBatch(incomingBatchId: Long, operationType: String): Boolean = {
if (!DELETE_OPERATION_OPT_VAL.equals(operationType)) {
- val identifier = options.getOrElse(STREAMING_CHECKPOINT_IDENTIFIER.key(), STREAMING_CHECKPOINT_IDENTIFIER.defaultValue())
- // get the latest checkpoint from the commit metadata to check if the microbatch has already been processed or not
- val lastCheckpoint = CommitUtils.getValidCheckpointForCurrentWriter(
- metaClient.get.getActiveTimeline.getWriteTimeline, SINK_CHECKPOINT_KEY, identifier)
- if (lastCheckpoint.isPresent) {
- latestCommittedBatchId = lastCheckpoint.get().toLong
+ if(incomingBatchId == 0) {
+ false
Review Comment:
Yes, I agree @yihua . Thanks. Closing this PR then without committing. @nsivabalan Let me know if any concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9510: [HUDI-6739] Skip checking Active Timeline to improve performance for multiple streaming writers
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9510:
URL: https://github.com/apache/hudi/pull/9510#issuecomment-1690189633
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "caf735fd3d8cba7d7e05ee7c8a08fcc6d810fb3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "caf735fd3d8cba7d7e05ee7c8a08fcc6d810fb3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* caf735fd3d8cba7d7e05ee7c8a08fcc6d810fb3f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9510: [HUDI-6739] Skip checking Active Timeline to improve performance for multiple streaming writers
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9510:
URL: https://github.com/apache/hudi/pull/9510#issuecomment-1690206192
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "caf735fd3d8cba7d7e05ee7c8a08fcc6d810fb3f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19428",
"triggerID" : "caf735fd3d8cba7d7e05ee7c8a08fcc6d810fb3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* caf735fd3d8cba7d7e05ee7c8a08fcc6d810fb3f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19428)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go closed pull request #9510: [HUDI-6739] Skip checking Active Timeline to improve performance for multiple streaming writers
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go closed pull request #9510: [HUDI-6739] Skip checking Active Timeline to improve performance for multiple streaming writers
URL: https://github.com/apache/hudi/pull/9510
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #9510: [HUDI-6739] Skip checking Active Timeline to improve performance for multiple streaming writers
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9510:
URL: https://github.com/apache/hudi/pull/9510#discussion_r1305990912
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieStreamingSink.scala:
##########
@@ -318,14 +318,18 @@ class HoodieStreamingSink(sqlContext: SQLContext,
private def canSkipBatch(incomingBatchId: Long, operationType: String): Boolean = {
if (!DELETE_OPERATION_OPT_VAL.equals(operationType)) {
- val identifier = options.getOrElse(STREAMING_CHECKPOINT_IDENTIFIER.key(), STREAMING_CHECKPOINT_IDENTIFIER.defaultValue())
- // get the latest checkpoint from the commit metadata to check if the microbatch has already been processed or not
- val lastCheckpoint = CommitUtils.getValidCheckpointForCurrentWriter(
- metaClient.get.getActiveTimeline.getWriteTimeline, SINK_CHECKPOINT_KEY, identifier)
- if (lastCheckpoint.isPresent) {
- latestCommittedBatchId = lastCheckpoint.get().toLong
+ if(incomingBatchId == 0) {
+ false
Review Comment:
The batch 0 may have already been processed so the last checkpoint still needs to be read. Correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org