You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2023/01/23 18:01:51 UTC
[GitHub] [hudi] codope opened a new pull request, #7737: [HUDI-5593] Move post-commit operations out of write transaction
codope opened a new pull request, #7737:
URL: https://github.com/apache/hudi/pull/7737
### Change Logs
As part of post-commit operations, the write client will async clean and archive if enabled. This PR moves that out of write transaction. Within the same JVM, we use InProcessLockProvider and if we execute async clean/archival within that then sometimes the async service will fail to acquire lock.
### Impact
No public API change. Resolves a test failure.
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed pull request #7737: [HUDI-5593] Move post-commit operations out of write transaction
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed pull request #7737: [HUDI-5593] Move post-commit operations out of write transaction
URL: https://github.com/apache/hudi/pull/7737
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7737: [HUDI-5593] Move post-commit operations out of write transaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7737:
URL: https://github.com/apache/hudi/pull/7737#issuecomment-1400787903
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4690c79e843d7be21418777ef76a98f6345563c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14563",
"triggerID" : "e4690c79e843d7be21418777ef76a98f6345563c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e4690c79e843d7be21418777ef76a98f6345563c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14563)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7737: [HUDI-5593] Move post-commit operations out of write transaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7737:
URL: https://github.com/apache/hudi/pull/7737#issuecomment-1400777480
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4690c79e843d7be21418777ef76a98f6345563c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e4690c79e843d7be21418777ef76a98f6345563c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e4690c79e843d7be21418777ef76a98f6345563c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #7737: [HUDI-5593] Move post-commit operations out of write transaction
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #7737:
URL: https://github.com/apache/hudi/pull/7737#issuecomment-1401392135
Closing in favor #7739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7737: [HUDI-5593] Move post-commit operations out of write transaction
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7737:
URL: https://github.com/apache/hudi/pull/7737#discussion_r1084393849
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -244,6 +242,7 @@ public boolean commitStats(String instantTime, List<HoodieWriteStat> stats, Opti
// We don't want to fail the commit if hoodie.fail.writes.on.inline.table.service.exception is false. We catch warn if false
try {
// do this outside of lock since compaction, clustering can be time taking and we don't need a lock for the entire execution period
+ postCommit(table, metadata, instantTime, extraMetadata, false);
Review Comment:
there is much more to this. this is skipping locks for inline cleaning. I will take over from here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7737: [HUDI-5593] Move post-commit operations out of write transaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7737:
URL: https://github.com/apache/hudi/pull/7737#issuecomment-1401100072
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4690c79e843d7be21418777ef76a98f6345563c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14563",
"triggerID" : "e4690c79e843d7be21418777ef76a98f6345563c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e4690c79e843d7be21418777ef76a98f6345563c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14563)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org