You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/10/29 02:27:07 UTC
[GitHub] [hudi] zhangyue19921010 opened a new pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
zhangyue19921010 opened a new pull request #3887:
URL: https://github.com/apache/hudi/pull/3887
https://issues.apache.org/jira/browse/HUDI-2648
## What is the purpose of the pull request
Hoodie will do lots of list/get/put/delete etc actions on filesystem.
Sometimes will meet the fileSystem performance issue or short service suspension, for example
Using hoodie on S3, and will throw lots of exception like this
```
com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 503, AWS Service: Amazon S3, AWS Request ID: J58R9W4E324DWJ1Y, AWS Error Code: SlowDown, AWS Error Message: Please reduce your request rate., S3 Extended Request ID: Pkd6VCJKTuq3NJ6vUQJB1TUznKGvAfjIZkmJOxt+j6oAD5roz+ojKf9rtcBOgUBw4sWeDwxdTv4=
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393)
at org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:1191)
at org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:1168)
at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:871)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1881)
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.lambda$mkdirs$25(HoodieWrapperFileSystem.java:632)
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.executeFuncWithTimeMetrics(HoodieWrapperFileSystem.java:100)
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.mkdirs(HoodieWrapperFileSystem.java:631)
at org.apache.hudi.table.MarkerFiles.create(MarkerFiles.java:195)
at org.apache.hudi.io.HoodieWriteHandle.createMarkerFile(HoodieWriteHandle.java:129)
at org.apache.hudi.io.HoodieCreateHandle.lambda$new$1(HoodieCreateHandle.java:97)
at org.apache.hudi.common.util.RetryHelper.start(RetryHelper.java:61)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:100)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:67)
at org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:34)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
It's because of meet the throttle of S3 list/get/put/delete.
https://aws.amazon.com/premiumsupport/knowledge-center/s3-request-limit-avoid-throttling/
This exception will cause app crashed, which can be solved by retry wrapper.
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000161779
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 04f93a2e6825a202642e061c831a06b4d03ccf36 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1033694266
Hi @nsivabalan and @danny0405 Thanks a lot for you review.
Now we add a property named supportsRetry each StorageScheme. And check it before retry.
Also and a new config named `hoodie.filesystem.operation.retry.exceptions` to give users an option to decide which Exceptions that needs to be re-tryed. It could solve the overprotection problem.
> In that case, I think we can just have the config alone. and let the user decide. From vinothchandar
updated: only check `hoodie.filesystem.operation.retry.enable` before using retryHelper.
PTAL :) Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1031944330
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1036985962
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833",
"triggerID" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"triggerType" : "PUSH"
}, {
"hash" : "c83f659752b5aa3f078b3135ad45f14596e6ea04",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c83f659752b5aa3f078b3135ad45f14596e6ea04",
"triggerType" : "PUSH"
}, {
"hash" : "9d1366b79213a74fe3eb8b70ec49fa50e31b168c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5937",
"triggerID" : "9d1366b79213a74fe3eb8b70ec49fa50e31b168c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* c83f659752b5aa3f078b3135ad45f14596e6ea04 UNKNOWN
* 9d1366b79213a74fe3eb8b70ec49fa50e31b168c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5937)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954363971
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* 7b6b7630351b60fb75c1e4b1767975636db42932 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-956857417
@hudi-bot azure run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r745226109
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java
##########
@@ -63,9 +63,14 @@ protected HoodieSparkTable(HoodieWriteConfig config, HoodieEngineContext context
public static <T extends HoodieRecordPayload> HoodieSparkTable<T> create(HoodieWriteConfig config, HoodieEngineContext context,
boolean refreshTimeline) {
HoodieTableMetaClient metaClient =
- HoodieTableMetaClient.builder().setConf(context.getHadoopConf().get()).setBasePath(config.getBasePath())
- .setLoadActiveTimelineOnLoad(true).setConsistencyGuardConfig(config.getConsistencyGuardConfig())
- .setLayoutVersion(Option.of(new TimelineLayoutVersion(config.getTimelineLayoutVersion()))).build();
+ HoodieTableMetaClient.builder()
+ .setConf(context.getHadoopConf().get())
+ .setBasePath(config.getBasePath())
+ .setLoadActiveTimelineOnLoad(true)
+ .setConsistencyGuardConfig(config.getConsistencyGuardConfig())
+ .setFileSystemGuardConfig(config.getFileSystemGuardConfig())
Review comment:
Sure, thing. Will get it done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-998434340
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* 7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954363971
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* b100a200602217c196c37e131a4768c5c8d97727 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-961589864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* b100a200602217c196c37e131a4768c5c8d97727 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] mincwang commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
mincwang commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-962556117
hi @zhangyue19921010 ，i think also require add about `setFileSystemGuardConfig` in `HoodieFlinkTable` calss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000105066
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* 1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688)
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000127666
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* 1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696)
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 04f93a2e6825a202642e061c831a06b4d03ccf36 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000123975
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r774391547
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieClient.java
##########
@@ -132,9 +132,14 @@ protected void initWrapperFSMetrics() {
}
protected HoodieTableMetaClient createMetaClient(boolean loadActiveTimelineOnLoad) {
- return HoodieTableMetaClient.builder().setConf(hadoopConf).setBasePath(config.getBasePath())
- .setLoadActiveTimelineOnLoad(loadActiveTimelineOnLoad).setConsistencyGuardConfig(config.getConsistencyGuardConfig())
- .setLayoutVersion(Option.of(new TimelineLayoutVersion(config.getTimelineLayoutVersion()))).build();
+ return HoodieTableMetaClient.builder()
Review comment:
Changed :)
##########
File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/HoodieFlinkTable.java
##########
@@ -51,9 +51,14 @@ protected HoodieFlinkTable(HoodieWriteConfig config, HoodieEngineContext context
public static <T extends HoodieRecordPayload> HoodieFlinkTable<T> create(HoodieWriteConfig config, HoodieFlinkEngineContext context) {
HoodieTableMetaClient metaClient =
- HoodieTableMetaClient.builder().setConf(context.getHadoopConf().get()).setBasePath(config.getBasePath())
- .setLoadActiveTimelineOnLoad(true).setConsistencyGuardConfig(config.getConsistencyGuardConfig())
- .setLayoutVersion(Option.of(new TimelineLayoutVersion(config.getTimelineLayoutVersion()))).build();
+ HoodieTableMetaClient.builder()
Review comment:
changed :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000129142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* 1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696)
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 04f93a2e6825a202642e061c831a06b4d03ccf36 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-998416466
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* e314a3c3cbe9a90b4d5f72d2b46a157985288ea1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222)
* 7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-992064349
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* fe0c868afdbc57efd8628c7380da7469e5108476 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325)
* e314a3c3cbe9a90b4d5f72d2b46a157985288ea1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1036950381
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833",
"triggerID" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"triggerType" : "PUSH"
}, {
"hash" : "c83f659752b5aa3f078b3135ad45f14596e6ea04",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c83f659752b5aa3f078b3135ad45f14596e6ea04",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 1139ead2daf93239741fa6e25b1dd66381fb5c9c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833)
* c83f659752b5aa3f078b3135ad45f14596e6ea04 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r805105304
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -253,7 +265,16 @@ public TimelineLayoutVersion getTimelineLayoutVersion() {
*/
public HoodieWrapperFileSystem getFs() {
if (fs == null) {
- FileSystem fileSystem = FSUtils.getFs(metaPath, hadoopConf.newCopy());
+ FileSystem fileSystem;
+ if (fileSystemRetryConfig.isFileSystemActionRetryEnable()) {
Review comment:
Ack and changed as your suggestion. Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1033599122
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833",
"triggerID" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 1139ead2daf93239741fa6e25b1dd66381fb5c9c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-966804166
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* b100a200602217c196c37e131a4768c5c8d97727 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954)
* fe0c868afdbc57efd8628c7380da7469e5108476 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r754700441
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FileSystemGuardConfig.java
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * The consistency guard relevant config options.
+ */
+@ConfigClassProperty(name = "FileSystem Guard Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "The filesystem guard related config options, to help deal with runtime exception like s3 list/get/put/delete performance issues.")
+public class FileSystemGuardConfig extends HoodieConfig {
+
+ public static final ConfigProperty<String> FILESYSTEM_RETRY_ENABLE = ConfigProperty
+ .key("hoodie.filesystem.action.retry.enabled")
+ .defaultValue("false")
+ .sinceVersion("0.10.0")
+ .withDocumentation("Enabled to handle S3 list/get/delete etc file system performance issue.");
+
+ public static final ConfigProperty<Long> INITIAL_RETRY_INTERVAL_MS = ConfigProperty
+ .key("hoodie.filesystem.action.retry.initial_interval_ms")
+ .defaultValue(100L)
+ .sinceVersion("0.10.0")
+ .withDocumentation("Amount of time (in ms) to wait, before retry to do operations on storage.");
+
+ public static final ConfigProperty<Long> MAX_RETRY_INTERVAL_MS = ConfigProperty
+ .key("hoodie.filesystem.action.retry.max_interval_ms")
+ .defaultValue(2000L)
+ .sinceVersion("0.10.0")
+ .withDocumentation("Maximum amount of time (in ms), to wait for next retry.");
+
+ public static final ConfigProperty<Integer> MAX_RETRY_NUMBERS = ConfigProperty
+ .key("hoodie.filesystem.action.retry.max_numbers")
Review comment:
can you help me understand, why do we need both max retry number and max internal ms ? I thought either one is good enough.
So, either 100*4 = 4 retries w/ 100 ms delay.
or 2000/100 = 20 retries w/ 100 ms delay.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/RetryHelper.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Random;
+
+public class RetryHelper<T> {
+ private static final Logger LOG = LogManager.getLogger(RetryHelper.class);
+ private HoodieWrapperFileSystem.CheckedFunction<T> func;
+ private int num;
+ private long maxIntervalTime;
+ private long initialIntervalTime = 100L;
+ private String taskInfo = "N/A";
+
+ public RetryHelper() {
+ }
+
+ public RetryHelper(String taskInfo) {
+ this.taskInfo = taskInfo;
+ }
+
+ public RetryHelper tryWith(HoodieWrapperFileSystem.CheckedFunction<T> func) {
+ this.func = func;
+ return this;
+ }
+
+ public RetryHelper tryNum(int num) {
+ this.num = num;
+ return this;
+ }
+
+ public RetryHelper tryTaskInfo(String taskInfo) {
+ this.taskInfo = taskInfo;
+ return this;
+ }
+
+ public RetryHelper tryMaxInterval(long time) {
+ maxIntervalTime = time;
+ return this;
+ }
+
+ public RetryHelper tryInitialInterval(long time) {
+ initialIntervalTime = time;
+ return this;
+ }
+
+ public T start() throws IOException {
+ int retries = 0;
+ boolean success = false;
+ RuntimeException exception = null;
+ T t = null;
Review comment:
can we please name the variables nicely.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FileSystemGuardConfig.java
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * The consistency guard relevant config options.
+ */
+@ConfigClassProperty(name = "FileSystem Guard Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "The filesystem guard related config options, to help deal with runtime exception like s3 list/get/put/delete performance issues.")
+public class FileSystemGuardConfig extends HoodieConfig {
+
+ public static final ConfigProperty<String> FILESYSTEM_RETRY_ENABLE = ConfigProperty
+ .key("hoodie.filesystem.action.retry.enabled")
Review comment:
"enable" would suffice. we don't need a "d" in the end, to be in line with other configs
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/RetryHelper.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Random;
+
+public class RetryHelper<T> {
+ private static final Logger LOG = LogManager.getLogger(RetryHelper.class);
+ private HoodieWrapperFileSystem.CheckedFunction<T> func;
+ private int num;
+ private long maxIntervalTime;
+ private long initialIntervalTime = 100L;
+ private String taskInfo = "N/A";
+
+ public RetryHelper() {
+ }
+
+ public RetryHelper(String taskInfo) {
+ this.taskInfo = taskInfo;
+ }
+
+ public RetryHelper tryWith(HoodieWrapperFileSystem.CheckedFunction<T> func) {
+ this.func = func;
+ return this;
+ }
+
+ public RetryHelper tryNum(int num) {
+ this.num = num;
+ return this;
+ }
+
+ public RetryHelper tryTaskInfo(String taskInfo) {
+ this.taskInfo = taskInfo;
+ return this;
+ }
+
+ public RetryHelper tryMaxInterval(long time) {
+ maxIntervalTime = time;
+ return this;
+ }
+
+ public RetryHelper tryInitialInterval(long time) {
+ initialIntervalTime = time;
+ return this;
+ }
+
+ public T start() throws IOException {
+ int retries = 0;
+ boolean success = false;
+ RuntimeException exception = null;
+ T t = null;
+ do {
+ long waitTime = Math.min(getWaitTimeExp(retries), maxIntervalTime);
+ try {
+ t = func.get();
+ success = true;
+ break;
+ } catch (RuntimeException e) {
+ // deal with RuntimeExceptions such like AmazonS3Exception 503
+ exception = e;
+ LOG.warn("Catch RuntimeException " + taskInfo + ", will retry after " + waitTime + " ms.", e);
+ try {
+ Thread.sleep(waitTime);
+ } catch (InterruptedException ex) {
+ // ignore InterruptedException here
+ }
+ retries++;
+ }
+ } while (retries <= num);
Review comment:
we can probably remove L88 and do retries++ here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r752872257
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java
##########
@@ -63,9 +63,14 @@ protected HoodieSparkTable(HoodieWriteConfig config, HoodieEngineContext context
public static <T extends HoodieRecordPayload> HoodieSparkTable<T> create(HoodieWriteConfig config, HoodieEngineContext context,
boolean refreshTimeline) {
HoodieTableMetaClient metaClient =
- HoodieTableMetaClient.builder().setConf(context.getHadoopConf().get()).setBasePath(config.getBasePath())
- .setLoadActiveTimelineOnLoad(true).setConsistencyGuardConfig(config.getConsistencyGuardConfig())
- .setLayoutVersion(Option.of(new TimelineLayoutVersion(config.getTimelineLayoutVersion()))).build();
+ HoodieTableMetaClient.builder()
+ .setConf(context.getHadoopConf().get())
+ .setBasePath(config.getBasePath())
+ .setLoadActiveTimelineOnLoad(true)
+ .setConsistencyGuardConfig(config.getConsistencyGuardConfig())
+ .setFileSystemGuardConfig(config.getFileSystemGuardConfig())
Review comment:
cc @danny0405 :) Hope you are interested in this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-990963440
> sure, makes sense if there are other cloud stores that needs this retry. Can you please address the feedback given already.
Sure, I will address the comments ASAP. Thanks a lot for your attention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] mincwang commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
mincwang commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r744206294
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java
##########
@@ -63,9 +63,14 @@ protected HoodieSparkTable(HoodieWriteConfig config, HoodieEngineContext context
public static <T extends HoodieRecordPayload> HoodieSparkTable<T> create(HoodieWriteConfig config, HoodieEngineContext context,
boolean refreshTimeline) {
HoodieTableMetaClient metaClient =
- HoodieTableMetaClient.builder().setConf(context.getHadoopConf().get()).setBasePath(config.getBasePath())
- .setLoadActiveTimelineOnLoad(true).setConsistencyGuardConfig(config.getConsistencyGuardConfig())
- .setLayoutVersion(Option.of(new TimelineLayoutVersion(config.getTimelineLayoutVersion()))).build();
+ HoodieTableMetaClient.builder()
+ .setConf(context.getHadoopConf().get())
+ .setBasePath(config.getBasePath())
+ .setLoadActiveTimelineOnLoad(true)
+ .setConsistencyGuardConfig(config.getConsistencyGuardConfig())
+ .setFileSystemGuardConfig(config.getFileSystemGuardConfig())
Review comment:
hi @zhangyue19921010 ，i think also require add about setFileSystemGuardConfig in HoodieFlinkTable calss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-963742359
Hi @vinothchandar and @bvaradar. Friendly ping. I notice that you are very familiar with hudi filesystem wrapper. So could you please pay a little attention for this patch if it's possible？According to our Hudi on S3 experience, it's a common issue to meet the S3 threshold, and may need this retry wrapper to solve it.
Looking forward to your replay. Thanks in advance! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-998445078
@zhangyue19921010 Thanks for patiently working through this, with us. I am wondering if we can guard this behavior for each storage scheme.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000124692
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* 1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696)
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 04f93a2e6825a202642e061c831a06b4d03ccf36 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000129142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* 1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696)
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 04f93a2e6825a202642e061c831a06b4d03ccf36 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r767386257
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FileSystemGuardConfig.java
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * The consistency guard relevant config options.
+ */
+@ConfigClassProperty(name = "FileSystem Guard Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "The filesystem guard related config options, to help deal with runtime exception like s3 list/get/put/delete performance issues.")
+public class FileSystemGuardConfig extends HoodieConfig {
Review comment:
Sure, "FileSystemRetryConfig" is more appropriate. Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1036950381
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833",
"triggerID" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"triggerType" : "PUSH"
}, {
"hash" : "c83f659752b5aa3f078b3135ad45f14596e6ea04",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c83f659752b5aa3f078b3135ad45f14596e6ea04",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 1139ead2daf93239741fa6e25b1dd66381fb5c9c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833)
* c83f659752b5aa3f078b3135ad45f14596e6ea04 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r790371241
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -253,7 +265,16 @@ public TimelineLayoutVersion getTimelineLayoutVersion() {
*/
public HoodieWrapperFileSystem getFs() {
if (fs == null) {
- FileSystem fileSystem = FSUtils.getFs(metaPath, hadoopConf.newCopy());
+ FileSystem fileSystem;
+ if (fileSystemRetryConfig.isFileSystemActionRetryEnable()) {
Review comment:
we can disable by default and let interested users enable on their end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-992071081
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* fe0c868afdbc57efd8628c7380da7469e5108476 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325)
* e314a3c3cbe9a90b4d5f72d2b46a157985288ea1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-992091282
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* e314a3c3cbe9a90b4d5f72d2b46a157985288ea1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954363971
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* 7b6b7630351b60fb75c1e4b1767975636db42932 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-983260291
> cos(Cloud Object Storage) need this patch help to retry https://cloud.tencent.com/product/cos
Thanks for @mincwang 's info, It looks like https://intl.cloud.tencent.com/?lang=en&pg= (or some other dfs ) needs hoodie to take care of retry logic. So would we please going on this patch and get it done.
What's your opinion? @nsivabalan and @vinothchandar Looking forward to your reply :>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000116943
> I am wondering if we can guard this behavior for each storage scheme.
Hi @vinothchandar Thanks a lot for your attention. Let me explain the design of this retry wrapper.
As this picture shows, we design a new wrapper named hoodieRetryWrapper to wrap common file system which will retry specific common filesystem operation.
Also we let HoodieWrapperFileSystem hold this RetryWrapperFileSystem as common filesystem so that there's no damaged for metrics collection etc.
In summary, hoodieRetryWrapper can wrap and guard all kinds of storage scheme as long as let hoodieRetryWrapper hold this storage scheme client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-961589864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* b100a200602217c196c37e131a4768c5c8d97727 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-956857417
@hudi-bot azure run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] mincwang commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
mincwang commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r744206294
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java
##########
@@ -63,9 +63,14 @@ protected HoodieSparkTable(HoodieWriteConfig config, HoodieEngineContext context
public static <T extends HoodieRecordPayload> HoodieSparkTable<T> create(HoodieWriteConfig config, HoodieEngineContext context,
boolean refreshTimeline) {
HoodieTableMetaClient metaClient =
- HoodieTableMetaClient.builder().setConf(context.getHadoopConf().get()).setBasePath(config.getBasePath())
- .setLoadActiveTimelineOnLoad(true).setConsistencyGuardConfig(config.getConsistencyGuardConfig())
- .setLayoutVersion(Option.of(new TimelineLayoutVersion(config.getTimelineLayoutVersion()))).build();
+ HoodieTableMetaClient.builder()
+ .setConf(context.getHadoopConf().get())
+ .setBasePath(config.getBasePath())
+ .setLoadActiveTimelineOnLoad(true)
+ .setConsistencyGuardConfig(config.getConsistencyGuardConfig())
+ .setFileSystemGuardConfig(config.getFileSystemGuardConfig())
Review comment:
hi @zhangyue19921010 ，i think also require add about `setFileSystemGuardConfig` in `HoodieFlinkTable` calss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1033534245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833",
"triggerID" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 04f93a2e6825a202642e061c831a06b4d03ccf36 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697)
* 1139ead2daf93239741fa6e25b1dd66381fb5c9c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954363971
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 99a04ae02f6da7bed79e0db8bcce087ff6a6c108 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954363971
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 99a04ae02f6da7bed79e0db8bcce087ff6a6c108 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945)
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-966804166
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* b100a200602217c196c37e131a4768c5c8d97727 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954)
* fe0c868afdbc57efd8628c7380da7469e5108476 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-992091282
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* e314a3c3cbe9a90b4d5f72d2b46a157985288ea1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000124692
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* 1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696)
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 04f93a2e6825a202642e061c831a06b4d03ccf36 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000105066
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* 1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688)
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000057583
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* 7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617)
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* 1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000055049
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* 7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617)
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-998396355
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* e314a3c3cbe9a90b4d5f72d2b46a157985288ea1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222)
* 7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-998434340
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* 7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r801239830
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/RetryHelper.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Random;
+
+public class RetryHelper<T> {
+ private static final Logger LOG = LogManager.getLogger(RetryHelper.class);
+ private CheckedFunction<T> func;
+ private int num;
+ private long maxIntervalTime;
+ private long initialIntervalTime = 100L;
+ private String taskInfo = "N/A";
+
+ public RetryHelper() {
+ }
+
+ public RetryHelper(String taskInfo) {
+ this.taskInfo = taskInfo;
+ }
+
+ public RetryHelper tryWith(CheckedFunction<T> func) {
+ this.func = func;
+ return this;
+ }
+
+ public RetryHelper tryNum(int num) {
+ this.num = num;
+ return this;
+ }
+
+ public RetryHelper tryTaskInfo(String taskInfo) {
+ this.taskInfo = taskInfo;
+ return this;
+ }
+
+ public RetryHelper tryMaxInterval(long time) {
+ maxIntervalTime = time;
+ return this;
+ }
+
+ public RetryHelper tryInitialInterval(long time) {
+ initialIntervalTime = time;
+ return this;
+ }
+
+ public T start() throws IOException {
+ int retries = 0;
+ T functionResult = null;
+
+ while (true) {
+ long waitTime = Math.min(getWaitTimeExp(retries), maxIntervalTime);
+ try {
+ functionResult = func.get();
+ break;
+ } catch (IOException | RuntimeException e) {
+ if (retries++ >= num) {
Review comment:
Generally i think catch any `IOException` or `RuntimeException` is overprotection and irrational.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1033599122
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833",
"triggerID" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 1139ead2daf93239741fa6e25b1dd66381fb5c9c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r810745537
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FileSystemRetryConfig.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * The file system retry relevant config options.
+ */
+@ConfigClassProperty(name = "FileSystem Guard Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "The filesystem retry related config options, to help deal with runtime exception like list/get/put/delete performance issues.")
+public class FileSystemRetryConfig extends HoodieConfig {
+
+ public static final ConfigProperty<String> FILESYSTEM_RETRY_ENABLE = ConfigProperty
+ .key("hoodie.filesystem.operation.retry.enable")
+ .defaultValue("false")
+ .sinceVersion("0.11.0")
+ .withDocumentation("Enabled to handle list/get/delete etc file system performance issue.");
+
+ public static final ConfigProperty<Long> INITIAL_RETRY_INTERVAL_MS = ConfigProperty
+ .key("hoodie.filesystem.operation.retry.initial_interval_ms")
+ .defaultValue(100L)
+ .sinceVersion("0.11.0")
+ .withDocumentation("Amount of time (in ms) to wait, before retry to do operations on storage.");
+
+ public static final ConfigProperty<Long> MAX_RETRY_INTERVAL_MS = ConfigProperty
+ .key("hoodie.filesystem.operation.retry.max_interval_ms")
+ .defaultValue(2000L)
+ .sinceVersion("0.11.0")
+ .withDocumentation("Maximum amount of time (in ms), to wait for next retry.");
+
+ public static final ConfigProperty<Integer> MAX_RETRY_NUMBERS = ConfigProperty
+ .key("hoodie.filesystem.operation.retry.max_numbers")
Review comment:
max_times seems better ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r810679685
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FileSystemRetryConfig.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * The file system retry relevant config options.
+ */
+@ConfigClassProperty(name = "FileSystem Guard Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "The filesystem retry related config options, to help deal with runtime exception like list/get/put/delete performance issues.")
+public class FileSystemRetryConfig extends HoodieConfig {
+
+ public static final ConfigProperty<String> FILESYSTEM_RETRY_ENABLE = ConfigProperty
+ .key("hoodie.filesystem.operation.retry.enable")
+ .defaultValue("false")
+ .sinceVersion("0.11.0")
+ .withDocumentation("Enabled to handle list/get/delete etc file system performance issue.");
+
+ public static final ConfigProperty<Long> INITIAL_RETRY_INTERVAL_MS = ConfigProperty
+ .key("hoodie.filesystem.operation.retry.initial_interval_ms")
+ .defaultValue(100L)
+ .sinceVersion("0.11.0")
+ .withDocumentation("Amount of time (in ms) to wait, before retry to do operations on storage.");
+
+ public static final ConfigProperty<Long> MAX_RETRY_INTERVAL_MS = ConfigProperty
+ .key("hoodie.filesystem.operation.retry.max_interval_ms")
+ .defaultValue(2000L)
+ .sinceVersion("0.11.0")
+ .withDocumentation("Maximum amount of time (in ms), to wait for next retry.");
+
+ public static final ConfigProperty<Integer> MAX_RETRY_NUMBERS = ConfigProperty
+ .key("hoodie.filesystem.operation.retry.max_numbers")
Review comment:
may be we can name this as "hoodie.filesystem.operation.retry.max.count"
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/RetryHelper.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.stream.Collectors;
+
+public class RetryHelper<T> {
+ private static final Logger LOG = LogManager.getLogger(RetryHelper.class);
+ private CheckedFunction<T> func;
+ private int num;
+ private long maxIntervalTime;
Review comment:
can come of these be final ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/RetryHelper.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.stream.Collectors;
+
+public class RetryHelper<T> {
Review comment:
java docs
##########
File path: hudi-common/src/test/java/org/apache/hudi/common/fs/TestFSUtilsWithRetryWrapperEnable.java
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version loop.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-loop.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.util.Progressable;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+/**
+ * Tests file system utils with retry wrapper enable.
+ * P.S extends TestFSUtils and setUp a HoodieWrapperFileSystem for metaClient which can test all the TestFSUtils uts with RetryWrapperEnable
+ */
+public class TestFSUtilsWithRetryWrapperEnable extends TestFSUtils {
+
+ private static final String EXCEPTION_MESSAGE = "Fake runtime exception here.";
+ private long maxRetryIntervalMs;
+ private int maxRetryNumbers;
+ private long initialRetryIntervalMs;
+
+ @Override
+ @BeforeEach
+ public void setUp() throws IOException {
+ initMetaClient();
+ basePath = "file:" + basePath;
+ FileSystemRetryConfig fileSystemRetryConfig = FileSystemRetryConfig.newBuilder().withFileSystemActionRetryEnabled(true).build();
+ maxRetryIntervalMs = fileSystemRetryConfig.getMaxRetryIntervalMs();
+ maxRetryNumbers = fileSystemRetryConfig.getMaxRetryNumbers();
+ initialRetryIntervalMs = fileSystemRetryConfig.getInitialRetryIntervalMs();
+
+ FakeRemoteFileSystem fakeFs = new FakeRemoteFileSystem(FSUtils.getFs(metaClient.getMetaPath(), metaClient.getHadoopConf()), 2);
+ FileSystem fileSystem = new HoodieRetryWrapperFileSystem(fakeFs, maxRetryIntervalMs, maxRetryNumbers, initialRetryIntervalMs, "");
+
+ HoodieWrapperFileSystem fs = new HoodieWrapperFileSystem(fileSystem, new NoOpConsistencyGuard());
+ metaClient.setFs(fs);
+ }
+
+ // Test the scenario that fs keeps retrying until it fails.
+ @Test
+ public void testProcessFilesWithExceptions() throws Exception {
Review comment:
Can we add one more test to cover this scenario
- to assert that call succeeds after 2(basically before hitting max retries) retries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-976185571
Hi @vinothchandar Thanks for your attention.
Actually S3 java SDK take care of these retry logic through config `fs.s3a.attempts.maximum`
The necessary to build a retry layer on hudi is to let hudi can deal with different kinds of filesystem not only S3.
After careful thinking maybe it's not urgent for this patch and will close it.
Maybe we can reopen this patch if there's another fs need to let hudi take care of retring.
Anyway, really appreciate for @nsivabalan and @vinothchandar 's attention. And **sorry to bother you guys**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] mincwang commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
mincwang commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-981339983
cos(Cloud Object Storage) need this patch help to retry https://cloud.tencent.com/product/cos
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000116943
> I am wondering if we can guard this behavior for each storage scheme.
Hi @vinothchandar Thanks a lot for your attention. Let me explain the framework of this retry wrapper.
As this picture shows, we design a new wrapper named hoodieRetryWrapper to wrap common file system which will retry specific common filesystem operation.
Also we let HoodieWrapperFileSystem hold this RetryWrapperFileSystem just like hold a common filesystem so that there's no damaged for metrics collection etc.
In summary, hoodieRetryWrapper can wrap and guard all kinds of storage scheme as long as let hoodieRetryWrapper hold this storage scheme client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000072854
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* 1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r774394645
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -253,7 +265,16 @@ public TimelineLayoutVersion getTimelineLayoutVersion() {
*/
public HoodieWrapperFileSystem getFs() {
if (fs == null) {
- FileSystem fileSystem = FSUtils.getFs(metaPath, hadoopConf.newCopy());
+ FileSystem fileSystem;
+ if (fileSystemRetryConfig.isFileSystemActionRetryEnable()) {
Review comment:
Emmm..., it's easy to make this as a property of the `StorageScheme` something like `FILE("file", append: false, retry: true)`
What I am more worried about is that this may mean that the retry function is turned on by default. And it may be hard for users to disable this retry if they don't need it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1029982604
@zhangyue19921010 : lets not wait for Vinoth. he has been busy recently. Lets go ahead for now.
can we do this.
add a property named supportsRetry or something to each StorageScheme. So, only for those storage schemes where retries are supported, we can check the config (fileSystemRetryConfig.isFileSystemActionRetryEnable()) and then invoke our retry Helper. for eg, in S3 this can be disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1036952661
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833",
"triggerID" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"triggerType" : "PUSH"
}, {
"hash" : "c83f659752b5aa3f078b3135ad45f14596e6ea04",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c83f659752b5aa3f078b3135ad45f14596e6ea04",
"triggerType" : "PUSH"
}, {
"hash" : "9d1366b79213a74fe3eb8b70ec49fa50e31b168c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9d1366b79213a74fe3eb8b70ec49fa50e31b168c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 1139ead2daf93239741fa6e25b1dd66381fb5c9c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833)
* c83f659752b5aa3f078b3135ad45f14596e6ea04 UNKNOWN
* 9d1366b79213a74fe3eb8b70ec49fa50e31b168c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-961589864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* b100a200602217c196c37e131a4768c5c8d97727 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r767386141
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/RetryHelper.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Random;
+
+public class RetryHelper<T> {
+ private static final Logger LOG = LogManager.getLogger(RetryHelper.class);
+ private HoodieWrapperFileSystem.CheckedFunction<T> func;
+ private int num;
+ private long maxIntervalTime;
+ private long initialIntervalTime = 100L;
+ private String taskInfo = "N/A";
+
+ public RetryHelper() {
+ }
+
+ public RetryHelper(String taskInfo) {
+ this.taskInfo = taskInfo;
+ }
+
+ public RetryHelper tryWith(HoodieWrapperFileSystem.CheckedFunction<T> func) {
+ this.func = func;
+ return this;
+ }
+
+ public RetryHelper tryNum(int num) {
+ this.num = num;
+ return this;
+ }
+
+ public RetryHelper tryTaskInfo(String taskInfo) {
+ this.taskInfo = taskInfo;
+ return this;
+ }
+
+ public RetryHelper tryMaxInterval(long time) {
+ maxIntervalTime = time;
+ return this;
+ }
+
+ public RetryHelper tryInitialInterval(long time) {
+ initialIntervalTime = time;
+ return this;
+ }
+
+ public T start() throws IOException {
+ int retries = 0;
+ boolean success = false;
+ RuntimeException exception = null;
+ T t = null;
+ do {
+ long waitTime = Math.min(getWaitTimeExp(retries), maxIntervalTime);
+ try {
+ t = func.get();
+ success = true;
+ break;
+ } catch (RuntimeException e) {
+ // deal with RuntimeExceptions such like AmazonS3Exception 503
+ exception = e;
+ LOG.warn("Catch RuntimeException " + taskInfo + ", will retry after " + waitTime + " ms.", e);
+ try {
+ Thread.sleep(waitTime);
+ } catch (InterruptedException ex) {
+ // ignore InterruptedException here
+ }
+ retries++;
+ }
+ } while (retries <= num);
Review comment:
emmm, we only do `++` when caught exception, so maybe can't move it out of `catch() {}` block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r767385839
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/RetryHelper.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Random;
+
+public class RetryHelper<T> {
+ private static final Logger LOG = LogManager.getLogger(RetryHelper.class);
+ private HoodieWrapperFileSystem.CheckedFunction<T> func;
+ private int num;
+ private long maxIntervalTime;
+ private long initialIntervalTime = 100L;
+ private String taskInfo = "N/A";
+
+ public RetryHelper() {
+ }
+
+ public RetryHelper(String taskInfo) {
+ this.taskInfo = taskInfo;
+ }
+
+ public RetryHelper tryWith(HoodieWrapperFileSystem.CheckedFunction<T> func) {
+ this.func = func;
+ return this;
+ }
+
+ public RetryHelper tryNum(int num) {
+ this.num = num;
+ return this;
+ }
+
+ public RetryHelper tryTaskInfo(String taskInfo) {
+ this.taskInfo = taskInfo;
+ return this;
+ }
+
+ public RetryHelper tryMaxInterval(long time) {
+ maxIntervalTime = time;
+ return this;
+ }
+
+ public RetryHelper tryInitialInterval(long time) {
+ initialIntervalTime = time;
+ return this;
+ }
+
+ public T start() throws IOException {
+ int retries = 0;
+ boolean success = false;
+ RuntimeException exception = null;
+ T t = null;
Review comment:
Sure thing, changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r767384153
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FileSystemGuardConfig.java
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * The consistency guard relevant config options.
+ */
+@ConfigClassProperty(name = "FileSystem Guard Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "The filesystem guard related config options, to help deal with runtime exception like s3 list/get/put/delete performance issues.")
+public class FileSystemGuardConfig extends HoodieConfig {
+
+ public static final ConfigProperty<String> FILESYSTEM_RETRY_ENABLE = ConfigProperty
+ .key("hoodie.filesystem.action.retry.enabled")
Review comment:
Sure, changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1036969317
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833",
"triggerID" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"triggerType" : "PUSH"
}, {
"hash" : "c83f659752b5aa3f078b3135ad45f14596e6ea04",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c83f659752b5aa3f078b3135ad45f14596e6ea04",
"triggerType" : "PUSH"
}, {
"hash" : "9d1366b79213a74fe3eb8b70ec49fa50e31b168c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5937",
"triggerID" : "9d1366b79213a74fe3eb8b70ec49fa50e31b168c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 1139ead2daf93239741fa6e25b1dd66381fb5c9c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833)
* c83f659752b5aa3f078b3135ad45f14596e6ea04 UNKNOWN
* 9d1366b79213a74fe3eb8b70ec49fa50e31b168c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5937)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1033479415
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 04f93a2e6825a202642e061c831a06b4d03ccf36 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697)
* 1139ead2daf93239741fa6e25b1dd66381fb5c9c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1033694266
Hi @nsivabalan and @danny0405 Thanks a lot for you review.
Now we add a property named supportsRetry each StorageScheme. And check it before retry.
Also and a new config named `hoodie.filesystem.operation.retry.exceptions` to give users an option to decide which Exceptions that needs to be re-tryed. It could solve the overprotection problem.
PTAL :) Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r810679925
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/RetryHelper.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.stream.Collectors;
+
+public class RetryHelper<T> {
+ private static final Logger LOG = LogManager.getLogger(RetryHelper.class);
+ private CheckedFunction<T> func;
+ private int num;
+ private long maxIntervalTime;
Review comment:
can some of these be final ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r774391278
##########
File path: hudi-common/src/test/java/org/apache/hudi/common/fs/TestFSUtilsWithRetryWrapperEnable.java
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version loop.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-loop.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.util.Progressable;
+import org.apache.hudi.common.util.RetryHelper;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+/**
+ * Tests file system utils with retry wrapper enable.
+ */
+public class TestFSUtilsWithRetryWrapperEnable extends TestFSUtils {
+
+ public RetryHelper retryHelper;
+ public static final String EXCEPTION_MESSAGE = "Fake runtime exception here.";
+
+ @Override
+ @BeforeEach
+ public void setUp() throws IOException {
+ initMetaClient();
+ FileSystemRetryConfig fileSystemRetryConfig = FileSystemRetryConfig.newBuilder().withFileSystemActionRetryEnabled(true).build();
+ retryHelper = new RetryHelper<>()
+ .tryMaxInterval(fileSystemRetryConfig.getMaxRetryIntervalMs())
+ .tryNum(fileSystemRetryConfig.getMaxRetryNumbers())
+ .tryInitialInterval(fileSystemRetryConfig.getInitialRetryIntervalMs());
+ FakeRemoteFileSystem fakeFs = new FakeRemoteFileSystem(FSUtils.getFs(metaClient.getMetaPath(), metaClient.getHadoopConf()), 1);
+ FileSystem fileSystem = new HoodieRetryWrapperFileSystem(fakeFs, retryHelper);
+ HoodieWrapperFileSystem fs = new HoodieWrapperFileSystem(fileSystem, new NoOpConsistencyGuard());
+ metaClient.setFs(fs);
+ }
+
+ @Test
+ public void testProcessFilesWithExceptions() throws Exception {
+ FakeRemoteFileSystem fakeFs = new FakeRemoteFileSystem(FSUtils.getFs(metaClient.getMetaPath(), metaClient.getHadoopConf()), 100);
+ FileSystem fileSystem = new HoodieRetryWrapperFileSystem(fakeFs, retryHelper);
+ HoodieWrapperFileSystem fs = new HoodieWrapperFileSystem(fileSystem, new NoOpConsistencyGuard());
+ metaClient.setFs(fs);
+ List<String> folders =
+ Arrays.asList("2016/04/15", ".hoodie/.temp/2/2016/04/15");
+ folders.forEach(f -> assertThrows(RuntimeException.class, () -> metaClient.getFs().mkdirs(new Path(new Path(basePath), f))));
Review comment:
oh, we code `public class TestFSUtilsWithRetryWrapperEnable extends TestFSUtils` just let TestFSUtilsWithRetryWrapperEnable run all the test from TestFSUtils using retryWrapped filesystem, overwriting `setUp() `
Also add a `testProcessFilesWithExceptions ` to test the scenario that retry until failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r774389446
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FileSystemRetryConfig.java
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * The file system retry relevant config options.
+ */
+@ConfigClassProperty(name = "FileSystem Guard Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "The filesystem retry related config options, to help deal with runtime exception like list/get/put/delete performance issues.")
+public class FileSystemRetryConfig extends HoodieConfig {
+
+ public static final ConfigProperty<String> FILESYSTEM_RETRY_ENABLE = ConfigProperty
+ .key("hoodie.filesystem.action.retry.enable")
+ .defaultValue("false")
+ .sinceVersion("0.10.0")
Review comment:
Sure, Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000116943
> I am wondering if we can guard this behavior for each storage scheme.
Hi @vinothchandar Thanks a lot for your attention. Let me explain the design of this retry wrapper.
As this picture shows, we design a new wrapper named hoodieRetryWrapper to wrap common file system which will retry specific common filesystem operation.
Also we let HoodieWrapperFileSystem hold this RetryWrapperFileSystem just like common filesystem so that there's no damaged for metrics collection etc.
In summary, hoodieRetryWrapper can wrap and guard all kinds of storage scheme as long as let hoodieRetryWrapper hold this storage scheme client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000055049
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* 7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617)
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000108859
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* 1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688)
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 04f93a2e6825a202642e061c831a06b4d03ccf36 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000072854
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* 1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r774391928
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/RetryHelper.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Random;
+
+public class RetryHelper<T> {
+ private static final Logger LOG = LogManager.getLogger(RetryHelper.class);
+ private HoodieWrapperFileSystem.CheckedFunction<T> func;
Review comment:
Sure, Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-966808987
looks like a good thing to have. have marked it as release blocker. will try to review it sooner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954363971
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* 7b6b7630351b60fb75c1e4b1767975636db42932 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951)
* b100a200602217c196c37e131a4768c5c8d97727 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-956857417
@hudi-bot azure run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954443545
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r790400915
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -253,7 +265,16 @@ public TimelineLayoutVersion getTimelineLayoutVersion() {
*/
public HoodieWrapperFileSystem getFs() {
if (fs == null) {
- FileSystem fileSystem = FSUtils.getFs(metaPath, hadoopConf.newCopy());
+ FileSystem fileSystem;
+ if (fileSystemRetryConfig.isFileSystemActionRetryEnable()) {
Review comment:
Guess, what vinoth refers to is, we should add a property named supportsRetry or something to each StorageScheme. So, only for those storage schemes where retries are supported, we can check the config (fileSystemRetryConfig.isFileSystemActionRetryEnable()) and then invoke our retry Helper.
@vinothchandar : can you please clarify on your above suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r790371241
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -253,7 +265,16 @@ public TimelineLayoutVersion getTimelineLayoutVersion() {
*/
public HoodieWrapperFileSystem getFs() {
if (fs == null) {
- FileSystem fileSystem = FSUtils.getFs(metaPath, hadoopConf.newCopy());
+ FileSystem fileSystem;
+ if (fileSystemRetryConfig.isFileSystemActionRetryEnable()) {
Review comment:
yeah, I am not sure if we can make it a property of StorageScheme. we should let users enable on a need basis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954363971
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 99a04ae02f6da7bed79e0db8bcce087ff6a6c108 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945)
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* 7b6b7630351b60fb75c1e4b1767975636db42932 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-962534306
hi @nsivabalan could you please help to review this PR at your convenience? Thanks a lot if you could give me a hand!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-966790719
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* b100a200602217c196c37e131a4768c5c8d97727 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954)
* fe0c868afdbc57efd8628c7380da7469e5108476 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-966790719
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* b100a200602217c196c37e131a4768c5c8d97727 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954)
* fe0c868afdbc57efd8628c7380da7469e5108476 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 closed pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 closed pull request #3887:
URL: https://github.com/apache/hudi/pull/3887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1033694266
Hi @nsivabalan and @danny0405 Thanks a lot for you review.
Now we add a property named supportsRetry each StorageScheme. And check it before retry.
Also and a new config named `hoodie.filesystem.operation.retry.exceptions` to give users an option to decide which Exceptions that needs to be re-tryed. It could solve the overprotection problem.
PTAL :) Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r804921690
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -253,7 +265,16 @@ public TimelineLayoutVersion getTimelineLayoutVersion() {
*/
public HoodieWrapperFileSystem getFs() {
if (fs == null) {
- FileSystem fileSystem = FSUtils.getFs(metaPath, hadoopConf.newCopy());
+ FileSystem fileSystem;
+ if (fileSystemRetryConfig.isFileSystemActionRetryEnable()) {
Review comment:
In that case, I think we can just have the config alone. and let the user decide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1030070769
> @zhangyue19921010 : lets not wait for Vinoth. he has been busy recently. Lets go ahead for now. can we do this. add a property named supportsRetry or something to each StorageScheme. So, only for those storage schemes where retries are supported, we can check the config (fileSystemRetryConfig.isFileSystemActionRetryEnable()) and then invoke our retry Helper. for eg, in S3 this can be disabled.
Sure thing. Will do it ASAP :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1032179002
> @zhangyue19921010 : lets not wait for Vinoth. he has been busy recently. Lets go ahead for now. can we do this. add a property named supportsRetry or something to each StorageScheme. So, only for those storage schemes where retries are supported, we can check the config (fileSystemRetryConfig.isFileSystemActionRetryEnable()) and then invoke our retry Helper. for eg, in S3 this can be disabled.
One thing we should be very careful with is in what exception/error we should do a retry here ? If it is a normal user triggered exception, should we still retry here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r772807071
##########
File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/HoodieFlinkTable.java
##########
@@ -51,9 +51,14 @@ protected HoodieFlinkTable(HoodieWriteConfig config, HoodieEngineContext context
public static <T extends HoodieRecordPayload> HoodieFlinkTable<T> create(HoodieWriteConfig config, HoodieFlinkEngineContext context) {
HoodieTableMetaClient metaClient =
- HoodieTableMetaClient.builder().setConf(context.getHadoopConf().get()).setBasePath(config.getBasePath())
- .setLoadActiveTimelineOnLoad(true).setConsistencyGuardConfig(config.getConsistencyGuardConfig())
- .setLayoutVersion(Option.of(new TimelineLayoutVersion(config.getTimelineLayoutVersion()))).build();
+ HoodieTableMetaClient.builder()
Review comment:
same.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FileSystemRetryConfig.java
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * The file system retry relevant config options.
+ */
+@ConfigClassProperty(name = "FileSystem Guard Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "The filesystem retry related config options, to help deal with runtime exception like list/get/put/delete performance issues.")
+public class FileSystemRetryConfig extends HoodieConfig {
+
+ public static final ConfigProperty<String> FILESYSTEM_RETRY_ENABLE = ConfigProperty
+ .key("hoodie.filesystem.action.retry.enable")
Review comment:
can we drop `action` from the property name, given action has a different meaning inside Hudi? May be `filesystem.operation`?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -253,7 +265,16 @@ public TimelineLayoutVersion getTimelineLayoutVersion() {
*/
public HoodieWrapperFileSystem getFs() {
if (fs == null) {
- FileSystem fileSystem = FSUtils.getFs(metaPath, hadoopConf.newCopy());
+ FileSystem fileSystem;
+ if (fileSystemRetryConfig.isFileSystemActionRetryEnable()) {
Review comment:
could we make this a property of the `StorageScheme`?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -138,8 +143,15 @@ private HoodieTableMetaClient(Configuration conf, String basePath, boolean loadA
public HoodieTableMetaClient() {}
public static HoodieTableMetaClient reload(HoodieTableMetaClient oldMetaClient) {
- return HoodieTableMetaClient.builder().setConf(oldMetaClient.hadoopConf.get()).setBasePath(oldMetaClient.basePath).setLoadActiveTimelineOnLoad(oldMetaClient.loadActiveTimelineOnLoad)
- .setConsistencyGuardConfig(oldMetaClient.consistencyGuardConfig).setLayoutVersion(Option.of(oldMetaClient.timelineLayoutVersion)).setPayloadClassName(null).build();
+ return HoodieTableMetaClient.builder()
Review comment:
can you please comment on these reformatted blocks to highlight what has actually changed
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieClient.java
##########
@@ -132,9 +132,14 @@ protected void initWrapperFSMetrics() {
}
protected HoodieTableMetaClient createMetaClient(boolean loadActiveTimelineOnLoad) {
- return HoodieTableMetaClient.builder().setConf(hadoopConf).setBasePath(config.getBasePath())
- .setLoadActiveTimelineOnLoad(loadActiveTimelineOnLoad).setConsistencyGuardConfig(config.getConsistencyGuardConfig())
- .setLayoutVersion(Option.of(new TimelineLayoutVersion(config.getTimelineLayoutVersion()))).build();
+ return HoodieTableMetaClient.builder()
Review comment:
could we please avoid non-essential style fixes in the PR. Makes it harder to review.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/RetryHelper.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Random;
+
+public class RetryHelper<T> {
+ private static final Logger LOG = LogManager.getLogger(RetryHelper.class);
+ private HoodieWrapperFileSystem.CheckedFunction<T> func;
Review comment:
Could we keep this generic? if we cannot avoid referencing any wrapper file system stuff here, then we need to move this back with the `fs` package?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-998416466
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* e314a3c3cbe9a90b4d5f72d2b46a157985288ea1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222)
* 7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000161779
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 04f93a2e6825a202642e061c831a06b4d03ccf36 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r774391778
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FileSystemRetryConfig.java
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * The file system retry relevant config options.
+ */
+@ConfigClassProperty(name = "FileSystem Guard Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "The filesystem retry related config options, to help deal with runtime exception like list/get/put/delete performance issues.")
+public class FileSystemRetryConfig extends HoodieConfig {
+
+ public static final ConfigProperty<String> FILESYSTEM_RETRY_ENABLE = ConfigProperty
+ .key("hoodie.filesystem.action.retry.enable")
Review comment:
Okay, Changed.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -138,8 +143,15 @@ private HoodieTableMetaClient(Configuration conf, String basePath, boolean loadA
public HoodieTableMetaClient() {}
public static HoodieTableMetaClient reload(HoodieTableMetaClient oldMetaClient) {
- return HoodieTableMetaClient.builder().setConf(oldMetaClient.hadoopConf.get()).setBasePath(oldMetaClient.basePath).setLoadActiveTimelineOnLoad(oldMetaClient.loadActiveTimelineOnLoad)
- .setConsistencyGuardConfig(oldMetaClient.consistencyGuardConfig).setLayoutVersion(Option.of(oldMetaClient.timelineLayoutVersion)).setPayloadClassName(null).build();
+ return HoodieTableMetaClient.builder()
Review comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000116943
> I am wondering if we can guard this behavior for each storage scheme.
Hi @vinothchandar Thanks a lot for your attention. Let me explain the design of this retry wrapper.
As this picture shows, we design a new wrapper named hoodieRetryWrapper to wrap common file system which will retry specific common filesystem operation.
Also we let HoodieWrapperFileSystem hold this RetryWrapperFileSystem just like hold a common filesystem so that there's no damaged for metrics collection etc.
In summary, hoodieRetryWrapper can wrap and guard all kinds of storage scheme as long as let hoodieRetryWrapper hold this storage scheme client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1033479415
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 04f93a2e6825a202642e061c831a06b4d03ccf36 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697)
* 1139ead2daf93239741fa6e25b1dd66381fb5c9c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1036952661
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833",
"triggerID" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"triggerType" : "PUSH"
}, {
"hash" : "c83f659752b5aa3f078b3135ad45f14596e6ea04",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c83f659752b5aa3f078b3135ad45f14596e6ea04",
"triggerType" : "PUSH"
}, {
"hash" : "9d1366b79213a74fe3eb8b70ec49fa50e31b168c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9d1366b79213a74fe3eb8b70ec49fa50e31b168c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 1139ead2daf93239741fa6e25b1dd66381fb5c9c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833)
* c83f659752b5aa3f078b3135ad45f14596e6ea04 UNKNOWN
* 9d1366b79213a74fe3eb8b70ec49fa50e31b168c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1036969317
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833",
"triggerID" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"triggerType" : "PUSH"
}, {
"hash" : "c83f659752b5aa3f078b3135ad45f14596e6ea04",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c83f659752b5aa3f078b3135ad45f14596e6ea04",
"triggerType" : "PUSH"
}, {
"hash" : "9d1366b79213a74fe3eb8b70ec49fa50e31b168c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5937",
"triggerID" : "9d1366b79213a74fe3eb8b70ec49fa50e31b168c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 1139ead2daf93239741fa6e25b1dd66381fb5c9c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833)
* c83f659752b5aa3f078b3135ad45f14596e6ea04 UNKNOWN
* 9d1366b79213a74fe3eb8b70ec49fa50e31b168c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5937)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r801239830
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/RetryHelper.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Random;
+
+public class RetryHelper<T> {
+ private static final Logger LOG = LogManager.getLogger(RetryHelper.class);
+ private CheckedFunction<T> func;
+ private int num;
+ private long maxIntervalTime;
+ private long initialIntervalTime = 100L;
+ private String taskInfo = "N/A";
+
+ public RetryHelper() {
+ }
+
+ public RetryHelper(String taskInfo) {
+ this.taskInfo = taskInfo;
+ }
+
+ public RetryHelper tryWith(CheckedFunction<T> func) {
+ this.func = func;
+ return this;
+ }
+
+ public RetryHelper tryNum(int num) {
+ this.num = num;
+ return this;
+ }
+
+ public RetryHelper tryTaskInfo(String taskInfo) {
+ this.taskInfo = taskInfo;
+ return this;
+ }
+
+ public RetryHelper tryMaxInterval(long time) {
+ maxIntervalTime = time;
+ return this;
+ }
+
+ public RetryHelper tryInitialInterval(long time) {
+ initialIntervalTime = time;
+ return this;
+ }
+
+ public T start() throws IOException {
+ int retries = 0;
+ T functionResult = null;
+
+ while (true) {
+ long waitTime = Math.min(getWaitTimeExp(retries), maxIntervalTime);
+ try {
+ functionResult = func.get();
+ break;
+ } catch (IOException | RuntimeException e) {
+ if (retries++ >= num) {
Review comment:
Generally i think catching any `IOException` or `RuntimeException` is overprotection and irrational.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1046401403
> LGTM. few minor comments. I will go ahead and land the patch. Please take it as a follow up patch.
> thanks for the contribution!
Thanks a lot for your review and merge. Sure I will do it ASAP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-966822668
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* fe0c868afdbc57efd8628c7380da7469e5108476 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-990963440
> sure, makes sense if there are other cloud stores that needs this retry. Can you please address the feedback given already.
Sure, I will address the comments ASAP. Thanks a lot for your attention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-966822668
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* fe0c868afdbc57efd8628c7380da7469e5108476 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-992071081
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* fe0c868afdbc57efd8628c7380da7469e5108476 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325)
* e314a3c3cbe9a90b4d5f72d2b46a157985288ea1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-998396355
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* e314a3c3cbe9a90b4d5f72d2b46a157985288ea1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222)
* 7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r772698180
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FileSystemRetryConfig.java
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * The file system retry relevant config options.
+ */
+@ConfigClassProperty(name = "FileSystem Guard Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "The filesystem retry related config options, to help deal with runtime exception like list/get/put/delete performance issues.")
+public class FileSystemRetryConfig extends HoodieConfig {
+
+ public static final ConfigProperty<String> FILESYSTEM_RETRY_ENABLE = ConfigProperty
+ .key("hoodie.filesystem.action.retry.enable")
+ .defaultValue("false")
+ .sinceVersion("0.10.0")
Review comment:
we need to fix this to 0.11.0
##########
File path: hudi-common/src/test/java/org/apache/hudi/common/fs/TestFSUtilsWithRetryWrapperEnable.java
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version loop.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-loop.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.util.Progressable;
+import org.apache.hudi.common.util.RetryHelper;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+/**
+ * Tests file system utils with retry wrapper enable.
+ */
+public class TestFSUtilsWithRetryWrapperEnable extends TestFSUtils {
+
+ public RetryHelper retryHelper;
+ public static final String EXCEPTION_MESSAGE = "Fake runtime exception here.";
+
+ @Override
+ @BeforeEach
+ public void setUp() throws IOException {
+ initMetaClient();
+ FileSystemRetryConfig fileSystemRetryConfig = FileSystemRetryConfig.newBuilder().withFileSystemActionRetryEnabled(true).build();
+ retryHelper = new RetryHelper<>()
+ .tryMaxInterval(fileSystemRetryConfig.getMaxRetryIntervalMs())
+ .tryNum(fileSystemRetryConfig.getMaxRetryNumbers())
+ .tryInitialInterval(fileSystemRetryConfig.getInitialRetryIntervalMs());
+ FakeRemoteFileSystem fakeFs = new FakeRemoteFileSystem(FSUtils.getFs(metaClient.getMetaPath(), metaClient.getHadoopConf()), 1);
+ FileSystem fileSystem = new HoodieRetryWrapperFileSystem(fakeFs, retryHelper);
+ HoodieWrapperFileSystem fs = new HoodieWrapperFileSystem(fileSystem, new NoOpConsistencyGuard());
+ metaClient.setFs(fs);
+ }
+
+ @Test
+ public void testProcessFilesWithExceptions() throws Exception {
+ FakeRemoteFileSystem fakeFs = new FakeRemoteFileSystem(FSUtils.getFs(metaClient.getMetaPath(), metaClient.getHadoopConf()), 100);
+ FileSystem fileSystem = new HoodieRetryWrapperFileSystem(fakeFs, retryHelper);
+ HoodieWrapperFileSystem fs = new HoodieWrapperFileSystem(fileSystem, new NoOpConsistencyGuard());
+ metaClient.setFs(fs);
+ List<String> folders =
+ Arrays.asList("2016/04/15", ".hoodie/.temp/2/2016/04/15");
+ folders.forEach(f -> assertThrows(RuntimeException.class, () -> metaClient.getFs().mkdirs(new Path(new Path(basePath), f))));
Review comment:
am I missing something here. I expected we will assert that the call succeeds after retries. Do we have tests to test the retry wrapper?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954363971
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 99a04ae02f6da7bed79e0db8bcce087ff6a6c108 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945)
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* 7b6b7630351b60fb75c1e4b1767975636db42932 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000057583
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* 7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617)
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* 1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] mincwang removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
mincwang removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-962556117
hi @zhangyue19921010 ，i think also require add about `setFileSystemGuardConfig` in `HoodieFlinkTable` calss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #3887:
URL: https://github.com/apache/hudi/pull/3887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-992064349
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* fe0c868afdbc57efd8628c7380da7469e5108476 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325)
* e314a3c3cbe9a90b4d5f72d2b46a157985288ea1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r767385794
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FileSystemGuardConfig.java
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * The consistency guard relevant config options.
+ */
+@ConfigClassProperty(name = "FileSystem Guard Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "The filesystem guard related config options, to help deal with runtime exception like s3 list/get/put/delete performance issues.")
+public class FileSystemGuardConfig extends HoodieConfig {
+
+ public static final ConfigProperty<String> FILESYSTEM_RETRY_ENABLE = ConfigProperty
+ .key("hoodie.filesystem.action.retry.enabled")
+ .defaultValue("false")
+ .sinceVersion("0.10.0")
+ .withDocumentation("Enabled to handle S3 list/get/delete etc file system performance issue.");
+
+ public static final ConfigProperty<Long> INITIAL_RETRY_INTERVAL_MS = ConfigProperty
+ .key("hoodie.filesystem.action.retry.initial_interval_ms")
+ .defaultValue(100L)
+ .sinceVersion("0.10.0")
+ .withDocumentation("Amount of time (in ms) to wait, before retry to do operations on storage.");
+
+ public static final ConfigProperty<Long> MAX_RETRY_INTERVAL_MS = ConfigProperty
+ .key("hoodie.filesystem.action.retry.max_interval_ms")
+ .defaultValue(2000L)
+ .sinceVersion("0.10.0")
+ .withDocumentation("Maximum amount of time (in ms), to wait for next retry.");
+
+ public static final ConfigProperty<Integer> MAX_RETRY_NUMBERS = ConfigProperty
+ .key("hoodie.filesystem.action.retry.max_numbers")
Review comment:
We use `(long) Math.pow(2, retryCount) * initialIntervalTime + random.nextInt(100);` to calculate sleep time before each retry. And we may need `MAX_RETRY_INTERVAL_MS` to control the maximum duration of a single sleep in case sleep too long`Math.min(getWaitTimeExp(retries), maxIntervalTime)`.
Also use `MAX_RETRY_NUMBERS` to control max retry numbers to limit total retry time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#discussion_r752872257
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java
##########
@@ -63,9 +63,14 @@ protected HoodieSparkTable(HoodieWriteConfig config, HoodieEngineContext context
public static <T extends HoodieRecordPayload> HoodieSparkTable<T> create(HoodieWriteConfig config, HoodieEngineContext context,
boolean refreshTimeline) {
HoodieTableMetaClient metaClient =
- HoodieTableMetaClient.builder().setConf(context.getHadoopConf().get()).setBasePath(config.getBasePath())
- .setLoadActiveTimelineOnLoad(true).setConsistencyGuardConfig(config.getConsistencyGuardConfig())
- .setLayoutVersion(Option.of(new TimelineLayoutVersion(config.getTimelineLayoutVersion()))).build();
+ HoodieTableMetaClient.builder()
+ .setConf(context.getHadoopConf().get())
+ .setBasePath(config.getBasePath())
+ .setLoadActiveTimelineOnLoad(true)
+ .setConsistencyGuardConfig(config.getConsistencyGuardConfig())
+ .setFileSystemGuardConfig(config.getFileSystemGuardConfig())
Review comment:
cc @danny0405 :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000127666
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* 1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696)
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 04f93a2e6825a202642e061c831a06b4d03ccf36 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000108859
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* 1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688)
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 04f93a2e6825a202642e061c831a06b4d03ccf36 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1000123975
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-1033534245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
}, {
"hash" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3325",
"triggerID" : "fe0c868afdbc57efd8628c7380da7469e5108476",
"triggerType" : "PUSH"
}, {
"hash" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4222",
"triggerID" : "e314a3c3cbe9a90b4d5f72d2b46a157985288ea1",
"triggerType" : "PUSH"
}, {
"hash" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4617",
"triggerID" : "7f7fc4b33d1602d891c9f4b370d9e9d3cab8363f",
"triggerType" : "PUSH"
}, {
"hash" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4a52761592b8be6b6d67ae2339aad19705a0e5c",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4688",
"triggerID" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"triggerType" : "PUSH"
}, {
"hash" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae8b009239445f17a9f2447aa9c430c74c8d6c56",
"triggerType" : "PUSH"
}, {
"hash" : "1f5392a205d4606d9e2a31ac9cd57f0f1c392aa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4696",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"triggerType" : "PUSH"
}, {
"hash" : "04f93a2e6825a202642e061c831a06b4d03ccf36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697",
"triggerID" : "1000123975",
"triggerType" : "MANUAL"
}, {
"hash" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833",
"triggerID" : "1139ead2daf93239741fa6e25b1dd66381fb5c9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* f4a52761592b8be6b6d67ae2339aad19705a0e5c UNKNOWN
* ae8b009239445f17a9f2447aa9c430c74c8d6c56 UNKNOWN
* 04f93a2e6825a202642e061c831a06b4d03ccf36 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4697)
* 1139ead2daf93239741fa6e25b1dd66381fb5c9c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5833)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954363971
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* 7b6b7630351b60fb75c1e4b1767975636db42932 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954363971
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* 7b6b7630351b60fb75c1e4b1767975636db42932 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951)
* b100a200602217c196c37e131a4768c5c8d97727 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954363971
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 99a04ae02f6da7bed79e0db8bcce087ff6a6c108 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954443545
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954363971
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* b100a200602217c196c37e131a4768c5c8d97727 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-961589864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2945",
"triggerID" : "99a04ae02f6da7bed79e0db8bcce087ff6a6c108",
"triggerType" : "PUSH"
}, {
"hash" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ec7c1e3c40af686b9a4dcc5af99ebd3671913d",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2948",
"triggerID" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"triggerType" : "PUSH"
}, {
"hash" : "7b6b7630351b60fb75c1e4b1767975636db42932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2951",
"triggerID" : "954443545",
"triggerType" : "MANUAL"
}, {
"hash" : "b100a200602217c196c37e131a4768c5c8d97727",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954",
"triggerID" : "b100a200602217c196c37e131a4768c5c8d97727",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82ec7c1e3c40af686b9a4dcc5af99ebd3671913d UNKNOWN
* b100a200602217c196c37e131a4768c5c8d97727 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2954)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3887: [HUDI-2648] Retry FileSystem action instead of failed directly.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3887:
URL: https://github.com/apache/hudi/pull/3887#issuecomment-954363971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org