You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "TengHuo (via GitHub)" <gi...@apache.org> on 2023/02/03 08:52:10 UTC
[GitHub] [hudi] TengHuo opened a new pull request, #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
TengHuo opened a new pull request, #7840:
URL: https://github.com/apache/hudi/pull/7840
### Change Logs
Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap. It will cause off heap memory leak if not close properly.
### Impact
No impact
### Risk level (write none, low medium or high below)
Low
### Documentation Update
N/A
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1418479372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896",
"triggerID" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"triggerType" : "PUSH"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943",
"triggerID" : "1416916044",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 8a1762e065e18c40ff4ed1b519056647d2640121 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on a diff in pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on code in PR #7840:
URL: https://github.com/apache/hudi/pull/7840#discussion_r1112550248
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/BitCaskDiskMap.java:
##########
@@ -452,6 +452,8 @@ private byte[] decompressBytes(final byte[] bytes) throws IOException {
return decompressBaos.toByteArray();
} catch (IOException e) {
throw new HoodieIOException("IOException while decompressing bytes", e);
+ } finally {
Review Comment:
Sorry for the late replay, quite busy recently.
Got it, thanks for reviewing. Let me add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1437912398
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896",
"triggerID" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"triggerType" : "PUSH"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943",
"triggerID" : "1416916044",
"triggerType" : "MANUAL"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14961",
"triggerID" : "1418831613",
"triggerType" : "MANUAL"
}, {
"hash" : "5330d2b92b913cb02ebae8f638390f2d28448523",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15307",
"triggerID" : "5330d2b92b913cb02ebae8f638390f2d28448523",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a1762e065e18c40ff4ed1b519056647d2640121 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14961)
* 5330d2b92b913cb02ebae8f638390f2d28448523 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15307)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1418837058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896",
"triggerID" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"triggerType" : "PUSH"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943",
"triggerID" : "1416916044",
"triggerType" : "MANUAL"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14961",
"triggerID" : "1418831613",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 8a1762e065e18c40ff4ed1b519056647d2640121 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14961)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1439527996
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896",
"triggerID" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"triggerType" : "PUSH"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943",
"triggerID" : "1416916044",
"triggerType" : "MANUAL"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14961",
"triggerID" : "1418831613",
"triggerType" : "MANUAL"
}, {
"hash" : "5330d2b92b913cb02ebae8f638390f2d28448523",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15307",
"triggerID" : "5330d2b92b913cb02ebae8f638390f2d28448523",
"triggerType" : "PUSH"
}, {
"hash" : "37d4b6f2f06b77a22650e2769d7837a72cccecb7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15330",
"triggerID" : "37d4b6f2f06b77a22650e2769d7837a72cccecb7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5330d2b92b913cb02ebae8f638390f2d28448523 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15307)
* 37d4b6f2f06b77a22650e2769d7837a72cccecb7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15330)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1415523287
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896",
"triggerID" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a1762e065e18c40ff4ed1b519056647d2640121 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1418384392
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896",
"triggerID" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"triggerType" : "PUSH"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943",
"triggerID" : "1416916044",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 8a1762e065e18c40ff4ed1b519056647d2640121 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #7840:
URL: https://github.com/apache/hudi/pull/7840#discussion_r1112766344
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/BitCaskDiskMap.java:
##########
@@ -452,6 +452,8 @@ private byte[] decompressBytes(final byte[] bytes) throws IOException {
return decompressBaos.toByteArray();
} catch (IOException e) {
throw new HoodieIOException("IOException while decompressing bytes", e);
+ } finally {
Review Comment:
Does the code
```java
try (InputStream in = new InflaterInputStream(new ByteArrayInputStream(bytes))) {
}
```
work correctly here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1420104551
Hi @danny0405
I found a lot of errors in the test, but can't find the root cause of why the expected commit failed.
```
1000360 [pool-1121-thread-1] WARN org.apache.hudi.utilities.deltastreamer.TestHoodieDeltaStreamer [] - Got error :
org.opentest4j.AssertionFailedError: Got=0, exp >=1 ==> expected: <true> but was: <false>
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55) ~[junit-jupiter-api-5.7.2.jar:5.7.2]
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:40) ~[junit-jupiter-api-5.7.2.jar:5.7.2]
at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:193) ~[junit-jupiter-api-5.7.2.jar:5.7.2]
at org.apache.hudi.utilities.deltastreamer.TestHoodieDeltaStreamer$TestHelpers.assertAtLeastNReplaceCommits(TestHoodieDeltaStreamer.java:411) ~[test-classes/:?]
at org.apache.hudi.utilities.deltastreamer.TestHoodieDeltaStreamer.lambda$testInlineClustering$10(TestHoodieDeltaStreamer.java:948) ~[test-classes/:?]
at org.apache.hudi.utilities.deltastreamer.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$1(TestHoodieDeltaStreamer.java:387) ~[test-classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
```
May I ask how I can locate the root cause of test failure? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1418797765
Two tests are failing, not sure whether they are related:
```xml
[ERROR] Errors:
[ERROR] TestHoodieDeltaStreamer.testHoodieIndexer:1163->deltaStreamerTestRunner:929->deltaStreamerTestRunner:884->deltaStreamerTestRunner:901 » Execution
[ERROR] TestHoodieDeltaStreamerWithMultiWriter.testUpsertsContinuousModeWithMultipleWritersWithoutConflicts:162 » Timeout
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1439610597
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896",
"triggerID" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"triggerType" : "PUSH"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943",
"triggerID" : "1416916044",
"triggerType" : "MANUAL"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14961",
"triggerID" : "1418831613",
"triggerType" : "MANUAL"
}, {
"hash" : "5330d2b92b913cb02ebae8f638390f2d28448523",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15307",
"triggerID" : "5330d2b92b913cb02ebae8f638390f2d28448523",
"triggerType" : "PUSH"
}, {
"hash" : "37d4b6f2f06b77a22650e2769d7837a72cccecb7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15330",
"triggerID" : "37d4b6f2f06b77a22650e2769d7837a72cccecb7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 37d4b6f2f06b77a22650e2769d7837a72cccecb7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15330)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1418829450
> Two tests are failing, not sure whether they are related:
>
> ```
> [ERROR] Errors:
> [ERROR] TestHoodieDeltaStreamer.testHoodieIndexer:1163->deltaStreamerTestRunner:929->deltaStreamerTestRunner:884->deltaStreamerTestRunner:901 » Execution
> [ERROR] TestHoodieDeltaStreamerWithMultiWriter.testUpsertsContinuousModeWithMultipleWritersWithoutConflicts:162 » Timeout
> ```
Got it, thanks @danny0405 for reviewing my PR.
Just checked the error log.
The first error of `TestHoodieDeltaStreamer.testHoodieIndexer` was caused by this exception in log
```log
2023-02-06T03:31:48.7485845Z Caused by: java.lang.IllegalArgumentException
2023-02-06T03:31:48.7486389Z at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:31)
2023-02-06T03:31:48.7487103Z at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:633)
2023-02-06T03:31:48.7487932Z at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionRequestedToInflight(HoodieActiveTimeline.java:698)
2023-02-06T03:31:48.7488832Z at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.saveWorkloadProfileMetadataToInflight(BaseCommitActionExecutor.java:147)
```
I think this error is not related with the change in this PR.
And also checked the error in the second run, they are different with the first one,
```log
2023-02-03T11:38:09.4481086Z [ERROR] Errors:
2023-02-03T11:38:09.4482352Z [ERROR] TestHoodieDeltaStreamer.testHoodieAsyncClusteringJobWithScheduleAndExecute:1369->deltaStreamerTestRunner:929->deltaStreamerTestRunner:884->deltaStreamerTestRunner:901 » Execution
2023-02-03T11:38:09.4483950Z [ERROR] TestHoodieDeltaStreamer.testHoodieAsyncClusteringJobWithScheduleAndExecute:1369->deltaStreamerTestRunner:929->deltaStreamerTestRunner:884->deltaStreamerTestRunner:896 » Timeout
2023-02-03T11:38:09.4485608Z [ERROR] TestHoodieDeltaStreamer.testHoodieAsyncClusteringJobWithScheduleAndExecute:1369->deltaStreamerTestRunner:929->deltaStreamerTestRunner:884->deltaStreamerTestRunner:901 » Execution
```
Looks like Hudi DeltaStreamer test cases have some issues, let me run it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on a diff in pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on code in PR #7840:
URL: https://github.com/apache/hudi/pull/7840#discussion_r1112768566
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/BitCaskDiskMap.java:
##########
@@ -452,6 +452,8 @@ private byte[] decompressBytes(final byte[] bytes) throws IOException {
return decompressBaos.toByteArray();
} catch (IOException e) {
throw new HoodieIOException("IOException while decompressing bytes", e);
+ } finally {
Review Comment:
yeah, it is the same thing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1438129988
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896",
"triggerID" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"triggerType" : "PUSH"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943",
"triggerID" : "1416916044",
"triggerType" : "MANUAL"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14961",
"triggerID" : "1418831613",
"triggerType" : "MANUAL"
}, {
"hash" : "5330d2b92b913cb02ebae8f638390f2d28448523",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15307",
"triggerID" : "5330d2b92b913cb02ebae8f638390f2d28448523",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5330d2b92b913cb02ebae8f638390f2d28448523 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15307)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1415500341
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a1762e065e18c40ff4ed1b519056647d2640121 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1418831613
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1439496725
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896",
"triggerID" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"triggerType" : "PUSH"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943",
"triggerID" : "1416916044",
"triggerType" : "MANUAL"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14961",
"triggerID" : "1418831613",
"triggerType" : "MANUAL"
}, {
"hash" : "5330d2b92b913cb02ebae8f638390f2d28448523",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15307",
"triggerID" : "5330d2b92b913cb02ebae8f638390f2d28448523",
"triggerType" : "PUSH"
}, {
"hash" : "37d4b6f2f06b77a22650e2769d7837a72cccecb7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d4b6f2f06b77a22650e2769d7837a72cccecb7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5330d2b92b913cb02ebae8f638390f2d28448523 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15307)
* 37d4b6f2f06b77a22650e2769d7837a72cccecb7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1439490271
commit 5330d2 failed in IT, let me try it again....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #7840:
URL: https://github.com/apache/hudi/pull/7840
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1419442568
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896",
"triggerID" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"triggerType" : "PUSH"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943",
"triggerID" : "1416916044",
"triggerType" : "MANUAL"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14961",
"triggerID" : "1418831613",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 8a1762e065e18c40ff4ed1b519056647d2640121 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14961)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1416916044
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a diff in pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on code in PR #7840:
URL: https://github.com/apache/hudi/pull/7840#discussion_r1109184358
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/BitCaskDiskMap.java:
##########
@@ -452,6 +452,8 @@ private byte[] decompressBytes(final byte[] bytes) throws IOException {
return decompressBaos.toByteArray();
} catch (IOException e) {
throw new HoodieIOException("IOException while decompressing bytes", e);
+ } finally {
Review Comment:
@TengHuo : Thanks for the PR. To be defensive, can you also update line 446 so that the initialization is within the try block. Rest looks good.
InputStream in = null;
try
{
in = new InflaterInputStream(new ByteArrayInputStream(bytes));
......
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1439668758
Thanks for reviewing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1415814207
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896",
"triggerID" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a1762e065e18c40ff4ed1b519056647d2640121 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1420196427
Does the tests run correctly in local?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1437906788
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896",
"triggerID" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"triggerType" : "PUSH"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943",
"triggerID" : "1416916044",
"triggerType" : "MANUAL"
}, {
"hash" : "8a1762e065e18c40ff4ed1b519056647d2640121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14961",
"triggerID" : "1418831613",
"triggerType" : "MANUAL"
}, {
"hash" : "5330d2b92b913cb02ebae8f638390f2d28448523",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5330d2b92b913cb02ebae8f638390f2d28448523",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a1762e065e18c40ff4ed1b519056647d2640121 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14896) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14943) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14961)
* 5330d2b92b913cb02ebae8f638390f2d28448523 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #7840: [MINOR] Close InflaterInputStream in finally when calling decompressBytes in BitCaskDiskMap
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on PR #7840:
URL: https://github.com/apache/hudi/pull/7840#issuecomment-1437871300
> Does the tests run correctly in local?
The test case should be good. It can pass in our internal Hudi version. Let me re-run it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org