You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2024/04/04 22:32:45 UTC
[PR] Fix some resource leak in tests [pinot]
Jackie-Jiang opened a new pull request, #12794:
URL: https://github.com/apache/pinot/pull/12794
Under topic #12793
- Fix the warning message of resource leak in `DirectMemory`
- Make resource leak warning more clear
- Fix resource leak in some tests (not all)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix some resource leak in tests [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #12794:
URL: https://github.com/apache/pinot/pull/12794#issuecomment-2038432829
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12794?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `0%` with `6 lines` in your changes are missing coverage. Please review.
> Project coverage is 27.97%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`cce90f8`)](https://app.codecov.io/gh/apache/pinot/pull/12794?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 207 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/pinot/pull/12794?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../pinot/segment/spi/memory/unsafe/DirectMemory.java](https://app.codecov.io/gh/apache/pinot/pull/12794?src=pr&el=tree&filepath=pinot-segment-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Fspi%2Fmemory%2Funsafe%2FDirectMemory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL21lbW9yeS91bnNhZmUvRGlyZWN0TWVtb3J5LmphdmE=) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12794?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...he/pinot/segment/spi/memory/unsafe/MmapMemory.java](https://app.codecov.io/gh/apache/pinot/pull/12794?src=pr&el=tree&filepath=pinot-segment-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Fspi%2Fmemory%2Funsafe%2FMmapMemory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL21lbW9yeS91bnNhZmUvTW1hcE1lbW9yeS5qYXZh) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12794?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #12794 +/- ##
=============================================
- Coverage 61.75% 27.97% -33.79%
+ Complexity 207 198 -9
=============================================
Files 2436 2461 +25
Lines 133233 134772 +1539
Branches 20636 20822 +186
=============================================
- Hits 82274 37698 -44576
- Misses 44911 94208 +49297
+ Partials 6048 2866 -3182
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/12794/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/12794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/12794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/12794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/12794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-61.71%)` | :arrow_down: |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/12794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.96% <0.00%> (-33.66%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.97% <0.00%> (-33.78%)` | :arrow_down: |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-27.73%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/12794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.97% <0.00%> (-33.79%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/12794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.96% <0.00%> (-33.78%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.96% <0.00%> (+0.23%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12794?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix some resource leak in tests [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12794:
URL: https://github.com/apache/pinot/pull/12794#discussion_r1559881321
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/memory/unsafe/DirectMemory.java:
##########
@@ -55,23 +54,18 @@ public void flush() {
}
@Override
- public void close()
- throws IOException {
+ public synchronized void close() {
if (!_closed) {
- synchronized (this) {
- if (!_closed) {
- Unsafer.UNSAFE.freeMemory(_address);
- _closed = true;
- }
- }
+ Unsafer.UNSAFE.freeMemory(_address);
Review Comment:
Why would multiple threads trying to close the same buffer? If that happens, that means the buffer ownership is messed up. In regular case, this method should only be invoked exactly once.
I think double-checked locking might have performance gain when used in a singleton, but not here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix some resource leak in tests [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12794:
URL: https://github.com/apache/pinot/pull/12794#discussion_r1555285434
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/memory/unsafe/MmapMemory.java:
##########
@@ -98,16 +98,11 @@ public void flush() {
}
@Override
- public void close()
- throws IOException {
+ public synchronized void close() {
try {
if (!_closed) {
- synchronized (this) {
- if (!_closed) {
- _section._unmapFun.unmap();
- _closed = true;
- }
- }
+ _section._unmapFun.unmap();
Review Comment:
Same here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix some resource leak in tests [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12794:
URL: https://github.com/apache/pinot/pull/12794#discussion_r1559881321
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/memory/unsafe/DirectMemory.java:
##########
@@ -55,23 +54,18 @@ public void flush() {
}
@Override
- public void close()
- throws IOException {
+ public synchronized void close() {
if (!_closed) {
- synchronized (this) {
- if (!_closed) {
- Unsafer.UNSAFE.freeMemory(_address);
- _closed = true;
- }
- }
+ Unsafer.UNSAFE.freeMemory(_address);
Review Comment:
Why would multiple threads trying to close the same buffer? If that happens, that means the buffer ownership is messed up. In regular case, this method should only be invoked exactly once.
After reading [this](https://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html), I'm even more worried about double-checked locking..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix some resource leak in tests [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12794:
URL: https://github.com/apache/pinot/pull/12794#discussion_r1555282507
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplIngestionAggregationTest.java:
##########
@@ -480,6 +480,8 @@ public void testBigDecimalTooBig() {
Assert.assertThrows(IllegalArgumentException.class, () -> {
mutableSegmentImpl.index(row, defaultMetadata);
});
+
+ mutableSegmentImpl.destroy();
Review Comment:
This won't destroy the segment if the test fails
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix some resource leak in tests [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #12794:
URL: https://github.com/apache/pinot/pull/12794
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix some resource leak in tests [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12794:
URL: https://github.com/apache/pinot/pull/12794#discussion_r1560757789
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/memory/unsafe/DirectMemory.java:
##########
@@ -55,23 +54,18 @@ public void flush() {
}
@Override
- public void close()
- throws IOException {
+ public synchronized void close() {
if (!_closed) {
- synchronized (this) {
- if (!_closed) {
- Unsafer.UNSAFE.freeMemory(_address);
- _closed = true;
- }
- }
+ Unsafer.UNSAFE.freeMemory(_address);
Review Comment:
Ok, lets close the other PR then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix some resource leak in tests [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12794:
URL: https://github.com/apache/pinot/pull/12794#discussion_r1559066330
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/memory/unsafe/DirectMemory.java:
##########
@@ -55,23 +54,18 @@ public void flush() {
}
@Override
- public void close()
- throws IOException {
+ public synchronized void close() {
if (!_closed) {
- synchronized (this) {
- if (!_closed) {
- Unsafer.UNSAFE.freeMemory(_address);
- _closed = true;
- }
- }
+ Unsafer.UNSAFE.freeMemory(_address);
Review Comment:
You are right, I didn't see that. Anyway, in the other PR I've changed that so we don't need to synchronize in case one thread close this after another.
What newer jvms does for a fact is to make synchronized by default is specially expensive. This is due to:
* Virtual threads being pinned with synchronize. This is secondary given we don't use them and it should be something that eventually will be improved in (even) more modern JVMs
* +UseBiasedLocking has been disabled since Java 15 and the plan is to remove it. See https://openjdk.org/jeps/374, [this](https://dev.to/vbochenin/java-17-migration-bias-locks-regression-2c5m#:~:text=What%20is%20a%20bias%20lock,the%20same%20thread%20is%20lightweight.) and [this](https://quarkus.io/blog/biased-locking-help/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix some resource leak in tests [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12794:
URL: https://github.com/apache/pinot/pull/12794#discussion_r1555397466
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/memory/unsafe/DirectMemory.java:
##########
@@ -55,23 +54,18 @@ public void flush() {
}
@Override
- public void close()
- throws IOException {
+ public synchronized void close() {
if (!_closed) {
- synchronized (this) {
- if (!_closed) {
- Unsafer.UNSAFE.freeMemory(_address);
- _closed = true;
- }
- }
+ Unsafer.UNSAFE.freeMemory(_address);
Review Comment:
The whole method is changed to `synchronized`. I don't remember where I read it, but for modern JVM double checking synchronization is not necessary, especially when we almost always only call it once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix some resource leak in tests [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12794:
URL: https://github.com/apache/pinot/pull/12794#discussion_r1555284969
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/memory/unsafe/DirectMemory.java:
##########
@@ -55,23 +54,18 @@ public void flush() {
}
@Override
- public void close()
- throws IOException {
+ public synchronized void close() {
if (!_closed) {
- synchronized (this) {
- if (!_closed) {
- Unsafer.UNSAFE.freeMemory(_address);
- _closed = true;
- }
- }
+ Unsafer.UNSAFE.freeMemory(_address);
Review Comment:
I think we have to recover the synchronization here. Otherwise a race condition may try to free an address that was already cleaned. That could end up producing a SIGSEV which would kill the server.
We can improve the code using double check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix some resource leak in tests [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12794:
URL: https://github.com/apache/pinot/pull/12794#discussion_r1555394047
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplIngestionAggregationTest.java:
##########
@@ -480,6 +480,8 @@ public void testBigDecimalTooBig() {
Assert.assertThrows(IllegalArgumentException.class, () -> {
mutableSegmentImpl.index(row, defaultMetadata);
});
+
+ mutableSegmentImpl.destroy();
Review Comment:
Currently we don't take care of resource leak during test failures. That can be addressed separately
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org