You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/03/01 21:35:20 UTC
[GitHub] [pulsar] lhotari opened a new pull request #14524: [Tests] Fix thread leak in MLTransactionMetadataStore
lhotari opened a new pull request #14524:
URL: https://github.com/apache/pulsar/pull/14524
### Motivation
- MLTransactionMetadataStore.internalPinnedExecutor wasn't closed
when MLTransactionMetadataStore.closeAsync was called
- problem was introduced by #14238 changes
- this issue causes tests to fail with OOME. This might also impact
production code in some way.
### Modifications
- shutdown the ExecutorService when MLTransactionMetadataStore.closeAsync is called
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #14524: [Tests] Fix thread leak in MLTransactionMetadataStore
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #14524:
URL: https://github.com/apache/pulsar/pull/14524#discussion_r817531242
##########
File path: pulsar-transaction/coordinator/src/main/java/org/apache/pulsar/transaction/coordinator/impl/MLTransactionMetadataStore.java
##########
@@ -442,15 +444,24 @@ public TransactionCoordinatorStats getCoordinatorStats() {
@Override
public CompletableFuture<Void> closeAsync() {
- return transactionLog.closeAsync().thenCompose(v -> {
- txnMetaMap.clear();
- this.timeoutTracker.close();
- if (!this.changeToCloseState()) {
- return FutureUtil.failedFuture(
- new IllegalStateException("Managed ledger transaction metadata store state to close error!"));
- }
+ if (changeToClosingState()) {
+ // Disable new tasks from being submitted
+ internalPinnedExecutor.shutdown();
+ return transactionLog.closeAsync().thenCompose(v -> {
+ txnMetaMap.clear();
+ this.timeoutTracker.close();
+ if (!this.changeToCloseState()) {
+ return FutureUtil.failedFuture(
+ new IllegalStateException(
+ "Managed ledger transaction metadata store state to close error!"));
+ }
+ // Shutdown the ExecutorService
+ MoreExecutors.shutdownAndAwaitTermination(internalPinnedExecutor, Duration.ofSeconds(5L));
Review comment:
this appears to be a blocking call.
But I can't suggest a better way
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
##########
@@ -446,7 +441,16 @@ public void close() throws PulsarServerException {
List<CompletableFuture<Void>> asyncCloseFutures = new ArrayList<>();
if (this.brokerService != null) {
- asyncCloseFutures.add(this.brokerService.closeAsync());
+ CompletableFuture<Void> brokerCloseFuture = this.brokerService.closeAsync();
+ if (this.transactionMetadataStoreService != null) {
+ asyncCloseFutures.add(brokerCloseFuture.thenAccept(__ -> {
Review comment:
should this be "whenComplete" o "handle" ? otherwise in case of failure we are not executing this code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #14524: [Tests] Fix thread leak in MLTransactionMetadataStore
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #14524:
URL: https://github.com/apache/pulsar/pull/14524#issuecomment-1056860303
@lhotari Yes, this should be a blocker for 2.10.0 release, will cherry-pick this PR and start a new VOTE for 2.10.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14524: [Tests] Fix thread leak in MLTransactionMetadataStore
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14524:
URL: https://github.com/apache/pulsar/pull/14524#issuecomment-1056850987
@codelipenghui Is the thread leak a blocker for 2.10 release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14524: [Tests] Fix thread leak in MLTransactionMetadataStore
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14524:
URL: https://github.com/apache/pulsar/pull/14524#discussion_r817534451
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
##########
@@ -446,7 +441,16 @@ public void close() throws PulsarServerException {
List<CompletableFuture<Void>> asyncCloseFutures = new ArrayList<>();
if (this.brokerService != null) {
- asyncCloseFutures.add(this.brokerService.closeAsync());
+ CompletableFuture<Void> brokerCloseFuture = this.brokerService.closeAsync();
+ if (this.transactionMetadataStoreService != null) {
+ asyncCloseFutures.add(brokerCloseFuture.thenAccept(__ -> {
Review comment:
thanks, good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14524: [Tests] Fix thread leak in MLTransactionMetadataStore
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14524:
URL: https://github.com/apache/pulsar/pull/14524#discussion_r817607666
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
##########
@@ -446,7 +441,16 @@ public void close() throws PulsarServerException {
List<CompletableFuture<Void>> asyncCloseFutures = new ArrayList<>();
if (this.brokerService != null) {
- asyncCloseFutures.add(this.brokerService.closeAsync());
+ CompletableFuture<Void> brokerCloseFuture = this.brokerService.closeAsync();
+ if (this.transactionMetadataStoreService != null) {
+ asyncCloseFutures.add(brokerCloseFuture.thenAccept(__ -> {
Review comment:
@eolivelli PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari merged pull request #14524: [Tests] Fix thread leak in MLTransactionMetadataStore
Posted by GitBox <gi...@apache.org>.
lhotari merged pull request #14524:
URL: https://github.com/apache/pulsar/pull/14524
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14524: [Tests] Fix thread leak in MLTransactionMetadataStore
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14524:
URL: https://github.com/apache/pulsar/pull/14524#issuecomment-1055909213
@lordcheng10 Could this help in reducing memory consumption of branch-2.10 (v2.10.0-candidate-2) & master branch version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14524: [Tests] Fix thread leak in MLTransactionMetadataStore
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14524:
URL: https://github.com/apache/pulsar/pull/14524#issuecomment-1056849155
btw. The thread leak is visible also in test output. #10195 added https://github.com/apache/pulsar/blob/master/buildtools/src/main/java/org/apache/pulsar/tests/ThreadLeakDetectorListener.java which logs threads that are active when the test ends. For example, when running
```
mvn -DredirectTestOutputToFile=false -DtestRetryCount=0 test -Pcore-modules -pl pulsar-broker -DexcludedGroups='' -Dtest=ReplicatorTest#testDoNotReplicateSystemTopic
```
this can be seen in the console:
```
[INFO] [stdout] 2022-03-02T13:37:21,120+0200 [main] WARN org.apache.pulsar.tests.ThreadLeakDetectorListener - Tests in class ReplicatorTest created thread id 1407 with name 'Thread[transaction_coordinator_TransactionCoordinatorID(id=0)thread_factory-468-1,5,main]'
[INFO] [stdout] 2022-03-02T13:37:21,120+0200 [main] WARN org.apache.pulsar.tests.ThreadLeakDetectorListener - Tests in class ReplicatorTest created thread id 1406 with name 'Thread[transaction_coordinator_TransactionCoordinatorID(id=1)thread_factory-467-1,5,main]'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org