You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "lhotari (via GitHub)" <gi...@apache.org> on 2024/04/25 14:22:09 UTC
[PR] [improve][ci] Disable test that causes OOME until the problem has been resolved [pulsar]
lhotari opened a new pull request, #22586:
URL: https://github.com/apache/pulsar/pull/22586
### Motivation
Unit test group 1 fails often with OOME.
### Modifications
The issue is most like related to https://github.com/apache/pulsar/pull/21495 and org.apache.pulsar.broker.service.ReplicatorSubscriptionTest#testWriteMarkerTaskOfReplicateSubscriptions .
Disable the test until the problem has been resolved.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][ci] Disable test that causes OOME until the problem has been resolved [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #22586:
URL: https://github.com/apache/pulsar/pull/22586#issuecomment-2077871792
There are more problems. Using heap dump from https://github.com/apache/pulsar/actions/runs/8835173621/attempts/1?pr=22583
```sql
select toString(this['stack.fn.arg$1']), count(*) from java.util.concurrent.CompletableFuture where this['stack.fn'] is not null group by 1 order by 2 desc
```
```
EXPR$0 | EXPR$1
-------------------------------------------------------------------------------------------------------------------------
org.apache.pulsar.broker.service.persistent.SystemTopic @ 0x100031c173f8 | 435,961
org.apache.pulsar.broker.resources.NamespaceResources$PartitionedTopicResources @ 0x10002b285920 | 56,647
| 144
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient @ 0x10000a69cfe0 | 17
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient @ 0x100011873a80 | 17
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient @ 0x100007eb5af8 | 13
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient @ 0x100018f4fd48 | 13
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient @ 0x10002b0e1900 | 12
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient @ 0x10001c039270 | 12
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient @ 0x10003be7e448 | 11
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient @ 0x10003bfa8e60 | 11
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient @ 0x10000c4b8f78 | 9
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient @ 0x10000c439b80 | 9
org.apache.pulsar.client.impl.BinaryProtoLookupService @ 0x10002d78f820 | 7
org.apache.pulsar.broker.service.persistent.PersistentTopic @ 0x1000281e0380 | 6
org.apache.pulsar.client.impl.PulsarClientImpl @ 0x10002d78dd90 | 5
org.apache.pulsar.client.impl.ConnectionHandler @ 0x10002d60cfc8 | 4
org.apache.pulsar.client.impl.ConnectionHandler @ 0x10002d619f28 | 4
org.apache.pulsar.client.impl.ConnectionHandler @ 0x100024745fc8 | 4
ASSIGN | 4
org.apache.pulsar.client.impl.ConnectionHandler @ 0x10002a0f8390 | 4
org.apache.pulsar.client.impl.ConnectionHandler @ 0x10002d6132f8 | 4
org.apache.pulsar.client.impl.ConnectionHandler @ 0x10002d796090 | 4
org.apache.pulsar.broker.namespace.OwnershipCache$OwnedServiceUnitCacheLoader @ 0x10001c0314e8 | 3
org.apache.pulsar.broker.namespace.OwnershipCache$OwnedServiceUnitCacheLoader @ 0x10003be76ef8 | 2
org.apache.pulsar.broker.service.persistent.GeoPersistentReplicator @ 0x1003f30fc0c8 | 2
org.apache.pulsar.metadata.impl.ZKMetadataStore @ 0x1003ef17fef0 | 2
org.apache.pulsar.client.impl.TableViewImpl @ 0x10002725bb30 | 2
org.apache.pulsar.client.impl.TableViewImpl @ 0x10002a1bf700 | 2
org.apache.pulsar.client.impl.TableViewImpl @ 0x10002d6684a0 | 2
org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService @ 0x1000182bb060 | 2
org.apache.pulsar.broker.systopic.TopicPoliciesSystemTopicClient @ 0x1003f0a66de0 | 2
org.apache.pulsar.broker.namespace.OwnershipCache$OwnedServiceUnitCacheLoader @ 0x10000c5040b0 | 2
org.apache.pulsar.broker.namespace.OwnershipCache$OwnedServiceUnitCacheLoader @ 0x100018f47fb0 | 2
java.util.concurrent.CompletableFuture @ 0x1003ed849b98 | 1
persistent://pulsar/global/removeClusterTest/__change_events | 1
org.apache.pulsar.client.impl.ReaderImpl @ 0x10002a0f22f0 | 1
java.util.concurrent.CompletableFuture @ 0x1003ef194a50 | 1
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$3215+0x00007f588ce12e20 @ 0x1000234ce2b8| 1
org.apache.pulsar.client.impl.ConnectionPool @ 0x10002d78e760 | 1
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$3215+0x00007f588ce12e20 @ 0x1003eb48e700| 1
org.apache.pulsar.client.impl.ReaderImpl @ 0x10002d613e88 | 1
org.apache.pulsar.metadata.coordination.impl.LockManagerImpl @ 0x100011873a20 | 1
org.apache.pulsar.metadata.coordination.impl.LockManagerImpl @ 0x10003be700d0 | 1
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$3215+0x00007f588ce12e20 @ 0x1003fda42068| 1
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$3215+0x00007f588ce12e20 @ 0x1000234ce410| 1
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$3215+0x00007f588ce12e20 @ 0x1003eb4dbbc0| 1
org.apache.pulsar.client.impl.ReaderImpl @ 0x1003fbf276b0 | 1
java.util.concurrent.CompletableFuture @ 0x100011edcc18 | 1
org.apache.pulsar.client.impl.ReaderImpl @ 0x10002d60d258 | 1
org.apache.pulsar.broker.service.persistent.SystemTopic @ 0x100011877ef0 | 1
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$3215+0x00007f588ce12e20 @ 0x1003fdae3248| 1
org.apache.pulsar.metadata.coordination.impl.LockManagerImpl @ 0x10001c02a548 | 1
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$3215+0x00007f588ce12e20 @ 0x100023429888| 1
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$3215+0x00007f588ce12e20 @ 0x1003fda421c0| 1
org.apache.pulsar.compaction.StrategicTwoPhaseCompactor @ 0x1000392ec5a0 | 1
java.util.concurrent.CompletableFuture @ 0x1003f0a67038 | 1
org.apache.pulsar.metadata.coordination.impl.LockManagerImpl @ 0x10000c4f6a20 | 1
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$3215+0x00007f588ce12e20 @ 0x1000234fcbe8| 1
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$3215+0x00007f588ce12e20 @ 0x1000234fca90| 1
org.apache.pulsar.metadata.coordination.impl.LockManagerImpl @ 0x100018f40f28 | 1
java.util.concurrent.CompletableFuture @ 0x100039d71458 | 1
org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService @ 0x10002a0a3ae8 | 1
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$3215+0x00007f588ce12e20 @ 0x1000234299e0| 1
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$3215+0x00007f588ce12e20 @ 0x1003fdae39b8| 1
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$3215+0x00007f588ce12e20 @ 0x1003eb48e858| 1
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$3215+0x00007f588ce12e20 @ 0x1003eb4dba68| 1
Total: 67 entries | 492,978
-------------------------------------------------------------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][ci] Disable test that causes OOME until the problem has been resolved [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #22586:
URL: https://github.com/apache/pulsar/pull/22586#issuecomment-2077575391
Just wondering if the problem is somehow related to namespace deletion with replication enabled.
The namespace deletion code is something that will need to be refactored in any case to get the concurrency into control.
https://github.com/apache/pulsar/blob/d7d54522933b63f6a74ec7139c6dedebe8ad9149/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java#L216-L344
The concurrency issue is explained in https://github.com/apache/pulsar/pull/22541#issuecomment-2071568113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][ci] Disable test that causes OOME until the problem has been resolved [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #22586:
URL: https://github.com/apache/pulsar/pull/22586#issuecomment-2077595847
the namespace deletion in the test might be the code that triggers the problem:
https://github.com/apache/pulsar/blob/e81a20d667aef7c0f888e88dbcf972196012ebea/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ReplicatorSubscriptionTest.java#L821-L824
@poorbarcode do you have a chance to debug this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][ci] Disable test that causes OOME until the problem has been resolved [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #22586:
URL: https://github.com/apache/pulsar/pull/22586
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][ci] Disable test that causes OOME until the problem has been resolved [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #22586:
URL: https://github.com/apache/pulsar/pull/22586#issuecomment-2077963851
```sql
select toString(this['result.ex.detailMessage']), count(*) from java.util.concurrent.CompletableFuture where this['result.ex.detailMessage'] is not null group by 1 order by 2 desc
```
```
EXPR$0 | EXPR$1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: | 209,380
Lock was not in valid state: Releasing | 23
BookKeeper client is closed | 15
org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.apache.bookkeeper.mledger.ManagedLedgerException$CursorAlreadyClosedException: Cursor was already closed| 2
Failed to close clients before deleting topic. | 1
Total: 5 entries | 209,421
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][ci] Disable test that causes OOME until the problem has been resolved [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #22586:
URL: https://github.com/apache/pulsar/pull/22586#issuecomment-2077351106
In one of the heap dumps, there was 251,029 lambdas which all reference a `__change_events` topic.
```sql
select this['arg$2.completeTopicName'], count(*) from "org.apache.pulsar.broker.resources.NamespaceResources$PartitionedTopicResources$$Lambda$1819+0x00007f08a8b65ee8" group by 1
```
```
EXPR$0 | EXPR$1
----------------------------------------------------------------------------------------------------------
persistent://pulsar/testReplicateSubBackLog-88c7c05a-0f07-4ed2-b82a-9b911afad922/__change_events| 251,029
----------------------------------------------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][ci] Disable test that causes OOME until the problem has been resolved [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #22586:
URL: https://github.com/apache/pulsar/pull/22586#issuecomment-2077437609
There are a few recent replicator related changes #21946, #21948 and #22537 . @poorbarcode please check if one of the changes is triggering the OOME issue possibly related to deletion. There are a lot of entries for `__change_events` topic for the replicated namespace.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][ci] Disable test that causes OOME until the problem has been resolved [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #22586:
URL: https://github.com/apache/pulsar/pull/22586#issuecomment-2077363787
In another heapdump
```sql
select this['arg$2.completeTopicName'], count(*) from "org.apache.pulsar.broker.resources.NamespaceResources$PartitionedTopicResources$$Lambda$3405+0x00007fae50f7b000" group by 1
```
```
EXPR$0 | EXPR$1
--------------------------------------------------------------------------------------------------------------
persistent://pulsar/testReplicateSubBackLog-acf0e69a-2836-430c-bf1e-c9babd059179/replication-disable| 4
persistent://pulsar/testReplicateSubBackLog-acf0e69a-2836-430c-bf1e-c9babd059179/__change_events | 277,899
Total: 2 entries | 277,903
--------------------------------------------------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org