You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "KannarFr (via GitHub)" <gi...@apache.org> on 2024/09/12 12:19:32 UTC
[I] [Bug] Offloaders initialization is not blocking [pulsar]
KannarFr opened a new issue, #23294:
URL: https://github.com/apache/pulsar/issues/23294
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### Version
3.3.1
### Minimal reproduce step
I have a broker in a cluster of 9 brokers managing 15k namespaces and around 70 000topics distributed on these namespaces and using offload on 50% of them.
Restart a broker.
### What did you expect to see?
Nothing wrong.
### What did you see instead?
```
Sep 12 12:13:00 -pulsar-broker-c3-n9 pulsar[1622527]: 2024-09-12T12:13:00,382+0000 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Unknown exception for ManagedLedgerException.
Sep 12 12:13:00 -pulsar-broker-c3-n9 pulsar[1622527]: java.lang.UnsupportedOperationException: null
Sep 12 12:13:00 -pulsar-broker-c3-n9 pulsar[1622527]: at org.apache.bookkeeper.mledger.impl.NullLedgerOffloader.readOffloaded(NullLedgerOffloader.java:52) ~[org.apache.pulsar-managed-ledger-3.3.1.jar:3.3.1]
```
```
Sep 12 12:09:02 -pulsar-broker-c3-n9 pulsar[1622527]: 2024-09-12T12:09:02,086+0000 [broker-client-shared-internal-executor-5-1] ERROR org.apache.pulsar.broker.PulsarService - create ledgerOffloader failed for namespace orga_8b852f0a-1135-4b4f-8eee-a538f5dd8b64a/pu
lsar_eb900453-73b3-4ebc-8e0e-ef3a38f52ddd83
Sep 12 12:09:02 -pulsar-broker-c3-n9 pulsar[1622527]: org.apache.pulsar.broker.PulsarServerException: java.io.IOException: No offloader found for driver 'aws-s3'. Please make sure you dropped the offloader nar packages under `${PULSAR_HOME}/offloaders`.
Sep 12 12:09:02 -pulsar-broker-c3-n9 pulsar[1622527]: at org.apache.pulsar.broker.PulsarService.createManagedLedgerOffloader(PulsarService.java:1505) ~[org.apache.pulsar-pulsar-broker-3.3.1.jar:3.3.1]
```
### Anything else?
It looks like the broker authorizes managed ledgers actions while the offloaders are not provided (yet).
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Offloaders initialization is not blocking [pulsar]
Posted by "KannarFr (via GitHub)" <gi...@apache.org>.
KannarFr commented on issue #23294:
URL: https://github.com/apache/pulsar/issues/23294#issuecomment-2435785789
Still having this on 3.3.2:
```
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: java.util.concurrent.CompletionException: java.lang.RuntimeException: org.apache.pulsar.broker.PulsarServerException: java.io.IOException: No offloader found for driver 'aws-s3'. Please make sure you dropped the offloader nar packages under `${PULSAR_HOME}/offloaders`.
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) ~[?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture.biApply(CompletableFuture.java:1313) ~[?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture.biApplyStage(CompletableFuture.java:1329) ~[?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture.thenCombine(CompletableFuture.java:2210) ~[?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at org.apache.pulsar.broker.service.BrokerService.getManagedLedgerConfig(BrokerService.java:1856) ~[org.apache.pulsar-pulsar-broker-3.3.2.jar:3.3.2]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at org.apache.pulsar.broker.service.BrokerService.lambda$createPersistentTopic$74(BrokerService.java:1688) ~[org.apache.pulsar-pulsar-broker-3.3.2.jar:3.3.2]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at org.apache.pulsar.broker.service.BrokerService.lambda$checkTopicAlreadyMigrated$80(BrokerService.java:1832) ~[org.apache.pulsar-pulsar-broker-3.3.2.jar:3.3.2]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) ~[?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) ~[?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at org.apache.pulsar.broker.service.AbstractTopic.lambda$getMigratedClusterUrlAsync$21(AbstractTopic.java:1369) ~[org.apache.pulsar-pulsar-broker-3.3.2.jar:3.3.2]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: Caused by: java.lang.RuntimeException: org.apache.pulsar.broker.PulsarServerException: java.io.IOException: No offloader found for driver 'aws-s3'. Please make sure you dropped the offloader nar packages under `${PULSAR_HOME}/offloaders`.
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at org.apache.pulsar.broker.service.BrokerService.lambda$getManagedLedgerConfig$87(BrokerService.java:1990) ~[org.apache.pulsar-pulsar-broker-3.3.2.jar:3.3.2]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture.biApply(CompletableFuture.java:1311) ~[?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: ... 20 more
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: Caused by: org.apache.pulsar.broker.PulsarServerException: java.io.IOException: No offloader found for driver 'aws-s3'. Please make sure you dropped the offloader nar packages under `${PULSAR_HOME}/offloaders`.
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at org.apache.pulsar.broker.PulsarService.createManagedLedgerOffloader(PulsarService.java:1530) ~[org.apache.pulsar-pulsar-broker-3.3.2.jar:3.3.2]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at org.apache.pulsar.broker.service.BrokerService.lambda$getManagedLedgerConfig$87(BrokerService.java:1987) ~[org.apache.pulsar-pulsar-broker-3.3.2.jar:3.3.2]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture.biApply(CompletableFuture.java:1311) ~[?:?]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: ... 20 more
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: Caused by: java.io.IOException: No offloader found for driver 'aws-s3'. Please make sure you dropped the offloader nar packages under `${PULSAR_HOME}/offloaders`.
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at org.apache.bookkeeper.mledger.offload.Offloaders.getOffloaderFactory(Offloaders.java:42) ~[org.apache.pulsar-managed-ledger-3.3.2.jar:3.3.2]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at org.apache.pulsar.broker.PulsarService.createManagedLedgerOffloader(PulsarService.java:1507) ~[org.apache.pulsar-pulsar-broker-3.3.2.jar:3.3.2]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at org.apache.pulsar.broker.service.BrokerService.lambda$getManagedLedgerConfig$87(BrokerService.java:1987) ~[org.apache.pulsar-pulsar-broker-3.3.2.jar:3.3.2]
Oct 24 16:50:18 pulsar-broker-c3-n1 pulsar[2274460]: at java.base/java.util.concurrent.CompletableFuture.biApply(CompletableFuture.java:1311) ~[?:?]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Offloaders initialization is not blocking [pulsar]
Posted by "FlorentinDUBOIS (via GitHub)" <gi...@apache.org>.
FlorentinDUBOIS commented on issue #23294:
URL: https://github.com/apache/pulsar/issues/23294#issuecomment-2485081106
Do we have any news about this bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Offloaders initialization is not blocking [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #23294:
URL: https://github.com/apache/pulsar/issues/23294#issuecomment-6380982978
Thanks @KannarFr for the report. On current master the offloader NARs are loaded synchronously when the first offloader is created (`PulsarService.createManagedLedgerOffloader` -> `OffloadersCache.getOrLoadOffloaders`), so "No offloader found for driver 'aws-s3'" means the scan of the offloaders directory did not load the aws-s3 NAR. A NAR that fails to load is only logged as a warning, and the result is cached. Since then the offloaders directory path has been normalized (#23438, #26372). Could you check whether this still happens on a currently supported release (4.0.x, 4.2.x or 5.0.x)? If it does, please share the broker log lines from startup containing "Searching for offloaders", "Found offloader", "Failed to load offloader" and "Found and loaded offloaders", plus your `offloadersDirectory` and `narExtractionDirectory` settings. If there's no update, this issue will be closed in a few weeks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org