You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/06/14 07:20:33 UTC
[GitHub] [pulsar] KKcorps opened a new issue #10903: PulsarStandalone throws error while creating Ledger
KKcorps opened a new issue #10903:
URL: https://github.com/apache/pulsar/issues/10903
**Describe the bug**
I am using `pulsar-broker-shaded` dependency to create `PulsarStandalone` cluster locally. However, the cluster fails at startup with the following exception -
```
java.io.IOException: Failed to instantiate ledger storage : org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
at org.apache.pulsar.shade.org.apache.bookkeeper.bookie.LedgerStorageFactory.createLedgerStorage(LedgerStorageFactory.java:35)
at org.apache.pulsar.shade.org.apache.bookkeeper.bookie.Bookie.buildLedgerStorage(Bookie.java:637)
at org.apache.pulsar.shade.org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:759)
at org.apache.pulsar.shade.org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152)
at org.apache.pulsar.shade.org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:120)
at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runBookies(LocalBookkeeperEnsemble.java:302)
at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:430)
at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:258)
at org.kharekartik.Main.main(Main.java:39)
Caused by: java.lang.RuntimeException: org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage not org.apache.pulsar.shade.org.apache.bookkeeper.bookie.LedgerStorage
at org.apache.pulsar.shade.org.apache.bookkeeper.common.util.ReflectionUtils.newInstance(ReflectionUtils.java:169)
at org.apache.pulsar.shade.org.apache.bookkeeper.bookie.LedgerStorageFactory.createLedgerStorage(LedgerStorageFactory.java:33)
... 8 more
```
The `pulsar-broker` unshaded dependency works correctly but I can't use it as it throws a lot of dependency convergence errors due to `maven enforcer` plugin. The plugin is present in parent pom and is mandatory so can't disable it.
**To Reproduce**
```java
final File clusterConfigFile = new File(Main.class.getClassLoader().getResource("standalone.conf").toURI());
ServiceConfiguration config =
PulsarConfigurationLoader.create((new FileInputStream(clusterConfigFile)), ServiceConfiguration.class);
config.setManagedLedgerDefaultEnsembleSize(1);
config.setManagedLedgerDefaultWriteQuorum(1);
config.setManagedLedgerDefaultAckQuorum(1);
String zkServers = "127.0.0.1";
config.setAdvertisedAddress("localhost");
PulsarStandalone
pulsarStandalone = PulsarStandaloneBuilder.instance().withConfig(config).withNoStreamStorage(true).build();
if (config.getZookeeperServers() != null) {
pulsarStandalone.setZkPort(Integer.parseInt(config.getZookeeperServers().split(":")[1]));
}
config.setZookeeperServers(zkServers + ":" + pulsarStandalone.getZkPort());
config.setConfigurationStoreServers(zkServers + ":" + pulsarStandalone.getZkPort());
config.setRunningStandalone(true);
pulsarStandalone.setConfigFile(clusterConfigFile.getAbsolutePath());
pulsarStandalone.setConfig(config);
pulsarStandalone.start();
PulsarAdmin admin =
PulsarAdmin.builder().serviceHttpUrl("http://localhost:" + config.getWebServicePort().get()).build();
admin.topics().createPartitionedTopic("test-topic", 3);
```
```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>pulsar-standalone-repro</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<pulsar.version>2.7.2</pulsar.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-broker-shaded</artifactId>
<version>${pulsar.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client-original</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client-admin-original</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client</artifactId>
<version>${pulsar.version}</version>
</dependency>
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client-admin</artifactId>
<version>${pulsar.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>8</source>
<target>8</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
```
The `standalone.conf` is the one supplied out of the box in pulsar package.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] KKcorps commented on issue #10903: PulsarStandalone throws error while creating Ledger
Posted by GitBox <gi...@apache.org>.
KKcorps commented on issue #10903:
URL: https://github.com/apache/pulsar/issues/10903#issuecomment-874751625
@eolivelli I am currently using pulsar-broker only. I want to use the shaded one because it would lead to fewer dependency convergence errors via maven enforcer plugin.
Anyways, the shaded one is also working now after the following changes in pom.
https://github.com/apache/pulsar/compare/v2.7.2...KKcorps:pulsar_broker_shaded_patch?expand=1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #10903: PulsarStandalone throws error while creating Ledger
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #10903:
URL: https://github.com/apache/pulsar/issues/10903#issuecomment-874224623
If you can use "pulsar-broker" instead of "pulsar-broker-shaded" life will be simpler.
I usually create a class like this one in order to launch PulsarService
https://github.com/datastax/pulsar-jms/blob/master/pulsar-jms/src/test/java/com/datastax/oss/pulsar/jms/utils/PulsarCluster.java
Btw BookKeeper has a way to workaround "shading"/"class name relocation"
see here
https://github.com/apache/bookkeeper/blob/4c6cf097d9b823c8e839e176a06266a58bcb7bf5/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/AbstractConfiguration.java#L82
in your broker.conf you can set `bookkeeper_allowShadedLedgerManagerFactoryClass` and `bookkeeper_shadedLedgerManagerFactoryClassPrefix`
I have not tried it, but I hope that helps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on issue #10903: PulsarStandalone throws error while creating Ledger
Posted by GitBox <gi...@apache.org>.
zymap commented on issue #10903:
URL: https://github.com/apache/pulsar/issues/10903#issuecomment-868150460
@KKcorps Why do you need to set up the standalone cluster in this way? Did you want to do some test or something else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #10903: PulsarStandalone throws error while creating Ledger
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #10903:
URL: https://github.com/apache/pulsar/issues/10903#issuecomment-1058889493
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] KKcorps commented on issue #10903: PulsarStandalone throws error while creating Ledger
Posted by GitBox <gi...@apache.org>.
KKcorps commented on issue #10903:
URL: https://github.com/apache/pulsar/issues/10903#issuecomment-868281329
Yes, I need it for testing using TestNG. The team doesn't want docker container for testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #10903: PulsarStandalone throws error while creating Ledger
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #10903:
URL: https://github.com/apache/pulsar/issues/10903#issuecomment-874224623
If you can use "pulsar-broker" instead of "pulsar-broker-shaded" life will be simpler.
I usually create a class like this one in order to launch PulsarService
https://github.com/datastax/pulsar-jms/blob/master/pulsar-jms/src/test/java/com/datastax/oss/pulsar/jms/utils/PulsarCluster.java
Btw BookKeeper has a way to workaround "shading"/"class name relocation"
see here
https://github.com/apache/bookkeeper/blob/4c6cf097d9b823c8e839e176a06266a58bcb7bf5/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/AbstractConfiguration.java#L82
in your broker.conf you can set `bookkeeper_allowShadedLedgerManagerFactoryClass` and `bookkeeper_shadedLedgerManagerFactoryClassPrefix`
I have not tried it, but I hope that helps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on issue #10903: PulsarStandalone throws error while creating Ledger
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on issue #10903:
URL: https://github.com/apache/pulsar/issues/10903#issuecomment-862008039
@KKcorps
`org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage` implements `org.apache.bookkeeper.bookie.LedgerStorage`
But after shading the interface become `org.apache.pulsar.shade.org.apache.bookkeeper.bookie.LedgerStorage`
Can you try update the config `ledgerStorageClass` in `bookkeeper.conf` to `org.apache.pulsar.shade.org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
`, it's the shaded version of DbLedgerStorage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org