You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/03/31 19:18:46 UTC
[GitHub] [pulsar] anshul-dhyani opened a new issue #6645: Error open RocksDB
database
anshul-dhyani opened a new issue #6645: Error open RocksDB database
URL: https://github.com/apache/pulsar/issues/6645
Unable to start pulsar in docker
**OS: Windows 10 home**
Below is the command I used :
```
docker run -d --restart=unless-stopped --name pulsar -p 6650:6650 -p 8081:8080 -v /c/Users/abc/dockerizedpulsardata:/pulsar/data apachepulsar/pulsar:2.5.0 bin/pulsar standalone
```
### I get this error:
```
18:48:18.651 [main] WARN org.apache.bookkeeper.util.EventLoopUtil - Could not use Netty Epoll event loop: failed to load the required native library
18:48:19.144 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
18:48:19.145 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
18:48:19.146 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
18:48:19.157 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://127.0.0.1:2181/ledgers : zkServers = 127.0.0.1:2181, ledgersRootPath = /ledgers.
18:48:19.164 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=10000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@4bdc8b5d
18:48:19.165 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
18:48:19.165 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
18:48:19.175 [main-SendThread(127.0.0.1:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18:48:19.178 [main-SendThread(127.0.0.1:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /127.0.0.1:59296, server: localhost/127.0.0.1:2181
18:48:19.182 [main-SendThread(127.0.0.1:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100052dc4e40001, negotiated timeout = 10000
18:48:19.185 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
18:48:19.216 [main] INFO org.apache.bookkeeper.discover.ZKRegistrationManager - INSTANCEID not exists in zookeeper. Not considering it for data verification
18:48:19.498 [main] INFO org.apache.bookkeeper.bookie.Bookie - instantiate ledger manager org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory
18:48:19.579 [main] ERROR org.apache.bookkeeper.bookie.Journal - Problems reading from data/standalone/bookkeeper0/current/lastMark (this is okay if it is the first time starting this bookie
18:48:19.582 [main] INFO org.apache.bookkeeper.bookie.Bookie - Using ledger storage: org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
18:48:19.592 [main] INFO org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - Started Db Ledger Storage
18:48:19.593 [main] INFO org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - - Number of directories: 1
18:48:19.594 [main] INFO org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - - Write cache size: 1024 MB
18:48:19.594 [main] INFO org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - - Read Cache: 1024 MB
18:48:19.604 [main] INFO org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - Creating single directory db ledger storage on data/standalone/bookkeeper0/current
18:48:20.050 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
18:48:20.075 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service.
java.io.IOException: Error open RocksDB database
at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:182) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.lambda$static$0(KeyValueStorageRocksDB.java:58) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.<init>(LedgerMetadataIndex.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>(SingleDirectoryDbLedgerStorage.java:161) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:149) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:129) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:775) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:136) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:105) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runBookies(LocalBookkeeperEnsemble.java:278) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.5.0.jar:2.5.0]
at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:403) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.5.0.jar:2.5.0]
at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:260) ~[org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:119) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
Caused by: org.rocksdb.RocksDBException: While fsync: a directory: Invalid argument
at org.rocksdb.RocksDB.open(Native Method) ~[org.rocksdb-rocksdbjni-5.13.3.jar:?]
at org.rocksdb.RocksDB.open(RocksDB.java:231) ~[org.rocksdb-rocksdbjni-5.13.3.jar:?]
at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:179) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
... 13 more
18:48:20.095 [Thread-1] ERROR org.apache.pulsar.PulsarStandaloneStarter - Shutdown failed: null
[AppClassLoader@18b4aac2] info AspectJ Weaver Version 1.9.2 built on Wednesday Oct 24, 2018 at 15:43:33 GMT
[AppClassLoader@18b4aac2] info register classloader sun.misc.Launcher$AppClassLoader@18b4aac2
[AppClassLoader@18b4aac2] info using configuration file:/pulsar/lib/org.apache.pulsar-pulsar-zookeeper-2.5.0.jar!/META-INF/aop.xml
[AppClassLoader@18b4aac2] info using configuration file:/pulsar/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.5.0.jar!/META-INF/aop.xml
[AppClassLoader@18b4aac2] info register aspect org.apache.pulsar.zookeeper.FinalRequestProcessorAspect
[AppClassLoader@18b4aac2] info register aspect org.apache.pulsar.zookeeper.ZooKeeperServerAspect
[AppClassLoader@18b4aac2] info register aspect org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect
```
### If I use below command
```
docker run -d --restart=unless-stopped --name pulsar -p 6650:6650 -p 8081:8080 -v /c/Users/abc/dockerizedpulsardata:/pulsar/data -v /c/Users/abc/dockerizedpulsarconf:/pulsar/conf apachepulsar/pulsar:2.5.0 bin/pulsar standalone
```
it ends up with below error
```
[AppClassLoader@18b4aac2] info AspectJ Weaver Version 1.9.2 built on Wednesday Oct 24, 2018 at 15:43:33 GMT
[AppClassLoader@18b4aac2] info register classloader sun.misc.Launcher$AppClassLoader@18b4aac2
[AppClassLoader@18b4aac2] info using configuration file:/pulsar/lib/org.apache.pulsar-pulsar-zookeeper-2.5.0.jar!/META-INF/aop.xml
[AppClassLoader@18b4aac2] info using configuration file:/pulsar/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.5.0.jar!/META-INF/aop.xml
[AppClassLoader@18b4aac2] info register aspect org.apache.pulsar.zookeeper.FinalRequestProcessorAspect
[AppClassLoader@18b4aac2] info register aspect org.apache.pulsar.zookeeper.ZooKeeperServerAspect
[AppClassLoader@18b4aac2] info register aspect org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect
ERROR StatusLogger File not found in file system or classpath: log4j2.yaml
ERROR StatusLogger Reconfiguration failed: No configuration found for '18b4aac2' at 'null' in 'null'
Exception in thread "main" java.io.FileNotFoundException: /pulsar/conf/standalone.conf (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at org.apache.pulsar.PulsarStandaloneStarter.<init>(PulsarStandaloneStarter.java:60)
at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:117)
[AppClassLoader@18b4aac2] info AspectJ Weaver Version 1.9.2 built on Wednesday Oct 24, 2018 at 15:43:33 GMT
```
it only works without volume sharing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] anshul-dhyani commented on issue #6645: Error while
running pulsar on docker
Posted by GitBox <gi...@apache.org>.
anshul-dhyani commented on issue #6645: Error while running pulsar on docker
URL: https://github.com/apache/pulsar/issues/6645#issuecomment-607502588
yes. third command works with ```-nfs -nss```.
But other two commands don't work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] anshul-dhyani commented on issue #6645: Error while
running pulsar on docker
Posted by GitBox <gi...@apache.org>.
anshul-dhyani commented on issue #6645: Error while running pulsar on docker
URL: https://github.com/apache/pulsar/issues/6645#issuecomment-607549416
Is this expected behavior or a bug ?
I can share volume for mongoDB on the same infra and that is also using linux as I can get into that container and can run linux commands
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #6645: Error while running pulsar
on docker
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #6645: Error while running pulsar on docker
URL: https://github.com/apache/pulsar/issues/6645#issuecomment-607556864
@anshul-dhyani sorry I am not very familiar with docker on windows. so it is hard to tell whether it is a bug at the Pulsar side or it is a bug on the docker side. But from the error message "org.rocksdb.RocksDBException: While fsync: a directory: Invalid argument", I guess it is a problem from the docker side. It seems that the volume mounted doesn't support fsync.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #6645: Error while running pulsar
on docker
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #6645: Error while running pulsar on docker
URL: https://github.com/apache/pulsar/issues/6645#issuecomment-606852673
@anshul-dhyani For Docker on Windows make sure to configure it to use Linux containers.
> Caused by: org.rocksdb.RocksDBException: While fsync: a directory: Invalid argument
The rocksdb error seems to be indicating that you are not using a linux container.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #6645: Error while running pulsar
on docker
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #6645: Error while running pulsar on docker
URL: https://github.com/apache/pulsar/issues/6645#issuecomment-607532436
the first two commands are mounting a windows directory to the Linux container. It seems that we are not able to write the mounted windows directory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #6645: Error while running pulsar
on docker
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #6645: Error while running pulsar on docker
URL: https://github.com/apache/pulsar/issues/6645#issuecomment-606906819
@anshul-dhyani unfortunately, I don't have a windows machine to verify that. but it seems that you have successfully run the standalone using `docker run -d --restart=unless-stopped --name pulsar -p 6650:6650 -p 8081:8080 apachepulsar/pulsar:2.5.0 bin/pulsar standalone`, no? Does the error in the last command happen when you send a message?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #6645: Error while running pulsar
on docker
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #6645: Error while running pulsar on docker
URL: https://github.com/apache/pulsar/issues/6645#issuecomment-607380930
@anshul-dhyani interesting. Can you run the third command and add `-nfw -nss` after `bin/pulsar standalone`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] anshul-dhyani commented on issue #6645: Error while
running pulsar on docker
Posted by GitBox <gi...@apache.org>.
anshul-dhyani commented on issue #6645: Error while running pulsar on docker
URL: https://github.com/apache/pulsar/issues/6645#issuecomment-607091604
@sijie no, that error occurred just by running that command.
could not reach to a step where I can send message.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] anshul-dhyani commented on issue #6645: Error while
running pulsar on docker
Posted by GitBox <gi...@apache.org>.
anshul-dhyani commented on issue #6645: Error while running pulsar on docker
URL: https://github.com/apache/pulsar/issues/6645#issuecomment-606891509
Hi @sijie : Thanks for prompt update.
I think Docker is using Linux containers.
I am running it on Windows 10 home and on windows Home Docker runs in a VM with Linux OS.
Or I need to check something else ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services