You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/06/23 09:11:02 UTC
Slack digest for #general - 2018-06-23
2018-06-22 10:22:07 UTC - Tomer Lev: Hi when trying to start a bookie I'm getting this error: ```Exception in thread "main" io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 1073741824 byte(s) of direct memory (used: 0, max: 536870912)```
----
2018-06-22 10:23:00 UTC - Tomer Lev: I tried to play with pulsar_env.conf and bkenv.conf memory parameters but nothing ... still getting this error
----
2018-06-22 10:31:50 UTC - Ivan Kelly: The machine only has 500 megs?
----
2018-06-22 10:31:55 UTC - Ivan Kelly: aws tiny?
----
2018-06-22 10:35:19 UTC - Tomer Lev: m5.large
----
2018-06-22 10:44:08 UTC - Tomer Lev: this is the current configs: ```BOOKIE_MEM=" -Xms2g -Xmx4g -XX:MaxDirectMemorySize=512m"```
----
2018-06-22 10:49:24 UTC - Ivan Kelly: It's trying to allocate a gig
----
2018-06-22 10:50:14 UTC - Tomer Lev: this is pulsar env: ```PULSAR_MEM=" -Xms4g -Xmx6g -XX:MaxDirectMemorySize=3g"```
----
2018-06-22 10:50:30 UTC - Tomer Lev: I believe it's respect bkenv and not pulsar_env ...
----
2018-06-22 10:51:29 UTC - Ivan Kelly: but the error is that you're trying allocate 1g, and there's only 512M available max, which agrees with BOOKIE_MEM
----
2018-06-22 10:52:28 UTC - Tomer Lev: changed to 2g
----
2018-06-22 10:52:32 UTC - Tomer Lev: ```Jun 22 10:52:12 <http://dev-pulsar-1.dev.bluerbn.com|dev-pulsar-1.dev.bluerbn.com> pulsar[4456]: Exception in thread "main" io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 1073741824 byte(s) of direct memory (used: 3221225472, max: 3221225472)```
----
2018-06-22 10:53:03 UTC - Ivan Kelly: @Karthikeyan Palanivelu posted an update to <https://github.com/apache/incubator-pulsar/issues/1991>, basically, you should be able to use TLS between the proxy and broker for publish and consume, but you'll have to block admin operations at the proxy
----
2018-06-22 10:53:26 UTC - Tomer Lev: ```BOOKIE_MEM=" -Xms2g -Xmx4g -XX:MaxDirectMemorySize=2g"```
----
2018-06-22 10:54:33 UTC - Tomer Lev: ```Exception in thread "main" io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 1073741824 byte(s) of direct memory (used: 2147483648, max: 2147483648)```
----
2018-06-22 10:54:48 UTC - Ivan Kelly: what are you running on this machine?
----
2018-06-22 10:54:57 UTC - Tomer Lev: right now nothing
----
2018-06-22 10:54:57 UTC - Ivan Kelly: pulsar and bookie?
----
2018-06-22 10:55:24 UTC - Tomer Lev: in the end yes I want to run them both on same machine
----
2018-06-22 10:55:39 UTC - Tomer Lev: but now I'm trying to start only bookeeper
----
2018-06-22 10:55:57 UTC - Ivan Kelly: nothing else is running? what does free -m say?
----
2018-06-22 10:56:19 UTC - Tomer Lev: ``` total used free shared buff/cache available
Mem: 7687 83 7402 8 201 7357
Swap: 0 0 0
```
----
2018-06-22 10:56:50 UTC - Ivan Kelly: and what is the stack trace from that Error?
----
2018-06-22 10:57:13 UTC - Tomer Lev: ```Exception in thread "main" io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 1073741824 byte(s) of direct memory (used: 2147483648, max: 2147483648)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640)
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594)
at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:186)
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:68)
at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledByteBufAllocator.java:181)
at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:91)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
at io.netty.buffer.Unpooled.directBuffer(Unpooled.java:145)
at org.apache.bookkeeper.bookie.storage.ldb.WriteCache.<init>(WriteCache.java:105)
at org.apache.bookkeeper.bookie.storage.ldb.WriteCache.<init>(WriteCache.java:85)
at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>(SingleDirectoryDbLedgerStorage.java:155)
at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:126)
at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:112)
at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:721)
at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:115)
at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:96)
at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:42)
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:299)
at org.apache.bookkeeper.server.Main.doMain(Main.java:218)
at org.apache.bookkeeper.server.Main.main(Main.java:200)
at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:252)```
----
2018-06-22 10:59:40 UTC - Ivan Kelly: how many ledger directories have you configured?
----
2018-06-22 11:00:32 UTC - Ivan Kelly: have you changed any other configuration?
----
2018-06-22 11:04:29 UTC - Tomer Lev: I used the documentation ansible installtion configuration file
----
2018-06-22 11:14:44 UTC - Tomer Lev: I tried to run with those configurations and it works: <https://raw.githubusercontent.com/apache/incubator-pulsar/master/conf/bookkeeper.conf>
----
2018-06-22 11:14:55 UTC - Tomer Lev: thanks for helping
----
2018-06-22 11:19:45 UTC - Ivan Kelly: the ansible linked from the website? I would guess that's built for very beefy machines. I've taken a note to check hardware specs for that
----
2018-06-22 11:20:59 UTC - Tomer Lev: OK I got it now thanks for helping :slightly_smiling_face: :+1:
----
2018-06-22 11:21:15 UTC - Ivan Kelly: actually, it already says to use i3.xlarge
----
2018-06-22 11:21:35 UTC - Ivan Kelly: which setup instructions are you using?
----
2018-06-22 11:23:01 UTC - Tomer Lev: sort of mixing of bare metal and aws because I had to customize this
----
2018-06-22 11:23:29 UTC - Tomer Lev: but the configurations was taken (by mistake) from ansible repository ...
----
2018-06-22 11:27:36 UTC - Ivan Kelly: ah, i3.xlarge are huge, so the conf is upped to match it
----
2018-06-22 16:29:14 UTC - Karthik Palanivelu: Hi All, Have anyone tried AWS ECS to stand up Pulsar? If so can you please share some of your learnings? I prefer to go with Containers.
----
2018-06-22 16:40:43 UTC - chris: i was able to deploy pulsar on eks. I used this tool to create the cluster <https://github.com/weaveworks/eksctl>
----
2018-06-22 16:45:23 UTC - chris: The only tricky part was there are no storage classes created by default so you need to manually create one before deploying resources that need storage volumes. <https://docs.aws.amazon.com/eks/latest/userguide/storage-classes.html>
----
2018-06-22 16:51:16 UTC - Karthik Palanivelu: Thanks @chris but EKS is not approved within my company
----
2018-06-22 18:17:58 UTC - Daniel Ferreira Jorge: Thank you @Matteo Merli!
----
2018-06-22 18:43:36 UTC - Daniel Ferreira Jorge: When we want to start a pulsar component there are commands such as "bin/pulsar bookie". Are there any commands to stop a component? Something like "bin/pulsar bookie stop"
----
2018-06-22 18:57:47 UTC - Ali Ahmed: not directly but you can use ```pulsar-daemon start bookie``` and ```pulsar-daemon start bookie```
----
2018-06-22 19:01:01 UTC - Ali Ahmed: @Karthikeyan Palanivelu I haven’t tried it but as I remember ECS supports docker compose specifications so you can try pulsar’s docker compose template
----
2018-06-22 19:02:25 UTC - Daniel Ferreira Jorge: @Ali Ahmed Are there any differences running pulsar components in the background as a daemon?
----
2018-06-22 19:03:43 UTC - Ali Ahmed: don’t think so except the logs will go to a file not std out. same can be done for brokers and zookeeper
----
2018-06-22 19:04:29 UTC - Daniel Ferreira Jorge: ok thank you!
----
2018-06-22 19:06:33 UTC - Karthik Palanivelu: Sure Thanks @Ali Ahmed
----
2018-06-22 20:44:17 UTC - Josh West: ```root@pulsar-zk-0:/pulsar# curl <http://localhost:8000/metrics>
# HELP jvm_threads_current Current thread count of a JVM
# TYPE jvm_threads_current gauge
jvm_threads_current 32.0
# HELP jvm_threads_daemon Daemon thread count of a JVM
# TYPE jvm_threads_daemon gauge
jvm_threads_daemon 13.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
jvm_threads_peak 33.0
# HELP jvm_threads_started_total Started thread count of a JVM
# TYPE jvm_threads_started_total counter
jvm_threads_started_total 259.0
# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
# TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
# HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
# TYPE jvm_threads_deadlocked_monitor gauge
jvm_threads_deadlocked_monitor 0.0
```
----
2018-06-22 20:44:37 UTC - Josh West: should more stats than these be being presented by the Prometheus stats exporter built into Pulsar's Zookeeper?
----
2018-06-22 20:45:10 UTC - Sijie Guo: I think there would be some zookeeper stats.
----
2018-06-22 20:45:17 UTC - Sijie Guo: but I need to double check
----
2018-06-22 21:11:30 UTC - Charles Hutchinson: We have been struggling to get our broker service to start. We have verified that the broker server can talk to the local cluster nodes on 2181 and the global cluster nodes on 2184. The logs show `Socket connection established to each node followed by this.
```:ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect```
----
2018-06-22 21:12:15 UTC - Charles Hutchinson: It eventually throws the following error and give up.
```ERROR [main:GlobalZooKeeperCache@74] - Failed to establish global zookeeper session: null
java.util.concurrent.TimeoutException```
----
2018-06-22 21:12:59 UTC - Charles Hutchinson: I'm not sure what we are missing. We have an odd number of nodes in the clusters and have the observer peertype set for the observers.
----
2018-06-22 21:13:35 UTC - Sijie Guo: can you telnet the zookeeper : telnet <zk-node> 2181
----
2018-06-22 21:13:41 UTC - Sijie Guo: and then type ‘urok’
----
2018-06-22 21:13:56 UTC - Sijie Guo: to see if it respond something back?
----
2018-06-22 21:14:06 UTC - Charles Hutchinson: We used netcat but yes.
----
2018-06-22 21:14:31 UTC - Sijie Guo: so the broker node can talk to both?
----
2018-06-22 21:14:37 UTC - Sijie Guo: I mean local and global?
----
2018-06-22 21:14:42 UTC - Charles Hutchinson: Yes
----
2018-06-22 21:15:11 UTC - Sijie Guo: are you using the same hostname as the zookeeperServers in the configuration?
----
2018-06-22 21:15:12 UTC - Charles Hutchinson: We can observe the communications with tcpdump on the zookeeper nodes.
----
2018-06-22 21:15:21 UTC - Charles Hutchinson: Yes
----
2018-06-22 21:15:23 UTC - Sijie Guo: oh i see
----
2018-06-22 21:16:39 UTC - Sijie Guo: on broker host, can you run - bin/pulsar zookeeper-shell -server localhost:2181 “ls /”
----
2018-06-22 21:16:50 UTC - Charles Hutchinson: We get the same error from the brokers at all datacenters.
----
2018-06-22 21:16:51 UTC - Sijie Guo: change localhost:2181 to your zookeeper servers
----
2018-06-22 21:18:20 UTC - Charles Hutchinson: Yes. ```ls /
[namespace, admin, zookeeper, ledgers, managed-ledgers]```
----
2018-06-22 21:19:41 UTC - Sijie Guo: is it the zookeeper server you used in the command “zookeeper-shell” same as the one configured in broker.conf zookeeperServers=
----
2018-06-22 21:19:56 UTC - Charles Hutchinson: It is
----
2018-06-22 21:20:14 UTC - Charles Hutchinson: It is the first of the 5
----
2018-06-22 21:20:44 UTC - Sijie Guo: can you DM me your line “zookeeperServer=“? just trying to see if any special characters there
----
2018-06-23 05:45:21 UTC - Yuvaraj Loganathan: Hello Pulsar Team , Thank you for the wonderful next gen Streaming/Messaging System. As an org we getting started with Streams of Data. There is an heavy debate going on between pulsar vs kafka. We are get tilted towards pulsar :slightly_smiling_face: (Because of architectural semantics). Will Keep posted on our progress here.
+1 : Sijie Guo
fire : Sijie Guo
----
2018-06-23 06:33:21 UTC - Ali Ahmed: @Yuvaraj Loganathan sure let us know if you publish any blogs
----
2018-06-23 06:35:34 UTC - Yuvaraj Loganathan: Sure :thumbsup:
----