You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/05/31 09:11:02 UTC
Slack digest for #general - 2018-05-31
2018-05-30 09:11:50 UTC - Guillaume LECROC: How long is the incubation ?
----
2018-05-30 09:18:20 UTC - Sijie Guo: @Guillaume LECROC we will be talking about graduating to TLP (top-level-project). it should happen soon.
----
2018-05-30 09:18:32 UTC - Sijie Guo: stay tuned
----
2018-05-30 11:38:53 UTC - Matti-Pekka Laaksonen: I just upgraded to Pulsar 2.0.0-rc1, and I'm having problems with the message event time. I have a process which creates messages and sets their event time, but when I receive the message in the next step, the event time is always 0, no matter what the original value was. This happens with the new TypedMessageBuilder and the 1.22.0 style MessageBuilder where I first create a message and the use the producer to send it. Has anyone else experienced this?
----
2018-05-30 13:01:34 UTC - jia zhai: @Matti-Pekka Laaksonen, Is it reproduce-able?
----
2018-05-30 13:02:58 UTC - jia zhai: Would you please help open a github issue with more details and reproduce steps for it?
----
2018-05-30 13:04:50 UTC - Matti-Pekka Laaksonen: @jia zhai I'll try to find time for it this friday, but from what I can tell, the eventTime never gets persisted to the message, and it always gets set to 0 somewhere along the way
----
2018-05-30 13:07:00 UTC - Matti-Pekka Laaksonen: Actually, I think it happens on this row: <https://github.com/apache/incubator-pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java#L186>
----
2018-05-30 13:07:01 UTC - jia zhai: // the timestamp that this event occurs. it is typically set by applications.
// if this field is omitted, `publish_time` can be used for the purpose of `event_time`.
optional uint64 event_time = 12 [default = 0];
----
2018-05-30 13:07:17 UTC - jia zhai: It seems the default value is 0.
----
2018-05-30 13:08:00 UTC - jia zhai: Yes, if not set, it’s value is 0
----
2018-05-30 13:08:15 UTC - Matti-Pekka Laaksonen: Yep, this explains the zero
----
2018-05-30 18:22:33 UTC - Matteo Merli: @Matti-Pekka Laaksonen Looking into it
----
2018-05-30 18:24:04 UTC - Ali Ahmed: also as part of 2.0 there is also a lof4j2 appender for pulsar
<https://mvnrepository.com/artifact/org.apache.pulsar/pulsar-log4j2-appender>
----
2018-05-30 18:25:02 UTC - Matteo Merli: @Idan The docs should be correct at this point. Please check out <http://pulsar.apache.org/docs/latest/clients/Java/#Producers-k7rrin>
----
2018-05-30 19:51:20 UTC - Josh West: where does the 2.0 Broker look inside Zookeeper to find the address(es) of the Bookkeeper bookies? what is the zookeeper key?
----
2018-05-30 19:55:44 UTC - Matteo Merli: all bookies register themselves under `/ledgers/available/...`
----
2018-05-30 19:57:07 UTC - Josh West: thanks @Matteo Merli
----
2018-05-30 19:57:17 UTC - Josh West: my bookeeper ledgers registered with the wrong name
----
2018-05-30 19:57:27 UTC - Josh West: this is all just testing ... so i could blow up the cluster and start over
----
2018-05-30 19:57:33 UTC - Josh West: but i'm curious how one would fix for real in this case
----
2018-05-30 19:58:11 UTC - Matteo Merli: there are 2 options for registration:
1. with Pod IP
2. with hostname
----
2018-05-30 19:58:44 UTC - Matteo Merli: the important part is that bookies should advertise a “stable” address
----
2018-05-30 19:59:06 UTC - Matteo Merli: the address is how BookKeeper client know where to find the data
----
2018-05-30 19:59:13 UTC - Josh West: nod nod
----
2018-05-30 19:59:18 UTC - Josh West: i have `useHostNameAsBookieID: "true"` set
----
2018-05-30 19:59:39 UTC - Josh West: ok i figured it out
----
2018-05-30 19:59:47 UTC - Josh West: i changed the service name for my stateful sets
----
2018-05-30 19:59:49 UTC - Matteo Merli: Yes, that works well with StatefulSets and PersistentVolumes
----
2018-05-30 19:59:51 UTC - Josh West: because i'm writing a Helm Chart
----
2018-05-30 19:59:58 UTC - Josh West: so i want Release.Name in it
----
2018-05-30 20:00:09 UTC - Josh West: but i left the StatefulSet configuration as `serviceName: bookkeeper`
----
2018-05-30 20:00:26 UTC - Matteo Merli: I see
----
2018-05-30 20:00:41 UTC - Josh West: so the bookies registered as `testing-pulsar-bookkeeper-0.bookkeeper.cluster.local` (and so on) instead of `testing-pulsar-bookkeeper-0.testing-pulsar-bookkeeper.cluster.local`
----
2018-05-30 20:01:22 UTC - Josh West: ^ Release Name == `testing`
----
2018-05-30 20:01:27 UTC - Josh West: ^ Chart Name == `pulsar`
----
2018-05-30 20:01:35 UTC - Josh West: aka the first two prefix fields in those hostnames
----
2018-05-30 20:01:37 UTC - Josh West: ugly, i know...
----
2018-05-30 20:01:59 UTC - Josh West: ok @Matteo Merli so lets say i go restart those statefulsets with the correct servicename this time
----
2018-05-30 20:02:23 UTC - Josh West: are there some `pulsar-admin` commands i should use to remove the old ledgers, which were never alive to begin with, from the system?
----
2018-05-30 20:02:31 UTC - Josh West: or delete them manually with `zookeeper-client` ?
----
2018-05-30 20:06:07 UTC - Matteo Merli: were the topics created?
----
2018-05-30 20:07:53 UTC - Josh West: what do you mean by topics?
----
2018-05-30 20:09:25 UTC - Matteo Merli: > should use to remove the old ledgers
By ledgers do you mean some data was already created or you just want to clear up ZK from the previous metadata?
----
2018-05-30 20:09:46 UTC - Josh West: the latter -- clear up the ZK
----
2018-05-30 20:10:27 UTC - Matteo Merli: I see, so is ZK not part of Helm?
----
2018-05-30 20:11:00 UTC - Josh West: it is
----
2018-05-30 20:11:07 UTC - Josh West: i'm just getting things right and made a mess here heh
----
2018-05-30 20:11:11 UTC - Josh West: want to clean up things -- just to learn
----
2018-05-30 20:11:14 UTC - Josh West: i could blow everything away
----
2018-05-30 20:11:20 UTC - Matteo Merli: there shouldn’t be the need to clear it up though
----
2018-05-30 20:11:33 UTC - Josh West: oh good
----
2018-05-30 20:11:36 UTC - Josh West: learns some more
----
2018-05-30 20:12:02 UTC - Matteo Merli: but in case, you can just use `bin/pulsar zookeeper-shell` and then do `rmr /ledgers`
----
2018-05-30 20:12:44 UTC - Josh West: oh nice
----
2018-05-30 20:13:26 UTC - Josh West: oh wow
----
2018-05-30 20:13:31 UTC - Josh West: the old ones disappeared automatically
----
2018-05-30 20:13:32 UTC - Josh West: very nice
----
2018-05-30 20:15:04 UTC - Matteo Merli: you mean the bookies under `/ledgers/available` ? The registration z-nodes are ephemeral, they disappear when the associated ZK session expires
----
2018-05-30 20:16:20 UTC - Josh West: yup
----
2018-05-30 20:16:24 UTC - Josh West: just learned that
----
2018-05-30 20:16:26 UTC - Josh West: wahoo
----
2018-05-30 22:58:32 UTC - William Fry: Anyone know the memory requirements for running pulsar?
----
2018-05-30 22:59:16 UTC - William Fry: Tried using the ansible playbook for AWS using a t2.medium and keep getting the following:
```May 30 22:51:10 ip-10-0-1-208 systemd: Starting Pulsar Broker...
May 30 22:51:10 ip-10-0-1-208 pulsar: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000001c0000000, 25769803776, 0) failed; error='Cannot allocate memory' (errno=12)
May 30 22:51:10 ip-10-0-1-208 pulsar: #
May 30 22:51:10 ip-10-0-1-208 pulsar: # There is insufficient memory for the Java Runtime Environment to continue.
May 30 22:51:10 ip-10-0-1-208 pulsar: # Native memory allocation (mmap) failed to map 25769803776 bytes for committing reserved memory.
May 30 22:51:10 ip-10-0-1-208 pulsar: # An error report file with more information is saved as:```
----
2018-05-30 23:00:22 UTC - Matteo Merli: It seems it’s configured to use 24 GB of mem, you can edit `conf/pulsar_env.sh` to lower that
----
2018-05-30 23:00:32 UTC - Matteo Merli: are you deploying with Ansible scripts?
----
2018-05-30 23:00:35 UTC - William Fry: Yep
----
2018-05-30 23:00:51 UTC - William Fry: Did a bit of customization to use a bastion host so it’s in a private subnet, but missed that config
----
2018-05-30 23:01:10 UTC - Matteo Merli: ok, these settings were based on a bigger VM, but you can certainly use a lot less mem
----
2018-05-30 23:01:24 UTC - William Fry: yeah I’m just doing a POC so didn’t want to blow through my budget :slightly_smiling_face:
----
2018-05-30 23:01:48 UTC - Matteo Merli: you can down to 512MB if there is not a huge number of topics
----
2018-05-30 23:01:53 UTC - William Fry: where is `max_heap_memory` defined?
----
2018-05-30 23:02:03 UTC - William Fry: cool, yeah just doing < 5 for now
----
2018-05-30 23:03:32 UTC - William Fry: ah, got it. it’s in the `deploy-pulsar.yml`
----
2018-05-30 23:03:33 UTC - William Fry: cheers
----
2018-05-30 23:04:15 UTC - Matteo Merli: ok, was searching as well :slightly_smiling_face:
----
2018-05-30 23:08:11 UTC - William Fry: have you seen this stacktrace?
----
2018-05-30 23:08:13 UTC - William Fry: ```May 30 23:04:56 ip-10-0-1-208 pulsar: 2018-05-30 23:04:56,143 - INFO - [main:PulsarBrokerStarter@84] - Using bookie configuration file /opt/pulsar/conf/bookkeeper.conf
May 30 23:04:56 ip-10-0-1-208 pulsar: 2018-05-30 23:04:56,266 - INFO - [main-SendThread(10.0.1.19:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server 10.0.1.19/10.0.1.19:2181. Will not attempt to authenticate using SASL (unknown error)
May 30 23:04:56 ip-10-0-1-208 pulsar: 2018-05-30 23:04:56,274 - INFO - [main-SendThread(10.0.1.19:2181):ClientCnxn$SendThread@876] - Socket connection established to 10.0.1.19/10.0.1.19:2181, initiating session
May 30 23:04:56 ip-10-0-1-208 pulsar: 2018-05-30 23:04:56,309 - INFO - [main-SendThread(10.0.1.19:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server 10.0.1.19/10.0.1.19:2181, sessionid = 0x163b34be1970000, negotiated timeout = 30000
May 30 23:04:56 ip-10-0-1-208 pulsar: 2018-05-30 23:04:56,326 - ERROR - [main:Bookie@420] - BookKeeper metadata doesn't exist in zookeeper. Has the cluster been initialized? Try running bin/bookkeeper shell metaformat
May 30 23:04:56 ip-10-0-1-208 pulsar: 2018-05-30 23:04:56,330 - ERROR - [main:Bookie@369] - Couldn't access cookie in zookeeper
May 30 23:04:56 ip-10-0-1-208 pulsar: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for BookKeeper metadata
May 30 23:04:56 ip-10-0-1-208 pulsar: at org.apache.bookkeeper.bookie.Bookie.getInstanceId(Bookie.java:423)
May 30 23:04:56 ip-10-0-1-208 pulsar: at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:316)
May 30 23:04:56 ip-10-0-1-208 pulsar: at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:481)
May 30 23:04:56 ip-10-0-1-208 pulsar: at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:313)
May 30 23:04:56 ip-10-0-1-208 pulsar: at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:116)
May 30 23:04:56 ip-10-0-1-208 pulsar: at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.<init>(PulsarBrokerStarter.java:162)
May 30 23:04:56 ip-10-0-1-208 pulsar: at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:233)
May 30 23:04:56 ip-10-0-1-208 pulsar: 2018-05-30 23:04:56,335 - ERROR - [main:PulsarBrokerStarter@227] - Uncaught exception in thread main: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for BookKeeper metadata```
----
2018-05-30 23:08:38 UTC - William Fry: looks like everything is fine in the Zookeeper instance logs
----
2018-05-30 23:09:03 UTC - William Fry: but the Pulsar instance is struggling, doesn’t look like either port is open according to `netstat`
----
2018-05-30 23:11:10 UTC - William Fry: Looks like it’s something to do with this: <https://github.com/apache/incubator-pulsar/issues/19> cc: @Matteo Merli
----
2018-05-30 23:17:06 UTC - William Fry: It may be because the ansible playbook still has `pulsar_version` as `1.22.0-incubating`
----
2018-05-30 23:25:48 UTC - William Fry: When updating to the latest release, I get the following stacktrace:
```
May 30 23:19:38 ip-10-0-1-208 pulsar: 23:19:38.834 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Malformed configuration file: /opt/pulsar/conf/bookkeeper.conf
May 30 23:19:38 ip-10-0-1-208 pulsar: org.apache.commons.configuration.ConfigurationException: Entry log file size should not be larger than 1073741824
May 30 23:19:38 ip-10-0-1-208 pulsar: at org.apache.bookkeeper.conf.ServerConfiguration.validate(ServerConfiguration.java:2259) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0]
May 30 23:19:38 ip-10-0-1-208 pulsar: at org.apache.pulsar.PulsarBrokerStarter.readBookieConfFile(PulsarBrokerStarter.java:93) [org.apache.pulsar-pulsar-broker-2.0.0-rc1-incubating.jar:2.0.0-rc1-incubating]
May 30 23:19:38 ip-10-0-1-208 pulsar: at org.apache.pulsar.PulsarBrokerStarter.access$2(PulsarBrokerStarter.java:89) [org.apache.pulsar-pulsar-broker-2.0.0-rc1-incubating.jar:2.0.0-rc1-incubating]
May 30 23:19:38 ip-10-0-1-208 pulsar: at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.<init>(PulsarBrokerStarter.java:186) [org.apache.pulsar-pulsar-broker-2.0.0-rc1-incubating.jar:2.0.0-rc1-incubating]
May 30 23:19:38 ip-10-0-1-208 pulsar: at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:274) [org.apache.pulsar-pulsar-broker-2.0.0-rc1-incubating.jar:2.0.0-rc1-incubating]
May 30 23:19:38 ip-10-0-1-208 pulsar: 23:19:38.844 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Uncaught exception in thread main: Malformed configuration file
May 30 23:19:38 ip-10-0-1-208 pulsar: java.lang.IllegalArgumentException: Malformed configuration file
May 30 23:19:38 ip-10-0-1-208 pulsar: at org.apache.pulsar.PulsarBrokerStarter.readBookieConfFile(PulsarBrokerStarter.java:100) ~[org.apache.pulsar-pulsar-broker-2.0.0-rc1-incubating.jar:2.0.0-rc1-incubating]
May 30 23:19:38 ip-10-0-1-208 pulsar: at org.apache.pulsar.PulsarBrokerStarter.access$2(PulsarBrokerStarter.java:89) ~[org.apache.pulsar-pulsar-broker-2.0.0-rc1-incubating.jar:2.0.0-rc1-incubating]
May 30 23:19:38 ip-10-0-1-208 pulsar: at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.<init>(PulsarBrokerStarter.java:186) ~[org.apache.pulsar-pulsar-broker-2.0.0-rc1-incubating.jar:2.0.0-rc1-incubating]
May 30 23:19:38 ip-10-0-1-208 pulsar: at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:274) ~[org.apache.pulsar-pulsar-broker-2.0.0-rc1-incubating.jar:2.0.0-rc1-incubating]
```
----
2018-05-30 23:26:03 UTC - William Fry: Happy to put together a PR once I figure out relevant changes to ansible playbook
----
2018-05-30 23:53:57 UTC - Matteo Merli: @William Fry Yes, there are a couple of changes to BookKeeper conf file template for 2.0
----
2018-05-30 23:54:22 UTC - William Fry: Yep, updated the conf files with the latest release and added in the ansible variables where appropriate
----
2018-05-30 23:54:52 UTC - Matteo Merli: Nice, is it starting up now?
----
2018-05-30 23:54:58 UTC - William Fry: Down to the last error which is the following:
```
ay 30 23:43:32 ip-10-0-1-208 pulsar: 23:43:32.002 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Uncaught exception in thread main: failed to allocate 134217728 byte(s) of direct memory (used: 536870912, max: 536870912)
May 30 23:43:32 ip-10-0-1-208 pulsar: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 134217728 byte(s) of direct memory (used: 536870912, max: 536870912)
May 30 23:43:32 ip-10-0-1-208 pulsar: at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
May 30 23:43:32 ip-10-0-1-208 pulsar: at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
May 30 23:43:32 ip-10-0-1-208 pulsar: at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
May 30 23:43:32 ip-10-0-1-208 pulsar: at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:186) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
May 30 23:43:32 ip-10-0-1-208 pulsar: at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:68) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
May 30 23:43:32 ip-10-0-1-208 pulsar: at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
May 30 23:43:32 ip-10-0-1-208 pulsar: at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledByteBufAllocator.java:181) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]
```
----
2018-05-30 23:55:17 UTC - William Fry: Imagine it has something to do with the disk mounts
----
2018-05-30 23:56:12 UTC - Matteo Merli: No, that’s around direct memory usage
----
2018-05-30 23:56:39 UTC - Matteo Merli: it exceeds the 512 MB for direct memory
----
2018-05-30 23:57:19 UTC - William Fry: Direct memory is disk space?
----
2018-05-30 23:57:21 UTC - William Fry: or also RAM?
----
2018-05-30 23:57:26 UTC - Matteo Merli: that’s RAM
----
2018-05-30 23:58:01 UTC - Matteo Merli: JVM has 2 classes of memory, heap which is object allocated and GCed
----
2018-05-30 23:58:27 UTC - Matteo Merli: and direct memory which is typically used for network and disk IO and it’s pooled
----
2018-05-30 23:58:52 UTC - William Fry: Ah gotcha, that’s helpful to know. I checked the machine and I have more than enough memory on the machine
----
2018-05-30 23:59:01 UTC - William Fry: Do I need to increase the 512MB limit then?
----
2018-05-30 23:59:02 UTC - William Fry: to 1g?
----
2018-05-30 23:59:59 UTC - Matteo Merli: yes, try that for `max_direct_memory`
----
2018-05-31 00:09:57 UTC - William Fry: We’re in business! :balloon:
----
2018-05-31 00:10:10 UTC - William Fry: Should I consolidate all the changes to `terraform-ansible` and throw up a PR?
----
2018-05-31 00:10:28 UTC - William Fry: @Matteo Merli
----
2018-05-31 00:10:40 UTC - Matteo Merli: :ok_hand::skin-tone-2:
----
2018-05-31 00:10:41 UTC - William Fry: // are there contributing guidelines?
----
2018-05-31 00:10:56 UTC - William Fry: I can just make a PR and document the changes
----
2018-05-31 00:12:13 UTC - Matteo Merli: Yes, the PR it's all that's needed. With Apache there are no CLA clauses to sign for contributors, just for committers
----
2018-05-31 00:35:09 UTC - William Fry: Sweet: <https://github.com/apache/incubator-pulsar/pull/1873>
100 : Sijie Guo
----