You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/09/11 09:11:03 UTC
Slack digest for #general - 2019-09-11
2019-09-10 10:33:08 UTC - vikash: @Vladimir Shchur is nuget also support .net framework
----
2019-09-10 10:33:09 UTC - vikash: <https://www.nuget.org/packages/Pulsar.Client/0.1.4>
----
2019-09-10 10:33:35 UTC - vikash: i am getting below Error
----
2019-09-10 10:33:41 UTC - vikash:
----
2019-09-10 10:34:13 UTC - vikash: but above code working fine in .net core
----
2019-09-10 10:34:42 UTC - vikash: just query because our c# application is in .net framework
----
2019-09-10 11:03:25 UTC - Vladimir Shchur: Hi! Thanks for report, will check if it possible to fix and update you
----
2019-09-10 13:20:36 UTC - Vladimir Shchur: @vikash Two binding redirects are needed to make it work:
`<dependentAssembly>`
` <assemblyIdentity name="System.Memory" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />`
` <bindingRedirect oldVersion="0.0.0.0-4.0.1.1" newVersion="4.0.1.0" />`
` </dependentAssembly>`
` <dependentAssembly>`
` <assemblyIdentity name="System.Threading.Tasks.Extensions" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />`
` <bindingRedirect oldVersion="0.0.0.0-4.2.0.1" newVersion="4.2.0.0" />`
` </dependentAssembly>`
----
2019-09-10 13:28:28 UTC - vikash: @Vladimir Shchur Thanks ,yes it works for me after changes above config
+1 : Vladimir Shchur
----
2019-09-10 14:31:36 UTC - Sijie Guo: Not just this one. I think the function worker needs some additional configurations as well.
----
2019-09-10 14:31:44 UTC - Sijie Guo: I think the documentation about this part is missing.
----
2019-09-10 14:31:54 UTC - Sijie Guo: We will try to enhance this part
----
2019-09-10 14:48:43 UTC - Kirill Merkushev: Is there any way for java api to know that the ledgerId is closed (or about to be closed) so that with manual seek we don’t get an exception for the last message in the closed ledger and for the current one we don’t go one position earlier than we actually have? I’m talking about this part <https://github.com/apache/pulsar/blob/master/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2729>
and the `seekAsync(MessageId)` method
----
2019-09-10 16:01:26 UTC - Kirill Merkushev: found the `org.apache.pulsar.client.impl.ConsumerImpl#getLastMessageIdAsync` but that’s a package-private. Is that safe to assume that ledgerId within the messageId it is the latest/active ledger?
----
2019-09-10 18:48:35 UTC - Tarek Shaar: It is recommended to co-locate ZooKeeper with the Pulsar brokers on the same VMs (possibly with Book Keeper too)? We process roughly 1M messages per business day.
----
2019-09-10 19:51:33 UTC - Sijie Guo: I think that method will be exposed in 2.5.
----
2019-09-10 19:51:51 UTC - Sijie Guo: I wouldn’t suggest making that assumption
----
2019-09-10 19:52:07 UTC - Sijie Guo: LedgerId is kind of internal to Pulsar so far
----
2019-09-10 19:56:38 UTC - Sijie Guo: What was the exception you are hitting? Can you show a code snippet?
----
2019-09-10 19:58:59 UTC - Sijie Guo: That’s probably fine. Although make sure you don’t share the disks for zk and bk. Also keep in mind with collocation, you might loose flexibility of scaling individual layers
----
2019-09-10 20:01:07 UTC - Tarek Shaar: I am trying to get the last sequence ID from my producer but I am always getting -1. I am using 2.4.1 on both client and broker sides
----
2019-09-10 20:04:34 UTC - Tarek Shaar: On a separate note, is the sequence as far as writing to disk is book keeper node stores the messages in the journal disk then acks back to the producer? Or does another disk write happen in ZK Journal as well before acking back to the producer?
----
2019-09-10 20:22:33 UTC - Sijie Guo: > is the sequence as far as writing to disk is book keeper node stores the messages in the journal disk then acks back to the producer?
yes
> Or does another disk write happen in ZK Journal as well before acking back to the producer?
no. pulsar didn’t talk to zk that much. onlly about the metadata part.
----
2019-09-10 20:24:53 UTC - Sijie Guo: can you explain more with a code example?
----
2019-09-10 20:26:13 UTC - Tarek Shaar: Thanks but I got a little confused when you said keep ZK disks and BK disks separate. So at which point does ZK use disk? I see there is a log file in the data directory that's relatively large. Do you have any documentation on when Pulsar uses ZK?
----
2019-09-10 20:27:38 UTC - Tarek Shaar: Here is the producer code.. Producer<byte[]> producer = client.newProducer().enableBatching(true).producerName("my-dev").batchingMaxMessages(2000).blockIfQueueFull(true).compressionType(CompressionType.LZ4).topic(topic).create();
----
2019-09-10 20:27:57 UTC - Tarek Shaar: Then I set the sequence ID on the producer like this
----
2019-09-10 20:27:59 UTC - Tarek Shaar: producer.newMessage().sequenceId(100).key("key1").value("test".getBytes()).sendAsync();
----
2019-09-10 20:28:11 UTC - Tarek Shaar: Then I shut down the producer and start it again
----
2019-09-10 20:28:29 UTC - Tarek Shaar: and get the Sequence ID like this
----
2019-09-10 20:28:30 UTC - Tarek Shaar: long last = producer.getLastSequenceId();
----
2019-09-10 20:28:38 UTC - Tarek Shaar: But is is always -1
----
2019-09-10 20:30:39 UTC - Ming: @Ming has joined the channel
----
2019-09-10 20:39:39 UTC - Sandeep Kotagiri: @Sandeep Kotagiri has joined the channel
----
2019-09-10 20:57:28 UTC - Sandeep Kotagiri: I was working with functions and state in a Kubernetes environment. Encountred the same issue as above. With some older posts from @Sijie Guo I was able to get this working. In Kubernetes environment, I had to set 'extraServerComponents' parameter in bookkeeper configuration file and the 'stateStorageServiceUrl' parameter in the functions_worker.yml configuration file. Kubernetes HELM chart has lower JVM settings that would also need to be bumped up to get this working properly.
----
2019-09-10 20:59:03 UTC - Sijie Guo: Did you enable message deduplication at the broker or the namespace? If not, Pulsar won’t record the information.
----
2019-09-10 21:01:14 UTC - Sandeep Kotagiri: @Sijie Guo I have a question on the very topic. I have three bookies running in my Kubernetes cluster. At the moment, I am configuring 'stateStorageServiceUrl' to be <bk://pulsar-bookkeeper-0:4181/>. Here 'pulsar-bookkeeper-0' is the first bookie in my statefulset. How do we configure to utilize the reminder of the bookies (pulsar-bookkeeper-1, pulsar-bookkeeper-2) for storing state?
----
2019-09-10 21:16:02 UTC - Tarek Shaar: Ok I'll try and let you know if it fails. Thanks
----
2019-09-10 22:19:41 UTC - BigSam: @BigSam has joined the channel
----
2019-09-10 23:09:39 UTC - Addison Higham: oh hrm... so for pulsar functions on a geo-replication cluster, I probably want to use a different function namespace for each region? it looks like my function is getting sent randomly around my cluster
----
2019-09-10 23:09:47 UTC - Addison Higham: and then unable to find the jar I uploaded
----
2019-09-10 23:09:49 UTC - Addison Higham: is that correct?
----
2019-09-10 23:09:55 UTC - Matteo Merli: Correct
----
2019-09-10 23:10:29 UTC - Addison Higham: makes sense, but that was surprising :stuck_out_tongue: good thing worker IDs have the hostname in it otherwise I would have been really confused
----
2019-09-10 23:50:33 UTC - Ali Ahmed: <https://twitter.com/Lanayx/status/1171508406072401926>
----
2019-09-11 07:15:25 UTC - Yuvaraj Loganathan: @Tarek Shaar ZK persists every write to disk before acknowledging it. Also as a good practice i would recommend to launch separate VM for zookeeper so that you will have better & superior fault tolerance level.
----
2019-09-11 08:35:49 UTC - divyasree: @Sijie Guo I am trying out topic compaction in pulsar. Is there any configuration need to be enabled for topic compaction or we can make the producer to produce data with specific key and read from the consumer with "readCompacted(true)?
----
2019-09-11 08:36:53 UTC - divyasree: If changes from the client side is enough for the compaction to work, when will the compaction get triggered. What is the default threshold for it to trigger?
----
2019-09-11 08:57:26 UTC - Mahesh: @Sijie Guo After configuring as suggested by Sandeep, I get this error now
01:50:35.109 [public/default/word-count-0] WARN org.apache.pulsar.functions.instance.JavaInstanceRunnable - Encountered internal server on opening table 'word-count', re-attempt in 100 milliseconds : Encountered internal server exception on stream 1026 : code = INTERNAL_SERVER_ERROR
----
2019-09-11 08:57:52 UTC - Mahesh: Can you please share the full list of config entries to be added to use table service in functions
----
2019-09-11 09:10:39 UTC - Vladimir Ontikov: Hi,
How many cores should be on a machine running Pulsar?
----