You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/02/23 09:11:03 UTC
Slack digest for #general - 2020-02-23
2020-02-22 11:28:35 UTC - Jon Bennett: <https://github.com/Homebrew/homebrew-core/pull/50580>
----
2020-02-22 14:20:02 UTC - Mikhail Veygman: @Devin G. Bost Yes I am talking about latency. If we are going to offload storage and retention I have content with 1 multiple points of failure and 2 some sort of mechanism to arbitrate what is coming from Pulsar currently with what is coming from history. I understand that Pulsar is set up for asynchronous IO but it doesn't help me in my situation because the order of operations on the Pulsar Broker/Bookkeeper at message receipt is: receive -> store -> republish. And I am trying to see if there is a way to paralellize store and republish operations.
----
2020-02-22 14:20:43 UTC - Mikhail Veygman: @Joe Francis Yes I know. I am tryng to see if there is a way that message storage pipeline can be parallized with publish pipeline
----
2020-02-22 14:26:54 UTC - Mikhail Veygman: Is there a way to configure a log level for the Pulsar Client is using when enabling SLF4J? I have used log4j and ch.qos.logback and in both cases the default log level is different from what I need. In case of log4j default log level is ERROR, which makes me miss message and in case of ch.qos.logback it is DEBUG which makes the output so verbose it is completely unhelpful. As far as I know there is no way to set the log level at SLF4J but there are on other implementations. So I am assuming that there is a way to set it onto the Pulsar Client.
----
2020-02-22 15:10:52 UTC - Fred: Is there any way to use pulsar built consumer through c++ , we can use in c# code
----
2020-02-22 15:34:14 UTC - Mikhail Veygman: Have you tried using pulsar C++ client? <https://pulsar.apache.org/docs/en/client-libraries-cpp/>
----
2020-02-22 16:57:49 UTC - Devin G. Bost: @Mikhail Veygman
> 1 multiple points of failure
>
How so?
> 2 some sort of mechanism to arbitrate what is coming from Pulsar currently with what is coming from history
>
What do you mean?
> the order of operations on the Pulsar Broker/Bookkeeper at message receipt is: receive -> store -> republish. And I am trying to see if there is a way to paralellize store and republish operations.
>
You're wanting to make a change to Pulsar core (as a Pulsar contributor) to increase Pulsar performance?
There's a performance optimization doc that I've been working on with @Penghui Li (who is the main author) and @Sijie Guo and others that might be helpful for you to read.
However, it's still an early draft. <https://docs.google.com/document/u/0/d/1BTLSMRCP13CCOWbGe_Nre82HuotWStM91nfe9RK46kI/mobilebasic|https://docs.google.com/document/u/0/d/1BTLSMRCP13CCOWbGe_Nre82HuotWStM91nfe9RK46kI/mobilebasic>
----
2020-02-22 17:34:57 UTC - Fred: my concern whether I can run c++ code in c# related services.
----
2020-02-22 17:35:45 UTC - Roman Popenov: Are you asking if there is a C# client?
----
2020-02-22 17:36:11 UTC - Fred: because we are using microservices in C# where we used to communicate one service to other service through proxy calls, because for pulsar there is no built in support for C# consumer and producer.
----
2020-02-22 17:36:29 UTC - Fred: yes C# client
----
2020-02-22 17:36:29 UTC - Roman Popenov: <https://pulsar.apache.org/docs/en/client-libraries/> - there is a 3rd party .NET client
----
2020-02-22 17:36:52 UTC - Roman Popenov: It’s not officially supported, but there is one: <https://github.com/fsharplang-ru/pulsar-client-dotnet>
----
2020-02-22 17:37:55 UTC - Fred: But you know we used that consumer which is very slow as compare to websocket one.
----
2020-02-22 17:38:37 UTC - Fred: Is there any idea behind where we can build c++ code in C#
----
2020-02-22 17:39:31 UTC - Roman Popenov: I would imagine one would need to write a wrapper
----
2020-02-22 17:40:55 UTC - Roman Popenov: I’ve never done it, but there seems to be blogposts on how to do it:
<https://dorodnic.com/blog/2014/12/10/calling-cpp-by-example/>
----
2020-02-22 18:19:37 UTC - Mikhail Veygman: Sorry I was referring to using external storage provider like ignite. Are you talking about Ignite fronting the connection to Pulsar?
And performance on non-persistent topics is sufficient but I am trying to have an option of persistent topic to behave like a non-persistent one in some respects, so that message can be forwarded to the Consumers ahead write if the consumer exists. The suggestions in the draft I've already implemented.
----
2020-02-22 18:20:09 UTC - Devin G. Bost: Apache Ignite has microsecond response times…
----
2020-02-22 18:20:46 UTC - Mikhail Veygman: It's not it was intentional. I am trying to be able to publish from many threads without having to deal with synchronization and locking.
----
2020-02-22 18:20:52 UTC - Devin G. Bost: > I am trying to have an option of persistent topic to behave like a non-persistent one in some respects, so that message can be forwarded to the Consumers ahead write if the consumer exists.
I don’t think this behavior is supported by Pulsar.
----
2020-02-22 18:20:58 UTC - Devin G. Bost: All writes must first hit the WAL.
----
2020-02-22 18:21:11 UTC - Devin G. Bost: There are many good reasons for this.
----
2020-02-22 18:21:31 UTC - Devin G. Bost: You don’t want to circumvent it because you could end up with an inconsistent system.
----
2020-02-22 18:21:41 UTC - Mikhail Veygman: Turned out to be unnecessary. Just needed to increase maximum number of IO threads per client.
----
2020-02-22 18:21:45 UTC - Devin G. Bost: Please read: <https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works>
----
2020-02-22 18:23:07 UTC - Devin G. Bost: e.g.
----
2020-02-22 18:26:30 UTC - Devin G. Bost: And seriously, it’s unnecessary. You might save maybe 2-3 milliseconds. If you need something that operates in microseconds instead of milliseconds, you’re using the wrong technology.
----
2020-02-22 18:29:21 UTC - Mikhail Veygman: Ok. Let me look further.
----
2020-02-22 18:29:36 UTC - Devin G. Bost: Good idea. :slightly_smiling_face:
----
2020-02-22 18:30:26 UTC - Mikhail Veygman: But Ignite won't work for what I need just like Redis for requirement of key-value. I need it to store a stream.
----
2020-02-22 18:30:55 UTC - Devin G. Bost: Then you might be best off just integrating with Bookkeeper directly.
----
2020-02-22 18:31:35 UTC - Devin G. Bost: If you have a suggestion for a change to the Pulsar architecture, please read the above link first (and seriously study it.)
----
2020-02-22 20:04:15 UTC - Mikhail Veygman: Will do.
----
2020-02-22 20:04:41 UTC - Devin G. Bost: Thanks!
----
2020-02-22 20:14:47 UTC - Joe Francis: > @Mikhail Veygman And I am trying to see if there is a way to paralellize store and republish operations.
----
2020-02-22 20:16:42 UTC - Joe Francis: Use non-persistent topic. Run a function to read the NP topic and publish to a persistent topic. You will get what you want, with all the pitfalls that come your use case. Phantom messages, loss of published messages etc
----
2020-02-22 22:57:23 UTC - Eugen: @Fred Have a look here <https://docs.microsoft.com/en-us/dotnet/standard/native-interop/pinvoke> P/Invoke is very easy to use, and may be sufficient. Using C++/CLI is more powerful, but also a little more complex getting started.
----
2020-02-23 03:04:26 UTC - Antti Kaikkonen: How to make a pulsar function read POJO inputs from an Avro encoded input topic? My function is working fine when the input topic 'user' is JSON encoded, but when the input topic is generated with `--schema-type 'avro'`, then I'm getting this error:
`java.lang.ClassCastException: class org.apache.avro.generic.GenericData$Record cannot be cast to class com.example.pulsartest.model.User (org.apache.avro.generic.GenericData$Record is in unnamed module of loader <http://java.net|java.net>.URLClassLoader @279f2327; com.example.pulsartest.model.User is in unnamed module of loader org.apache.pulsar.functions.utils.functioncache.FunctionClassLoaders$ParentFirstClassLoader @706377dd)` .
I have tried with `--custom-schema-inputs "{"user":"avro"}` which produces the same error message as without custom-schema-inputs. I also tried with `--custom-schema-inputs "{"user":"com.example.pulsartest.model.User"}"` which doesn't start the function with the this error: `Reason: com.example.pulsartest.model.User does not implement org.apache.pulsar.client.api.Schema`
----
2020-02-23 06:05:33 UTC - Antti Kaikkonen: I think <https://github.com/apache/pulsar/issues/5503> might be related. The function works if I set the input type to org.apache.pulsar.client.api.schema.GenericRecord, but then I have to write ugly code like `long age = (long) user.getField("age");` . Please let me know if there is a better solution, thanks!
----
2020-02-23 06:16:52 UTC - Sijie Guo: Yes I think there is the issue. It requires a fix.
ok : Antti Kaikkonen
----
2020-02-23 06:20:24 UTC - Eugen: I'm trying to bring up a pulsar cluster for the first time, following <http://pulsar.apache.org/docs/en/deploy-bare-metal/> . The 3-node ZK cluster seems to run fine, but only the first bookie node stays up. Bookie nodes 2 and 3 go down after 30 seconds (matching some timeout interval I see in the log). The problem is there are so many different errors in the logs, so for someone running bookies for the first time - and unsuccessfully, it's hard to see which error is responsible. Here are my config changes:
``` diff conf/bookkeeper.conf{.orig,}
51c51
< # listeningInterface=eth0
---
> listeningInterface=eth0
582c582
< zkServers=localhost:2181
---
> zkServers=172.31.47.33:2181,172.31.33.219:2181,172.31.41.15:2181
621c621
< # extraServerComponents=
---
> extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent```
and this are the first ERRORs in node 2:
```05:38:10.170 [main] ERROR org.apache.bookkeeper.bookie.Journal - Problems reading from data/bookkeeper/ledgers/current/lastMark (this is okay if it is the first time starting this bookie
05:38:10.750 [main] ERROR org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to initialize DNS Resolver <http://org.apache.bookkeeper.net|org.apache.bookkeeper.net>.ScriptBasedMapping, used default subnet resolver : java.lang.RuntimeException: No network topology script is found when using script based DNS resolver.
05:38:43.781 [main-EventThread] ERROR org.apache.curator.framework.imps.EnsembleTracker - Invalid config event received: {server.2=172.31.33.219:2888:3888:participant, server.1=172.31.47.33:2888:3888:participant, server.3=172.31.41.15:2888:3888:participant, version=0}
05:38:43.781 [main-EventThread] ERROR org.apache.curator.framework.imps.EnsembleTracker - Invalid config event received: {server.2=172.31.33.219:2888:3888:participant, server.1=172.31.47.33:2888:3888:participant, server.3=172.31.41.15:2888:3888:participant, version=0}
05:38:45.232 [bookie-io-1-2] ERROR org.apache.bookkeeper.proto.WriteEntryProcessorV3 - Ledger fenced while writing entry:0 to ledger:9
05:38:45.281 [bookie-io-1-2] ERROR org.apache.bookkeeper.proto.WriteEntryProcessorV3 - Ledger fenced while writing entry:0 to ledger:16
05:38:45.294 [bookie-io-1-2] ERROR org.apache.bookkeeper.proto.WriteEntryProcessorV3 - Ledger fenced while writing entry:0 to ledger:17
05:38:45.299 [bookie-io-1-2] ERROR org.apache.bookkeeper.proto.WriteEntryProcessorV3 - Ledger fenced while writing entry:0 to ledger:19
05:38:45.409 [main-EventThread] INFO org.apache.distributedlog.bk.SimpleLedgerAllocator - Ledger allocator /stream/storage/streams_000000000000000000_000000000000000001_000000000000000000/<default>/allocation moved to phase ERROR : version = 0.
05:38:45.409 [main-EventThread] ERROR org.apache.distributedlog.bk.SimpleLedgerAllocator - Fail mark ledger 44 as allocated under /stream/storage/streams_000000000000000000_000000000000000001_000000000000000000/<default>/allocation :
05:38:45.419 [main-EventThread] INFO org.apache.distributedlog.bk.SimpleLedgerAllocator - Ledger allocator /stream/storage/streams_000000000000000001_000000000000000001_000000000000000000/<default>/allocation moved to phase ERROR : version = 0.
05:38:45.421 [main-EventThread] ERROR org.apache.distributedlog.bk.SimpleLedgerAllocator - Fail mark ledger 51 as allocated under /stream/storage/streams_000000000000000001_000000000000000001_000000000000000000/<default>/allocation :
05:38:45.427 [main-EventThread] INFO org.apache.distributedlog.bk.SimpleLedgerAllocator - Ledger allocator /stream/storage/streams_000000000000000002_000000000000000001_000000000000000000/<default>/allocation moved to phase ERROR : version = 0.```
Where to look first for the problem?
----
2020-02-23 06:22:12 UTC - Devin G. Bost: We always ran our bare-metal on docker, so I don't think I'll be able to help much with this one.
----
2020-02-23 06:23:04 UTC - Devin G. Bost: It seems like you're running into network or related issues.
----
2020-02-23 06:23:30 UTC - Devin G. Bost: Maybe it's a port conflict?
----
2020-02-23 06:24:43 UTC - Devin G. Bost: If I was concerned about network communication issues, I'd startup Wireshark and assess what's happening.
----
2020-02-23 06:32:29 UTC - Antti Kaikkonen: Ok, thanks! Maybe I will try to write a SerDe class and use it with `--custom-serde-inputs` to manually deserialize the Avro encoded POJO.
----
2020-02-23 06:36:10 UTC - Sijie Guo: I don’t think that work.
----
2020-02-23 06:36:26 UTC - Sijie Guo: loop @CongBo and @Penghui Li to help fix the issue.
----
2020-02-23 06:37:12 UTC - Penghui Li: ok
----
2020-02-23 06:37:23 UTC - Sijie Guo: @Eugen: can you remove this `extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent`?
----
2020-02-23 06:37:56 UTC - Sijie Guo: If you are deploying a cluster first time, try to deploy a minimal cluster first.
----
2020-02-23 06:38:06 UTC - Eugen: is it _that_ experimental? ok, will give it a shot!
----
2020-02-23 06:39:47 UTC - Eugen: then just restart, or would I need to cleanup state first?
----
2020-02-23 06:40:46 UTC - Sijie Guo: It is hard to tell what are the errors from the log . A best way for a person to get family with the steps is to get a minimal cluster set up first.
> then just restart, or would I need to cleanup state first?
just restart
----
2020-02-23 06:42:34 UTC - Eugen: by 'minimal cluster' you mean 2 bookies?
----
2020-02-23 06:43:06 UTC - Eugen: anyways, all 3 processes stay up now, with function state disabled
----
2020-02-23 06:44:17 UTC - Sijie Guo: I mean minimal components and features.
• zk, bk, brokers
• disabled functions worker, disable state
• disable security and etc
----
2020-02-23 06:45:17 UTC - Eugen: and errors stopped flowing - thanks @Sijie Guo
----
2020-02-23 07:40:01 UTC - Eugen: not so sure any more it really was the function state, because in the meantime I fixed a zk setting; will try to re-enable function state again later; currently, bookie processes don't die, which is good
----