You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/11/13 09:11:03 UTC
Slack digest for #general - 2019-11-13
2019-11-12 09:28:26 UTC - Sijie Guo: I see. current debezium source writes key/value schema to a topic. but jdbc doesn’t handle key/value schema right now.
Can you create a github issue? We can triage to add this feature. @tuteng and @Penghui Li can help with this.
+1 : Penghui Li
----
2019-11-12 12:11:17 UTC - sundar: Thanks for all your help, the pulsar admin is now working.
----
2019-11-12 12:14:18 UTC - sundar: But we hit another problem...we created an issue in github for the same which on completion, we feel the setup will be over.
<https://github.com/apache/pulsar/issues/5630>
Can you please check this out? Thanks in advance.
----
2019-11-12 15:15:37 UTC - Benjamin Egelund-Müller: Hello, does anyone have experience or a helpful tip on how to (reasonably efficiently) lookup the ~100 latest messages on a topic?
----
2019-11-12 16:17:52 UTC - Addison Higham: I am not aware of any plans, but I am sure there would be more interest in the future, the pulsar community is growing pretty quickly!
----
2019-11-12 16:52:49 UTC - Pedro Cardoso: Is there a timeout for creating a pulsar function? I get an HTTP 500 response intermittently when deploying a pulsar function
```
bin/pulsar-admin functions create \
--jar <path_to_executable.jar> \
--name rollingsum \
--classname RollingSum \
--inputs <non-persistent://public/default/transaction-input> \
--output <non-persistent://public/default/transaction-output>
HTTP 500 Internal Server Error
Reason: HTTP 500 Internal Server Error
```
----
2019-11-12 16:53:24 UTC - Pedro Cardoso: Is there a way to get more information on why the server failed?
----
2019-11-12 16:55:01 UTC - Matteo Merli: There could be 2 kind of approaches:
1. If you know the time, you can create a reader on that timestamp (or seek a consumer)
2. Keep a subscription with 100 unacked messages so that when you reconnect you can always replay them
----
2019-11-12 17:02:23 UTC - Pedro Cardoso: Found the following in the pulsar broker logs (this is a minikube-based cluster):
```
16:53:46.360 [DL-io-0] ERROR org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl - Unable to allocate memory
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 117440519, max: 134217728)
```
The quick fix is just to increase bookie memory right? Or is there something else I'm missing?
----
2019-11-12 17:05:49 UTC - Benjamin Egelund-Müller: With approach 1., do you mean keeping track of recently written message IDs in an external system?
----
2019-11-12 17:11:29 UTC - Dave: @Dave has joined the channel
----
2019-11-12 17:17:42 UTC - Benjamin Egelund-Müller: Naively, I’d hope I could look up the latest message ID from the broker, subtract N from that number to get the message ID of the Nth latest row, then create a reader on that message ID. I’m getting the feeling things are not that simple – but why exactly not? (I’m still learning how Pulsar works internally)
----
2019-11-12 17:39:03 UTC - Alvaro Ruiz Ramirez: @Alvaro Ruiz Ramirez has joined the channel
----
2019-11-12 17:51:21 UTC - Devin G. Bost: Sorry for my delay.
We’re using functions, as well as Java consumers and producers. We also have Go consumer/producers.
We haven’t unloaded the topics or namespaces because we can’t afford to have the data loss.
----
2019-11-12 17:51:46 UTC - Devin G. Bost: How can I help with this issue? This issue impacts us enough that I’d like to help with the fix if possible.
----
2019-11-12 18:56:49 UTC - Ryan Samo: Is there a way to get the count of messages that exist in a topic?
----
2019-11-12 18:57:39 UTC - Jerry Peng: @Ryan Samo not an exact count but you can get the number of BK entries or batches of messages
----
2019-11-12 18:58:04 UTC - Jerry Peng: If batching is turned off then its a one to one ratio with messages
----
2019-11-12 18:58:28 UTC - Ryan Samo: Ok, just looking for a useful way to tie out from producer to consumer
----
2019-11-12 18:59:00 UTC - Ryan Samo: So using the bk shell then you mean @Jerry Peng ?
----
2019-11-12 18:59:24 UTC - Jerry Peng: ./bin/puslar-admin topics stats-internal <TOPIC>
----
2019-11-12 18:59:50 UTC - Ryan Samo: Ok thanks!
----
2019-11-12 19:01:41 UTC - Retardust: It's not a messages actually. It's db replication state. Journal log
----
2019-11-12 19:07:28 UTC - Retardust: I have incomming topic, mapper java app and downstream topic. I need to save ordering guaranties and i wondering am i lose them on acking)
----
2019-11-12 20:04:35 UTC - Aafaq Ahmad: @Aafaq Ahmad has joined the channel
----
2019-11-12 21:25:09 UTC - Oleg Kozlov: Hi everyone, can someone explain what is the meaning of proxy_to_broker_url field in the CommandConnect message, and is it required if we connect to our brokers via Pulsar Proxy?
----
2019-11-12 21:28:12 UTC - Matteo Merli: Yes, it's required when the client goes through a proxy
----
2019-11-12 21:29:30 UTC - Matteo Merli: if there's a proxy, the Lookup will have the `ProxyThroughServiceUrl` field set to true
----
2019-11-12 21:30:37 UTC - Matteo Merli: at that point, the client will connect to the original service url (the proxy) and it will mark to proxy the connection to the broker returned in the lookup
----
2019-11-12 21:35:06 UTC - Oleg Kozlov: Hi, thanks for the response
----
2019-11-12 21:35:23 UTC - Oleg Kozlov: We are implementing our own client, using Proxy -> brokers connection, on Kubernetes
----
2019-11-12 21:35:53 UTC - Oleg Kozlov: Just to clarify - does the client have to do a Lookup before connecting to Proxy to figure out the broker url?
----
2019-11-12 21:40:02 UTC - Oleg Kozlov: Just to make sure we understand the flow:
1) Client connects to proxy
2) Sends CommandLookupTopic, and gets broker url in CommandLookupTopicResponse
3) Closes connection
4) Creates another connection with proxy_to_broker_url set to that broker URL ?
----
2019-11-12 21:45:38 UTC - Matteo Merli: > 3) Closes connection
Typically you keep the connections pooled, both for lookup and for creating producers
----
2019-11-12 21:46:36 UTC - Matteo Merli: Correct, that's more or less the sequence. The client doesn't know initially that it's going through the proxy
----
2019-11-12 21:49:54 UTC - Oleg Kozlov: Is there any way to make a specific connection from client to proxy be able to work for any topic on any of the brokers?
----
2019-11-12 21:50:12 UTC - Oleg Kozlov: not just topics owned by specific given broker?
----
2019-11-12 21:50:50 UTC - Matteo Merli: no, the proxy maintains a 1-1 connection between client and a specific broker
----
2019-11-12 21:51:41 UTC - Matteo Merli: this is done on purpose so that the proxy is very simple and efficient. After the initial connect/connected handshake, it will just become a dumb TCP proxy
----
2019-11-12 21:52:49 UTC - Oleg Kozlov: Chris, thank you for the response, this helps. We ended up using the last option (terminating SSL on pulsar proxy and going plain text proxy <-> broker), it's working now.
----
2019-11-12 21:54:47 UTC - Oleg Kozlov: got it, then, it's not clear what is the actual benefit of using the proxy, if we have a kubernetes load balancer in front any way, what does it give us?
----
2019-11-12 21:58:24 UTC - Matteo Merli: That you don't have to expose each broker IP to the client
----
2019-11-12 21:58:43 UTC - Matteo Merli: without proxy, the client will need to have direct TCP connectivity to brokers
----
2019-11-12 21:59:12 UTC - Matteo Merli: you cannot mask brokers behind a single VIP load balancer
----
2019-11-12 22:00:04 UTC - Matteo Merli: the proxy, being completely stateless, can be put behind a VIP
----
2019-11-12 22:01:27 UTC - Oleg Kozlov: i see, thanks
----
2019-11-12 22:01:47 UTC - Oleg Kozlov: this is very helpful, we'll update our flow to include the lookup
----
2019-11-12 22:14:27 UTC - Matteo Merli: :+1:
----
2019-11-12 22:45:26 UTC - Matthew Simoneau: @Matthew Simoneau has joined the channel
----
2019-11-13 01:08:53 UTC - Dennis Yung: @Naveen Kumar it is actually in Confluent's own repository. You need to add the following:
```
<repository>
<id>confluent</id>
<name>Confluent</name>
<url>http://packages.confluent.io/maven/</url>
<releases>
<enabled>true</enabled>
<checksumPolicy>fail</checksumPolicy>
</releases>
<snapshots>
<enabled>true</enabled>
<checksumPolicy>fail</checksumPolicy>
</snapshots>
</repository>
```
Also note that a confluent schema registry is required for the avro converter to work (you must supply a schema registry argument). The messages will be serialized as KeyValue<bytes, bytes> in pulsar, with the bytes being serialized avro records and required further deserialization
----
2019-11-13 04:31:25 UTC - Raja CSP: @Raja CSP has joined the channel
----
2019-11-13 07:46:13 UTC - Endre Karlson: hi guys, how is pulsar doing for exactly once semantics and does it differ in local vs geo replicated
----
2019-11-13 08:13:12 UTC - Jasper Li: I have faced this issue before and increased brokers' memory to solve it.
----