You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/04/01 09:11:04 UTC
Slack digest for #general - 2020-04-01
2020-03-31 09:49:42 UTC - Sijie Guo: For some requests, they are redirected.
----
2020-03-31 09:50:46 UTC - Sijie Guo: the retention policy is applied to each topic (partition).
----
2020-03-31 09:55:02 UTC - Sijie Guo: You don’t need to create a global zookeeper. You just need to add clusters via `pulsar-admin clusters create` to add clusters.
Here is one of the example to show how to do geo-replication without global zookeeper : <https://gist.github.com/sijie/79364497eaa349bf58d9fb760561f930>
+1 : Hiroyuki Yamada
----
2020-03-31 09:55:41 UTC - Poul Henriksen: Thanks!
----
2020-03-31 10:33:45 UTC - Prasanth: @Prasanth has joined the channel
----
2020-03-31 10:41:43 UTC - Esakkimuthu: @Esakkimuthu has joined the channel
----
2020-03-31 13:44:07 UTC - Kirill Kosenko: @Kirill Kosenko has joined the channel
----
2020-03-31 14:36:58 UTC - Rattanjot Singh: when running pulsar function in localrun for javanative it works fine.But for sdk it isn't working as expected.
``` FunctionConfig functionConfig = new FunctionConfig();
functionConfig.setName("test1");
functionConfig.setInputs(Collections.singleton("testTopic1,testTopic2"));
functionConfig.setClassName(VirtualTopicMultiple.class.getName());
functionConfig.setRuntime(FunctionConfig.Runtime.JAVA);
functionConfig.setLogTopic("logger");
LocalRunner localRunner = LocalRunner.builder().functionConfig(functionConfig).brokerServiceUrl("<pulsar://localhost:6650>").build();
localRunner.start(true);```
This works fine if its a javanative but for sdk it doesn't work. Do we need to add anything else for sdk?
Also if I run as create(as cluster) it works. I want to debug it so that i can gain more insight.
Thanks
----
2020-03-31 14:59:29 UTC - Kartik Gupta: @Sijie Guo Can you give some sample function name for the same?
Just to give a little more context, we are exposing pulsar in kubernetes with istio through pulsar-proxy service but are facing issues for the admin http calls.Hence want to seperate out the admin port as a seperate service without pulsar-proxy but this can result in redirection happening..so want to confirm and test it out
----
2020-03-31 15:06:07 UTC - Olav Groenaas Gjerde: Please dont use `@here`
+1 : Fabien LD, David Kjerrumgaard, Rattanjot Singh
----
2020-03-31 15:10:08 UTC - Ebere Abanonu: @Sijie Guo am waiting for the presto deployment guide. I need to test a lib with it. Thanks
----
2020-03-31 15:14:45 UTC - Chris: How does pulsar IO (specifically sources) work with schema? Does it attempt to decode the messages coming off the source with the schema of the destination topic?
----
2020-03-31 15:21:22 UTC - Ebere Abanonu: @Chris that is the role of pulsar schema registration. A connector looks up the schema for the topic it is consuming from to decode the message. Have I understood your question?
----
2020-03-31 15:21:50 UTC - Chris: You did, thanks
----
2020-03-31 15:34:03 UTC - Martina Oefelein: Still looking for some hints on what I’m doing wrong
----
2020-03-31 15:37:58 UTC - Ebere Abanonu: I encountered same error on docker for desktop....
----
2020-03-31 15:38:40 UTC - Ebere Abanonu: Probably it has to do with conf
----
2020-03-31 15:41:36 UTC - Evan Furman: The environment name can be anything, it’s just a way to organize your clusters. I usually opt to match this to the `cluster_name` in `broker.conf`.
The service url should be the http endpoint for your broker cluster. For us, it’s an ELB <http://pulsar-XXXXXXXX.elb.us-east-1.amazonaws.com:8080>
----
2020-03-31 15:42:32 UTC - Evan Furman: Becauses pulsar manager can manage multiple clusters, you have to manually add each as it’s own “Environment”
----
2020-03-31 15:44:29 UTC - Evan Furman: If you’re running the cluster on your local machine, the service url will likely be <http://localhost:8080>
----
2020-03-31 15:48:46 UTC - Martina Oefelein: I tried localhost:8080 already. Will try prefixing httP://
----
2020-03-31 16:00:17 UTC - Sijie Guo: if the request is required to be served by owner broker, it is redirected to the owner. things like unload, compact, and etc.
----
2020-03-31 16:01:46 UTC - Sijie Guo: what errors did you see when you say “for sdk it isn’t working as expected.“?
----
2020-03-31 16:06:04 UTC - Sijie Guo: since you are running pulsar standalone is a different container, so the url should be the address of your standalone container that the pulsar manager can access. `http://<standalone-container>:8080`
you can checkout <https://www.youtube.com/watch?v=wGgEx1M17O0&list=PLqRma1oIkcWhWAhKgImEeRiQi5vMlqTc-&index=4&t=65s|this video> after 57:00 minutes. I actually explained how pulsar manager connects to pulsar broker.
----
2020-03-31 16:07:11 UTC - Sijie Guo: functions / source connectors are “consumers” of an input topic. so they both will try to decode messages from the input topic based on the schema information associated with the topic.
----
2020-03-31 17:12:07 UTC - Martina Oefelein: Entering the name of the container didn’t work. But I got it working by using the IP address of the container in the form <http://172.17.0.2:8080/>
I got this IP address via:
`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <containername>`
Thanks!
----
2020-03-31 17:12:37 UTC - Martina Oefelein:
----
2020-03-31 17:15:20 UTC - Sijie Guo: for people who is interested in <#C0106RFKAHM|kop> #KafkaonPulsar, we are also live streaming our KoP webinar on Youtube - <https://www.youtube.com/watch?v=gL6hzRtij8M> You are welcome to join us!
+1 : Pierre-Yves Lebecq, David Kjerrumgaard
----
2020-03-31 17:19:43 UTC - Rattanjot Singh: The functionality is not working.
If i have the same function created(deployed in cluster)
It works.
Also in localrun when I produce the message in the input topics(here testTopic1) it isn't stopping at the debug points set . While if I set the debug points for the local run for javanative it stops at the debug point.
So I wanted to know if we had to set something extra for Localrunner for sdk.
----
2020-03-31 17:53:22 UTC - Srinivas115: @Srinivas115 has joined the channel
----
2020-03-31 17:58:37 UTC - Rattanjot Singh: The only difference is output is not being set. I am setting the output in the code. Is giving output necessary in functionConfig?
----
2020-03-31 18:20:50 UTC - David Kjerrumgaard: @Sijie Guo Great demo on KoP. Where can we find the MQTT interface for Pulsar?
----
2020-03-31 18:25:01 UTC - Tanner Nilsson: @Kelvin Sajere @Eugen did either of you get past this? I'm experiencing the same thing, tenants/namespaces/topics all load, but clusters doesn't and so I can't add new tenants because there is no cluster. This is while connecting to pulsar standalone.
----
2020-03-31 18:35:33 UTC - Rattanjot Singh: Also is possible that we can give a regex for input Topics for a pulsar function.
----
2020-03-31 18:43:01 UTC - Rattanjot Singh: Because let's say if list of input topic is huge. Giving a regex is better than the whole list
----
2020-03-31 18:51:59 UTC - Sijie Guo: @David Kjerrumgaard - We are still working on supporting AMQP and MQTT protocols on Pulsar. It is not yet released. but they are coming soon.
+1 : David Kjerrumgaard, Pierre Zemb, Roman Popenov
party-parrot : Pierre Zemb, Chris, Roman Popenov, Manjunath Ghargi
tada : Pierre Zemb, Roman Popenov
rocket : Pierre Zemb, Roman Popenov
----
2020-03-31 18:57:20 UTC - Sijie Guo: > it isn’t stopping at the debug points set
what are the debug points?
----
2020-03-31 19:07:54 UTC - Eric Simon: Do pulsar functions support the ability to schedule triggers? Similar to Lambda functions, you can trigger a function based on a cron sequence. I suppose you could hack something together with scheduling messages to be sent and then having that as an input for scheduling - is there any other way?
----
2020-03-31 19:18:27 UTC - David Kjerrumgaard: @Eric Simon Currently there is not, but that sounds like a feature they should have. A more pluggable scheduling mechanism would be nice :smiley:
+1 : Eric Simon
----
2020-03-31 19:19:27 UTC - Evan Furman:
----
2020-03-31 19:21:03 UTC - Evan Furman: This is what a scale up from 200 to 400 consumers looks like
----
2020-03-31 19:35:36 UTC - Evan Furman: @Penghui Li I know it is off hours for you — will try to be available later tonight (CDT). You asked for topic stats, anything else I can pull for you to get more insight?
----
2020-03-31 19:43:35 UTC - Eric Simon: Architecturally speaking - where would it make the most sense to embed that logic?
----
2020-03-31 19:49:35 UTC - Rattanjot Singh: debug points are in my function that i have written. it works if i give
```functionConfig.setOutput("output");```
Also is it possible that we can give a regex for input Topics for a pulsar function.
----
2020-03-31 19:53:11 UTC - Sijie Guo: > is it possible that we can give a regex for input Topics for a pulsar function.
Yes you can do that. You can use `--topics-pattern`
----
2020-03-31 19:53:21 UTC - Kamal Gupta: @Kamal Gupta has joined the channel
----
2020-03-31 19:54:33 UTC - Sijie Guo: > it works if i give
I see. You need to set the output topic to make sure the results are produced to the output topic.
----
2020-03-31 20:01:45 UTC - Evan Furman:
----
2020-03-31 20:15:58 UTC - Manjunath Ghargi: @Manjunath Ghargi has joined the channel
----
2020-03-31 20:28:29 UTC - Manjunath Ghargi: <!here>: I'm looking for "*ActiveMQ -> Camel -> Pulsar*" integration, where messages produced in ActiveMQ should be picked by camel and copied over to Pulsar topics.
I see camel has Pulsar components supported from this doc -> <https://camel.apache.org/components/latest/pulsar-component.html>
Can someone share the reference implementation or any examples to connect Camel with Pulsar?
----
2020-03-31 20:52:19 UTC - Sijie Guo: @Manjunath Ghargi Can you avoid using `@ here` ?
----
2020-03-31 20:55:37 UTC - Sijie Guo: <https://medium.com/thg-tech-blog/a-feather-in-their-caps-4c4083cf9f46> this is blog is about Camel with Pulsar.
You can reference Camel’s example here how to use camel-pulsar component : <https://github.com/apache/camel-examples/tree/master/examples/camel-example-spring-pulsar>
----
2020-03-31 20:58:29 UTC - David Kjerrumgaard: I was researching that but got a bit sidetracked. I never did find the proper place, and was just going to create a PIP in order to start the discussion on this subject.
----
2020-03-31 20:59:02 UTC - Eric Simon: Awesome. Once you create that do mind sharing it here? I will add my thoughts as well.
+1 : David Kjerrumgaard
----
2020-03-31 21:26:27 UTC - Kirill Merkushev: exactly what I searched for
----
2020-03-31 21:43:36 UTC - Sijie Guo: cool
----
2020-03-31 21:44:45 UTC - Sijie Guo: It seems that the high latency are from those individual deleted messages. It seems like some of the messages are not acked correctly. @Penghui Li when you wake up, it would be nice to checkout why those messages are not acked correctly.
----
2020-03-31 21:45:27 UTC - Sijie Guo: have you created the environment yet?
----
2020-03-31 21:46:07 UTC - Sijie Guo: cool. yeah. you need to provide a service url that the pulsar manager is able to access.
----
2020-03-31 21:47:17 UTC - Tanner Nilsson: yes, the environment is created. I can connect and view tenants, namespaces, and topics in the standalone environment, but when I click on the clusters it just spins and I get the same 500 error in the browser console. I cannot create tenants in pulsar-manager when connected to puslar standalone because it can't get the cluster. But if I use the pulsar-admin cli, I can list clusters and create tenants.
----
2020-03-31 21:52:34 UTC - Sijie Guo: I see. What is the output when you tried to list the clusters?
----
2020-03-31 21:53:25 UTC - Sijie Guo: I am guessing that you have a wrong service url in the cluster configuration that pulsar manager is not able to connect to.
----
2020-03-31 21:53:50 UTC - Sijie Guo: Can you run `bin/pulsar-admin cluster list` and `bin/pulsar-admin cluster get <cluster>` ?
----
2020-03-31 22:17:45 UTC - David Kjerrumgaard: Is there a way to define message properties inside a Typed Pulsar Function ?
----
2020-03-31 22:25:01 UTC - David Kjerrumgaard: It seems really hack-ish to do this....
----
2020-04-01 00:43:29 UTC - Hiroyuki Yamada: @Penghui Li @Sijie Guo
> Version 2.5.1 is cut, I think we can fix it at version 2.5.2 or 2.6.0
Is there any rough schedule about the release ?
Sorry it sounds pushing but I’m not and just want to help (testing mostly due to the limited knowledge about Pulsar) since the out of order messages in Key_Shared is really a critical thing to many of users including us.
----
2020-04-01 00:44:02 UTC - Sijie Guo: @David Kjerrumgaard that’s the current approach. I think Addison was proposing an change to return a record that is allowing people to customize and control the acknowledgement behavior.
+1 : David Kjerrumgaard
----
2020-04-01 00:45:36 UTC - Sijie Guo: @Hiroyuki Yamada: @Penghui Li is on it.
We are cutting 2.5.1 today. If needed, we can cut 2.5.2 release next week once Penghui figure out the issue.
----
2020-04-01 00:46:54 UTC - Hiroyuki Yamada: Thank you ! Good to know.
I’ll work on testing once the new versions are released.
----
2020-04-01 00:48:42 UTC - Addison Higham: I do need to get back to that...
----
2020-04-01 00:49:26 UTC - Penghui Li: Ok, let me try to fix it.
man-bowing : Hiroyuki Yamada
+1 : Hiroyuki Yamada
----
2020-04-01 00:51:46 UTC - Evan Furman: @Tim Corbett and I can be available if you wanna jump on a zoom or something. Lmk. Otherwise can pull anything else you need.
+1 : Tim Corbett
----
2020-04-01 01:03:23 UTC - Penghui Li: @Evan Furman Could you please give me topic stats, not internal stats? I want to know if the unacked messages are in a consumer or multiple consumers. Or are there any messages the consumer can't process succeed?
----
2020-04-01 01:04:00 UTC - Tim Corbett: We're new to this, so if you have a specific command line to run that's probably easiest for us
----
2020-04-01 01:04:13 UTC - Tim Corbett: Or at least which option will best describe what you're asking for
----
2020-04-01 01:06:05 UTC - Penghui Li: I have test with 200+ consumers yesterday, the consumer ack all receive messages and continuous testing half hour, there are not too many individual acks.
----
2020-04-01 01:06:40 UTC - Penghui Li: $bin/pulsar-admin topics stats <your-topic-name>
----
2020-04-01 01:15:16 UTC - Tim Corbett: On it!
----
2020-04-01 01:25:06 UTC - Evan Furman:
----
2020-04-01 01:34:25 UTC - Penghui Li: Thanks, you can use $bin/pulsar-admin topics peek-messages -s mp-pulsar-consumer <persistent://public/default/mp-explicit-partitioned-0> to peek the earliest message that not acked.
----
2020-04-01 01:35:13 UTC - Tim Corbett: Hmm, the above log was during a backlog apparently, it may not be representative fully
----
2020-04-01 01:35:27 UTC - Tim Corbett: That was just a steady state consumer load of 300
----
2020-04-01 01:37:08 UTC - Evan Furman:
----
2020-04-01 01:38:27 UTC - Penghui Li: I want to confirm that this message is a message that it can be consumed normally?
----
2020-04-01 01:38:54 UTC - Tim Corbett: Yes. All the messages we send are the same currently
----
2020-04-01 02:15:43 UTC - Penghui Li: Could you please check if the broker log contains
```rewind because no available consumer found for key```
----
2020-04-01 02:16:00 UTC - Evan Furman: sure, looking
----
2020-04-01 02:43:26 UTC - Penghui Li: According to your pulsar-perf logs, the consumer is almost stuck right? Is the consumer can resume it-self?
----
2020-04-01 02:45:18 UTC - xue: Instantiate in function constructor
ApplicationContext applicationContext = new ClassPathXmlApplicationContext("classpath:dubbo-consumer.xml");
The dubbo-consumer.xml file in the jar package cannot be read. The file not found error is displayed,
But if it is initialized in the process function, it can be instantiated successfully。
----
2020-04-01 02:47:26 UTC - Penghui Li: I gradually expanded from the initial 200 consumers to 600 consumers, still can't reproduce the problem.
----
2020-04-01 02:50:30 UTC - xue:
----
2020-04-01 02:55:30 UTC - Evan Furman: nothing unfortunately :confused:
```[ec2-user@ip-10-3-22-162 log]$ sudo grep "rewind" messages
[ec2-user@ip-10-3-22-162 log]$```
----
2020-04-01 03:03:10 UTC - Penghui Li: Ok, thanks
----
2020-04-01 03:47:41 UTC - Rattanjot Singh: @Justin Grimes
----
2020-04-01 04:32:42 UTC - Penghui Li: @Evan Furman How many brokers and bookies in your cluster and do you install the Grafana dashboard? I suspect that the broker reading the messages from the bookie. From your logs, the rate of consumption has dropped a lot since one point in time.
----
2020-04-01 09:02:36 UTC - Kartik Gupta: thanks @Sijie Guo
----