You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/07/17 09:11:03 UTC
Slack digest for #general - 2019-07-17
2019-07-16 09:38:09 UTC - Rui Fu: @Rui Fu has joined the channel
----
2019-07-16 09:44:14 UTC - Vladimir Shchur: Hi, guys! Can someone clear up to me what is the difference between Close and Unsubscribe for Consumer? From code it looks like the only difference is that Close calls failPendingReceive while Unsubscribe doesn't, am I correct?
----
2019-07-16 09:50:59 UTC - Sijie Guo: close doesn’t delete subscription. unsuscribe will delete the subscription
----
2019-07-16 10:06:42 UTC - Vladimir Shchur: Sorry, what do you mean by "delete subscription"? Are you saying for example if there is an exclusive subscription mode and consumer was closed, no one can take it over? What is a purpose of close then, when should client call it?
----
2019-07-16 10:20:02 UTC - Sijie Guo: close is usually called when your application is shutting down. the you still need the subscription to consume from last position.
unsuscribe is called when your application doesn’t need the subscription anymore.
----
2019-07-16 10:24:08 UTC - Vladimir Shchur: I see, thank you!
----
2019-07-16 12:00:45 UTC - vikash: @jia zhai Thanks for your reply now i able to start standalone server with authentoication enabled,i have used below property
+1 : jia zhai
----
2019-07-16 12:01:24 UTC - jia zhai: welcome
----
2019-07-16 12:01:25 UTC - vikash: but when i am trying to authenticate with Websocket Cleint written in c# request.Headers.Add("Authorization", "Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ2aXN1ci11c2VyIn0.Y78f_iXB9vyslrrpfEvlyk_poSowClb4HtksUCU5wAA");
----
2019-07-16 12:01:35 UTC - vikash: i have always failed authentocate
----
2019-07-16 12:02:17 UTC - vikash: Pulsar Dashboard also not working even i have used like this
----
2019-07-16 12:02:18 UTC - vikash: docker run -d -p 80:80 -e SERVICE_URL=<http://192.168.1.248:8080> -e JWT_TOKEN=eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ2aXN1ci11c2VyIn0.Y78f_iXB9vyslrrpfEvlyk_poSowClb4HtksUCU5wAA pulsar
----
2019-07-16 12:02:25 UTC - vikash: may i know what i missed
----
2019-07-16 12:05:21 UTC - vikash: only this log i get
----
2019-07-16 12:05:21 UTC - vikash: va-v2.4.0" 19
17:18:35.062 [main] INFO org.apache.pulsar.PulsarStandalone - This operation requires super-user access
17:18:35.078 [pulsar-web-57-8] INFO org.eclipse.jetty.server.RequestLog - 192.168.1.248 - - [16/Jul/2019:17:18:35 +0530] "GET /admin/v2/tenants HTTP/1.1" 401 54 "-" "Pulsar-Java-v2.4.0" 10
17:18:35.079 [main] INFO org.apache.pulsar.PulsarStandalone - This operation requires super-user access
17:18:35.151 [pulsar-io-50-2] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/coordinate-participants] Rewind from 2414:0 to 2414:0
17:18:42.037 [SessionTracker] INFO org.apache.zookeeper.server.ZooKeeperServer - Expiring session 0x100012b5193000b, timeout of 30000ms exceeded
17:18:42.037 [ProcessThread(sid:0 cport:2181):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x100012b5193000b
17:18:55.347 [pulsar-web-57-3] INFO org.eclipse.jetty.server.RequestLog - 172.17.0.2 - - [16/Jul/2019:17:18:55 +0530] "GET /admin/v2/clusters HTTP/1.1" 200 14 "-" "python-requests/2.22.0" 11
17:18:55.380 [pulsar-web-57-7] INFO org.eclipse.jetty.server.RequestLog - 172.17.0.2 - - [16/Jul/2019:17:18:55 +0530] "GET /admin/v2/clusters/standalone HTTP/1.1" 401 54 "-" "python-requests/2.22.0" 5
17:19:04.967 [pulsar-web-57-1] INFO org.eclipse.jetty.server.RequestLog - 192.168.1.248 - - [16/Jul/2019:17:19:04 +0530] "GET /admin/v2/persistent/public/functions/coordinate/stats HTTP/1.1" 200 823 "-" "Pulsar-Java-v2.4.0" 17
----
2019-07-16 12:49:17 UTC - vikash: Actually its solved now i have missed superUserRoles= Parameter ,now able to authenticate
+1 : jia zhai
----
2019-07-16 13:52:03 UTC - ishara: Hello I am using pulsar standalone via the docker image and I am trying to modify the default configuration file. At the moment i have this in my docker-compose file : "command: /bin/bash -c
"/conf/apply-config-from-env.py /conf/standalone.conf
&&bin/pulsar standalone"" and I have my file linked to the conf in the docker with volumes. But when I start my standalone my changes are not applied at all ! Did anyone encounter this problem ?
----
2019-07-16 15:36:43 UTC - Grant Wu: Any idea what might cause this exception?
```
Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: 2019-07-16 15:15:37.349 INFO ConnectionPool:72 | Created connection for <pulsar://pulsar-broker.petuum-system:6650;> line: 1, column: 6]
```
----
2019-07-16 17:11:50 UTC - Matteo Merli: Looks like some JSON parsing issue. What service URL are you using? http:// or pulsar://
----
2019-07-16 17:12:05 UTC - Grant Wu: We’re using the Pulsar Python client
----
2019-07-16 17:12:11 UTC - Grant Wu: I think I may have a lead on this, I’ll let you know later
----
2019-07-16 19:09:13 UTC - Devin G. Bost: When brokers are restarted, do they re-download Jars for Pulsar functions when those functions were deployed with URLs for the Jars? We've noticed that if the Jars no longer exist in our Jar repo (due to them getting cleaned-up), our functions will start up in a dead state and need to be updated with a valid Jar URL.
----
2019-07-16 20:13:29 UTC - Grant Wu: Never mind, this is unrelated to Pulsar, this is a stupid Apache Livy issue that’s been around for a while that blocks Pulsar usage within Spark jobs in Livy
----
2019-07-16 22:10:36 UTC - Grant Wu: In case you’re curious at all, <https://issues.apache.org/jira/browse/LIVY-322?focusedCommentId=16886493&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16886493>
----
2019-07-16 22:18:48 UTC - Matteo Merli: Oh I see. I think we had a discussion some time back on routing the C++ logs back to python (as we do for Go)
----
2019-07-16 22:19:54 UTC - Grant Wu: Yes, I may pursue implementing that as a workaround
----
2019-07-16 22:20:01 UTC - Grant Wu: But this is really nonsense from Livy
----
2019-07-17 07:38:06 UTC - Olivier Chicha: Hello,
If I use a key_shared subscription,
let say: At the begining I have only one consumer on my key_shared subscription: consumer 1.
- Consumer1 receive message 1 (with hash 0.6)
- Message2 with hash 0.7 arrives on the broker but has not been yet sent to Consumer1
- Consumer2 is added to the subscription and gets the range 0.5..1
- Message3 arrive with hash 0.8
What will happend?
- Will the broker wait for Message1 to be acknowledged before sending message2 and Message 3 to consumer 2 ?
- Will message 3 be sent to to consumer2 and Message2 to consumer1 ?
In fact on our side we would like to be sure that consumer 2 will not receive message2 or Message3 before Message1 has been effectively acknowledged, so that we are sure that the message are effectively processed in the right order event if the consumer are not on the same process.
Is this something that is granted?
----
2019-07-17 07:45:31 UTC - Yong Zhang: I think it doesn’ t re-download Jars after brokers restarted. It’s downloaded when you create a function.
----
2019-07-17 07:55:51 UTC - Penghui Li: @Olivier Chicha key_shared subscription working principle can refer to the following :
```
1.Each consumer serves a fixed range of hash value
2.The whole range of hash value could be covered by all the consumers.
3.Once a consumer is removed, the left consumers could still serve the whole range.
Initializing with a fixed hash range, by default 2 << 5.
First consumer added, hash range looks like:
0 -> 65536(consumer-1)
Second consumer added, will find a biggest range to split:
0 -> 32768(consumer-2) -> 65536(consumer-1)
While a consumer removed, The range for this consumer will be taken over
by other consumer, consumer-2 removed:
0 -> 65536(consumer-1)
In this approach use skip list map to maintain the hash range and consumers.
Select consumer will return the ceiling key of message key hashcode % range size.
```
----
2019-07-17 08:09:22 UTC - Olivier Chicha: Thanks @Penghui Li for your answer.
This is effectivelly what I understood, but, as far as I can understand, does not answer to my question.
Here is what could be another version of my question (maybe simplier and clearer).
For two messages with the same key,
am I granted that the first message will be aknowledged before the second message is received by the new consumer (in case the key hash is allocated to a new consumer before the second message is received by the broker)?
----
2019-07-17 08:16:11 UTC - Penghui Li: if key hash of that message take over by the new consumer, the following messages with that key will dispatch to the new consumer.
----
2019-07-17 08:20:56 UTC - Penghui Li: Currently, the broker does not care whether the previous message was acknowledged. a new consumer takeover the hash range, the message will be distributed to this consumer.
----
2019-07-17 08:21:22 UTC - Sijie Guo: @Olivier Chicha You are not guaranteed at this moment. When a new consumer joins, the hash ring is changed. Broker doesn’t wait until any all the outstanding messages delivered to the first consumer acknowledged. It is actually hard to do so given the distributed environment.
I guess you are looking for a solution for a better control how keys are distributed to consumers. there is an ongoing efforts introducing sticky consumer for key_shared subscription. You can specify a key_range to consume message when you join the key_shared subscription. That might be the thing you are looking for.
----
2019-07-17 08:47:39 UTC - Olivier Chicha: I looked at it, but it does not provide a simple solution as well to my issue.
Our case is we have "micro services" that ignore each other and that provide service by consuming messages from a given topic.
The stickyness grant us the ordering per key, while it allows for load balancing of the messages across the different services.
Services can be added or removed by the orchestrator.
the key_shared subscription would respond to our case perfectly except when a new service is added, it is the only case where the message processing ordering is not granted.
If we use the feature you describe in (which I guess is the <https://github.com/apache/pulsar/issues/4169>), this requires that we implement a distributed resource allocation mechanism on top the key_shared subscription (which is not a small thing)
Note that if you don't grant the ordering of the processing of messages across consumer for a given key, it might also mean that we could have a race condition where the new client would receive message1 that the old client did not succeed to process (crashed) after having processed message2, => the ordering of the message is then not granted at consumer level. (unless I am missing something).
it would be really great for us if you could find a way to grant this ordering without having to implement on our side a distributed resource allocation mechanism.
Best regards,
----
2019-07-17 08:55:16 UTC - Gurgen Hovhannisyan: @Gurgen Hovhannisyan has joined the channel
----
2019-07-17 08:58:36 UTC - Gurgen Hovhannisyan: Hi
I have some issues, please help :
working with python 2 and pulsar 2.4.0 version, Ubuntu 18.04.2 LTS (Bionic Beaver) running in VirtualBox under Windows 10
The following schema doesn't work at all during create_producer, I want to pass array of maps (list of dictionaries), why it doesn't work ? please ?
class MySchema(Record):
a = String()
b = Long()
# array of dict, json data
data = Array(Map(String()))
this crashes during instance creation, even I didn't reach to send.
producer = client.create_producer(topic='mytopic15', schema=AvroSchema(MySchema))
----
2019-07-17 08:59:00 UTC - Sijie Guo: @Olivier Chicha I see what you are saying now. I think it can achieve by providing an option form the consumer. The option is used for telling the broker to wait until previous outstanding messages to be acked before reassigning a range. does that work for your use case?
----
2019-07-17 09:00:59 UTC - Olivier Chicha: Yes That would definitively do the trick as far as I understand.
----
2019-07-17 09:02:07 UTC - Sijie Guo: ^ @Penghui Li can you help looking into adding this option?
----
2019-07-17 09:04:11 UTC - Penghui Li: Ok
----
2019-07-17 09:07:24 UTC - Penghui Li: To ensure that messages are not processed twice by two consumers, if a new consumer connected, we should wait the old consumer have already ack all message dispatched
----
2019-07-17 09:07:38 UTC - Penghui Li: @Sijie Guo I understand it right?
----
2019-07-17 09:09:28 UTC - Sijie Guo: @Penghui Li correct. I think the idea is to “delay” splitting the hash ring until a given condition is met. The condition for the use case described by @Olivier Chicha is wait until the messages for a given ring are acknowledged and split and reassign the ring.
+1 : Penghui Li
----
2019-07-17 09:09:41 UTC - Penghui Li: If we use sticky consumer, messages are not assigned to two consumers, can this feature not be applied to this scenario?
----