You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/06/30 09:11:06 UTC
Slack digest for #general - 2020-06-30
2020-06-29 14:03:42 UTC - Hao Zhang: @Hao Zhang has joined the channel
----
2020-06-29 14:03:46 UTC - huzongtang: @huzongtang has joined the channel
----
2020-06-29 14:34:11 UTC - David Lanouette: Hi Sundar. Sorry for the late reply. Have you figured it out? If not, can you post a complete code snippet? (either directly here or in a gist.)
----
2020-06-29 17:46:59 UTC - Tamer: Are you able to create it from the `pulsar-admin` cli? Seems to me the path to the jar file is not correct.
If you can have it accessible of http that will be easier to test (e.g. deploy to S3 or MinIO with a preSigned URL)
Otherwise the jar has to be local on the broker/function worker instance
----
2020-06-29 19:24:17 UTC - Rahul Vashishth: > If a message isn’t tracked by any subscription, it doesn’t exist in any subscription backlog. In this case, Pulsar considers it ready for deletion.
@Ali Ahmed how would we retain the message on topic without subscription. I was testing kasque cluster and i could see message remain on topic which don't have any subscription.
----
2020-06-29 19:37:02 UTC - Ali Ahmed: set a retention policy based on size or time that will retain messages regardless of subscription
+1 : Rahul Vashishth
----
2020-06-29 20:28:33 UTC - David Lanouette: Hi @rwaweber . I'm late to the thread, and I'm not sure if you figured out your issue. But I noticed something and thought I'd point it out. In your output (on gist), it looks like the last message may not have been read. `"readPosition": "1438:30501"` seems like it still hasn't read that last message on ledger 1438. That would prevent that ledger from being removed.
----
2020-06-29 20:38:26 UTC - Sijie Guo: <http://pulsar.apache.org/docs/en/admin-api-overview/|http://pulsar.apache.org/docs/en/admin-api-overview/>
----
2020-06-29 20:38:48 UTC - Sijie Guo: You can read this section for more details about using PulsarAdmin api
----
2020-06-30 03:38:41 UTC - Joshua Eric: @Joshua Eric has joined the channel
----
2020-06-30 03:40:30 UTC - Ambud Sharma: Hi, not sure if this question has already been asked (simple slack search didn't show it).
Looking for pointers to how is keyshared subscription handled under the hood for partitioned topics when partition count changes? Does the broker consume from multiple ledgers to support this or the consumer ordering will change?
----
2020-06-30 03:43:53 UTC - Joshua Eric: Hello all, having trouble with schemas and pulsar functions in Python. I am producing a message with an Avro schema, then modifying it with a function, then consuming it. In the Python function, if I simply return `input` without modifying it, it works properly. I cannot figure out how to get `input` to be formatted as the schema type.
```producer = client.create_producer('<persistent://public/default/in>', schema=AvroSchema(Person))
producer.send(Person(phone='3135551234'))```
```bin/pulsar-admin functions localrun \
--tenant public \
--namespace default \
--py /Users/xxxxx/Sites/pulsar-python-functions/functions/test-python-library.zip \
--classname TestEtl.TestEtl \
--inputs <persistent://public/default/in> \
--output <persistent://public/default/out> \
--schema-type Person```
```class TestEtl(Function):
def process(self, input, context):
raise Exception(input)```
```consumer = client.subscribe(topic='<persistent://public/default/out>', subscription_name='ew', schema=AvroSchema(Person), consumer_type=pulsar.ConsumerType.Shared)```
The data I am after is in the `input` variable, but it is not of type `Person`. I assumed I should be able to do `input.phone = "<tel:+1231231234|+1231231234>"`
Thanks for any help that can be provided!
----
2020-06-30 03:46:40 UTC - Ambud Sharma: Checked <https://github.com/apache/pulsar/pull/4079/files> but it only shows 1 PersistentTopic <https://github.com/apache/pulsar/pull/4079/files#diff-5cc001bf2b64aad097762e99c5b78e4aR97>
----
2020-06-30 05:32:14 UTC - Matteo Merli: Key-Shared on a partitioned topic is basically a composition of key-shared on the individual partitions.
If you increase the number of partitions and now keys are hashing to different partitions, then ordering will be (briefly) broken.
----