You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/08/20 09:11:03 UTC
Slack digest for #general - 2019-08-20
2019-08-19 09:39:06 UTC - Sijie Guo: @jia zhai Can you comment?
----
2019-08-19 10:22:28 UTC - Mahesh: hi
One question regarding failover consumer in pulsar. Lets say I have three consumer consuming from a topic t1.
Consumer1 - Active consumer
Consumer2 - Failover consumer
Consumer3 - Failover Consumer
All these 3 consumers have created a fail over subscription s1. If consumer1, goes down, consumer2 becomes the active consumer. What if the last consumer to go down is the active consumer ? How does pulsar handles this ?
----
2019-08-19 10:41:33 UTC - Sijie Guo: You mean there is no consumer active?
----
2019-08-19 11:03:16 UTC - Martin Svensson: Thanks for the suggestion but although I maybe get a better initial burst when reading it seems like a higher buffer at best gives similar performance but makes the broker struggle more.
----
2019-08-19 11:09:51 UTC - Alexandre DUVAL: @Sijie Guo
----
2019-08-19 11:11:08 UTC - Alexandre DUVAL: Because I deleted one and recreated with the same naming with schéma but the previous schéma seems to be used. Is it possible ?
----
2019-08-19 11:24:16 UTC - Sijie Guo: @Alexandre DUVAL did you delete the schema?
----
2019-08-19 11:25:36 UTC - Alexandre DUVAL: IIRC yes, so in any case the schema has to be deleted even if we delete thé topic? Will ensure to delete thé schéma.
----
2019-08-19 11:48:24 UTC - Sijie Guo: if you don’t specify delete schema option, it will not delete the schema
----
2019-08-19 11:49:32 UTC - Sijie Guo: yes. there is a tradeoff here. we can consider an option to by pass broker to read data directly from tiered storage.
----
2019-08-19 11:49:57 UTC - Mahesh: @Sijie Guo yes,
----
2019-08-19 11:51:56 UTC - Sijie Guo: it means that there consumers in this subscription. pulsar will keep the messages until the new consumers connect.
----
2019-08-19 12:01:38 UTC - Alexandre DUVAL: Oh there is an option, okay, thx!
----
2019-08-19 12:02:17 UTC - Alexandre DUVAL: Why this behavior, I mean a schema is always attached to a topic, so a topic deletion should automatically removes the schema, no?
----
2019-08-19 12:09:40 UTC - balaji: I am trying to use protobuf with pulsar and documentation in official pulsar page is returning 404. <http://pulsar.apache.org/docs/en/admin-api-schemas/concepts-schema-registry.md#supported-schema-formats>
----
2019-08-19 12:15:15 UTC - Mahesh: Does pulsar provide any notification mechanism for such a case, where once a consumer thats part of the failover subscription goes down, it should notify other consumers.
----
2019-08-19 12:15:40 UTC - Mahesh: that are part of the same failover subscription
----
2019-08-19 12:19:30 UTC - Sijie Guo: There is a consumer state change listener when active consumer changes
----
2019-08-19 12:22:12 UTC - Sijie Guo: Because schema was added later so bunch of tools doesn’t delete the schema when deleting the topic. So we ended up adding the flag
----
2019-08-19 12:54:44 UTC - Mahesh: yes, I tried that already. But it notifies other standby consumers when active goes down and not the other way around
----
2019-08-19 13:35:25 UTC - Martin Svensson: that sounds like a really good option. I imagine that the logic would exist between the broker and the client where the client will receive access tokens to the tiered storage on a ledger / topic basis.
----
2019-08-19 15:15:17 UTC - Ryan Samo: Hey guys, is it possible to load client side certificates (pem) into the client without having the certs physically on the machine? Like using a stream or even hard coding the certificates as strings inside the code and then load them from there? All examples I see of using certs pulls them in from the local file system.
----
2019-08-19 15:18:30 UTC - Ming Fang: are you concerned about physical security of the cert? You can try storing it in tmpfs, e.g. /dev/shm in Ubuntu
----
2019-08-19 15:36:28 UTC - Ryan Samo: @Ming Fang More like a hashi-vault use case where you call a separate system to obtain your cert and you then have it already loaded in memory. At this point you have to write it to a file locally so that Pulsar can read it back in. I’d like to just pass it straight to the Pulsar client
----
2019-08-19 16:07:45 UTC - Alexandre DUVAL: Do you have example of how to use Avro schema in Context.publish in pulsar function?
----
2019-08-19 16:08:46 UTC - Alexandre DUVAL: If the topic defined in publish doesn't exists, it should create a topic with the passed avro schema. But Context.publish takes schema parameter as string so I'm a bit confused.
----
2019-08-19 16:10:25 UTC - Alexandre DUVAL: Or what should be the object passed if the schema type is "avro"?
----
2019-08-19 16:11:36 UTC - pradeep:
----
2019-08-19 16:18:46 UTC - Luke Lu: With proper impl of read-ahead, reading through broker should be a lot better as s3/gcs throughput is not bad.
----
2019-08-19 16:19:58 UTC - Alexandre DUVAL: seems to be related to function configuration argument -st?
----
2019-08-19 16:20:03 UTC - Alexandre DUVAL: Not sure how to define/use it
----
2019-08-19 16:21:47 UTC - Alexandre DUVAL: or maybe --output-serde-classname
----
2019-08-19 16:22:11 UTC - Alexandre DUVAL: But I have no idea of the output topic at this moment, it's computed in the function.
----
2019-08-19 16:22:32 UTC - Alexandre DUVAL: @Sijie Guo
----
2019-08-19 16:25:38 UTC - Alexandre DUVAL: (open this thread in channel to get all context^^)
----
2019-08-19 17:46:12 UTC - Jon Featherstone: @Jon Featherstone has joined the channel
----
2019-08-19 18:02:51 UTC - Jerry Peng: @Alexandre DUVAL example of publishing to a topic from a function with a schema:
```
context.newOutputMessage(publishTopic, Schema.AVRO(Foo.class)).value(output).sendAsync();
```
----
2019-08-19 18:03:23 UTC - Jerry Peng: A full example:
<https://github.com/apache/pulsar/blob/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/PublishFunction.java>
----
2019-08-19 18:03:23 UTC - Alexandre DUVAL: Ohh thanks, is that documented somewhere? I didn't find it.
----
2019-08-19 18:03:30 UTC - Alexandre DUVAL: ok, thx
----
2019-08-19 18:03:32 UTC - Alexandre DUVAL: :slightly_smiling_face:
----
2019-08-19 18:05:31 UTC - Jerry Peng: There are some docs here:
<https://pulsar.apache.org/api/pulsar-functions/>
----
2019-08-19 18:05:35 UTC - Jerry Peng: but they are old
----
2019-08-19 18:05:55 UTC - Jerry Peng: feel free to update them or add new docs :slightly_smiling_face:
----
2019-08-19 18:06:01 UTC - Alexandre DUVAL: And the value should be Schema.encode output?
----
2019-08-19 18:06:14 UTC - Alexandre DUVAL: @Jerry Peng
----
2019-08-19 18:07:15 UTC - Jerry Peng: > And the value should be Schema.encode output
Just the POJO if the schema is AVRO
----
2019-08-19 18:07:33 UTC - Jerry Peng: the schema will serialize the POJO to avro binary format automatically
----
2019-08-19 18:08:03 UTC - Alexandre DUVAL: okay, <https://pulsar.apache.org/api/pulsar-functions/> Context doesn't have method newMessageOutput, it that normal?
----
2019-08-19 18:08:17 UTC - Jerry Peng: The docs are old :slightly_smiling_face:
----
2019-08-19 18:08:22 UTC - Jerry Peng: we need to update
----
2019-08-19 18:08:26 UTC - Alexandre DUVAL: okay :wink:
----
2019-08-19 18:08:27 UTC - Alexandre DUVAL: ty
----
2019-08-19 18:10:59 UTC - Jerry Peng: @Sijie Guo @Matteo Merli do you guys know how to generate new java docs for Pulsar Functions SDK?
<https://github.com/apache/pulsar/issues/4979>
----
2019-08-19 18:43:54 UTC - Alexandre DUVAL: Nothing related, do you know a way to clean all schemas without topic existing?
----
2019-08-19 18:43:59 UTC - Alexandre DUVAL: @Jerry Peng
----
2019-08-19 18:53:42 UTC - Jerry Peng: There is not a way to do that.
----
2019-08-19 18:54:05 UTC - Jerry Peng: Though I believe there was code that was merged that will delete schemas when topics are deleted
----
2019-08-19 18:54:55 UTC - Alexandre DUVAL: there is a --deleteSchema options
----
2019-08-19 18:55:00 UTC - Alexandre DUVAL: but my topics are already deleted
----
2019-08-19 18:55:04 UTC - Alexandre DUVAL: and not their schemas :slightly_smiling_face:
----
2019-08-19 20:38:15 UTC - Poule: if I produce a non-schema-compliant message to a topic, it is supposed to reject it?
----
2019-08-19 20:46:40 UTC - Poule: I have a topic with a Avro schema, I just sent a test message to the topic and now presto complains
----
2019-08-19 21:31:47 UTC - Jerry Peng: you can also use the cli to delete schemas if you know which topics they were attached to
----
2019-08-19 21:31:57 UTC - Jerry Peng: but there is currently no way to just list all the schemas
----
2019-08-19 21:32:07 UTC - Jerry Peng: though not a hard feature to add
----
2019-08-19 21:32:34 UTC - Alexandre DUVAL: indeed
----
2019-08-19 21:36:44 UTC - Alexandre DUVAL: There is already getallschemas method in SchemaRegistryServiceWithSchemaDataValidator.java
----
2019-08-19 21:37:10 UTC - Alexandre DUVAL: oh noes it's all schema's versions mb :stuck_out_tongue:
----
2019-08-19 21:46:18 UTC - Alexandre DUVAL: not that easy, SchemaLocator depends on key
----
2019-08-19 21:46:23 UTC - Alexandre DUVAL: (key = schemaId)
----
2019-08-19 21:58:33 UTC - Tomasz Majewski: @Tomasz Majewski has joined the channel
----
2019-08-20 02:17:06 UTC - Sijie Guo: what clients are you using to produce the messages?
----
2019-08-20 02:58:11 UTC - Poule: the CLI `pulsar-client produce`
----
2019-08-20 03:08:38 UTC - Sijie Guo: `pulsar-client produce` raw bytes. currently broker doesn’t reject raw bytes by default (for compatibility with non-java languages). You can turn on rejecting raw bytes by setting `isSchemaValidationEnforced` to true in broker.conf
----
2019-08-20 03:16:00 UTC - Poule: ah the doc is not up-to-date that why I did not see the config param.. but it's definitely in `<https://github.com/apache/pulsar/blob/master/conf/broker.conf>`
----
2019-08-20 03:16:10 UTC - Poule: thanks
----
2019-08-20 05:07:41 UTC - Poule: at that point when I have malformed messages in the topic, what can I do to remove the malformed messages from the topic?
----
2019-08-20 05:11:42 UTC - Poule: problem is presto now fails on every query on `org.apache.avro.AvroRuntimeException: Malformed data`
----
2019-08-20 05:14:28 UTC - Poule: rendering it unusable
----
2019-08-20 05:38:54 UTC - Poule: I filed an issue in gh
----
2019-08-20 05:49:07 UTC - xuhuajie: @xuhuajie has joined the channel
----
2019-08-20 06:46:38 UTC - wenyicai: @wenyicai has joined the channel
----
2019-08-20 08:08:54 UTC - Martin Svensson: How is the ordering guarantee affected by repartitioning? (especially interested in the replaying of old messages case)
----
2019-08-20 08:25:57 UTC - Jianfeng Qiao: I disabled backlog quota check via setting backlogQuotaCheckEnabled=false in broker.conf, why I still run into the exception says "Cannot create producer on topic with backlog quota exceeded org.apache.pulsar.client.api.PulsarClientException$ProducerBlockedQuotaExceededError: Cannot create producer on topic with backlog quota exceeded"
----
2019-08-20 08:27:03 UTC - Jianfeng Qiao: Anyone know this issue?
----