You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/10/09 09:11:03 UTC
Slack digest for #general - 2019-10-09
2019-10-08 09:58:29 UTC - Vladimir Shchur: Hi! I'm working on adding support for Key_Shared subscription in dotnet client, but when testing I get broker error on consumer creation
```
19:49:45.396 [ForkJoinPool.commonPool-worker-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/172.17.0.13:34168] Created new producer: Producer{topic=PersistentTopic{topic=<persistent://public/default/topic-de885a14d0d64f8cb1dc365213cdbda0>}, client=/172.17.0.13:34168, producerName=PartitionedProducer, producerId=1}
19:49:45.436 [pulsar-io-23-4] WARN org.apache.pulsar.broker.service.ServerCnx - [/172.17.0.13:34168] Got exception UninitializedMessageException : Message was missing required fields. (Lite runtime could not determine which fields were missing).
org.apache.pulsar.shaded.com.google.protobuf.v241.UninitializedMessageException: Message was missing required fields. (Lite runtime could not determine which fields were missing).
at org.apache.pulsar.shaded.com.google.protobuf.v241.AbstractMessageLite$Builder.newUninitializedMessageException(AbstractMessageLite.java:298) ~[org.apache.pulsar-protobuf-shaded-2.1.0-incubating.jar:2.1.0-incubating]
at org.apache.pulsar.common.api.proto.PulsarApi$BaseCommand$Builder.build(PulsarApi.java:26302) ~[org.apache.pulsar-pulsar-common-2.3.2.jar:2.3.2]
at org.apache.pulsar.common.api.PulsarDecoder.channelRead(PulsarDecoder.java:86) ~[org.apache.pulsar-pulsar-common-2.3.2.jar:2.3.2]
```
Do I need to enable it on broker? I'm running 2.4.1 pulsar. Same story with Java client
----
2019-10-08 11:20:00 UTC - Vladimir Shchur: Just realized that logs show 2.3.2 version of libs, will try to update it
----
2019-10-08 11:52:50 UTC - Joose Helle: @Joe Francis "Basically any message after the oldest unacked message is kept. The delete cursor will not move past the oldest unacked message."
Is there any workaround to this? It would be great to be able to let Pulsar delete all the new acked messages after acking them (even if there is some old unacked messages).
----
2019-10-08 12:06:13 UTC - Raman Gupta: My consumers are reporting this warning on Pulsar 2.4.1:
```
org.apa.pul.cli.imp.UnAckedMessageTracker : [ConsumerBase{subscription='my-sub', consumerName='my-consumer', topic='<persistent://dev/my-ns/q>'}] 3600 messages have timed-out
```
It looks like <https://github.com/apache/pulsar/issues/2221> -- I'm also consuming from multiple threads -- but that was supposed to have been fixed. The only other diff might be is that I have 3 threads, *in 3 processes*, consuming the topic. I also note that all 3 processes report the same number of timed-out messages (3600) in all cases, and all output that log at least 5-6 times and one of them 13 times. BTW, all messages are acked (backlog size is 0) and I use the defaults for message TTL, so as far as I know having any unacked messages time out is impossible.
----
2019-10-08 13:04:52 UTC - Jack: Hi all, i have a pulsar bookie running for some time. But today, it stopped and i can't run again it? this is log when i run again. Someone tell me a problem. Thanks you
----
2019-10-08 13:16:12 UTC - Raman Gupta: On the face of it it seems like the bookie can't connect to zookeeper. Have you checked the network and/or zk cluster?
----
2019-10-08 13:23:43 UTC - Jack: Everything works well
----
2019-10-08 14:26:38 UTC - Ming: Hi, If I introduce a new endpoint, where is the swagger or some sort of doc generator to be updated?
-1 : Kendall Magesh-Davis, Jack
no_bell : Robin Bartholdson, Jack
face_with_symbols_on_mouth : Jack
hankey : Vladimir Shchur
----
2019-10-08 14:30:40 UTC - Ming: apologize! I am new to the community/channel. I shouldn't have used channel notification.
----
2019-10-08 14:32:39 UTC - Addison Higham: there are annotations placed on API endpoints and then you can run `src/gen-swagger.sh`
+1 : Ming, David Kjerrumgaard
----
2019-10-08 14:35:14 UTC - Retardust: -
----
2019-10-08 15:20:18 UTC - Raman Gupta: Hmm, is it possible these messages were present in the topic before the subscription was created? But in that case, if my subscription was created at "latest" position (which it was), then these messages should have remained unacked and therefore unexpired.
----
2019-10-08 15:22:20 UTC - Addison Higham: okay, got myself into a bit of a pickle on a testing cluster, but a good learning experience. I left something running that filled up disk, but now I can't make any of the API calls work to delete the subscription and the ledgers because none of the bookies are healthy.
----
2019-10-08 15:23:39 UTC - Raman Gupta: Can you make the disk bigger?
----
2019-10-08 15:24:01 UTC - Addison Higham: it seems like I could manually delete some ledgers using BK cli...but that will leave things inconsistent
----
2019-10-08 15:24:05 UTC - Addison Higham: is there another way to do that?
----
2019-10-08 15:24:40 UTC - Addison Higham: I could... but mostly I just want to recover by deleting the data (it isn't important, just some stress test data from another system)
----
2019-10-08 15:25:15 UTC - Raman Gupta: Sure, but making the disk bigger maybe allows you to bring up the bookies, so you can delete the data. Then you can resize back down.
----
2019-10-08 15:28:36 UTC - Addison Higham: yeah, I know what you mean, was just trying to be lazy. Technically, the disks aren't *completely* full, they just hit there limit so they marked themselves unhealthly, so maybe I could just try adjust that limit...
----
2019-10-08 15:29:35 UTC - Raman Gupta: Lazy is good :slightly_smiling_face:
----
2019-10-08 15:33:56 UTC - Addison Higham: that worked BTW, raised the `diskUsageThreshold`, restarted my bookies, made the calls
+1 : Raman Gupta
----
2019-10-08 15:34:04 UTC - Addison Higham: now I just need to clear up the disk by deleting the topics
----
2019-10-08 15:36:15 UTC - Raman Gupta: Note to self: always reserve some free space on a FS for cases like this! `tune2fs -m` to the rescue.
----
2019-10-08 15:36:16 UTC - Matteo Merli: @Joose Helle The data is stored as an immutable log, so it’s not possible to selectively physically delete messages individually.
Though, if the messages are acked, they won’t be dispatched again.
----
2019-10-08 15:47:51 UTC - Addison Higham: okay... deleted the topics, but it seems deletion of old ledgers isn't eager
----
2019-10-08 16:37:57 UTC - Xiaolei Zhu: Hi all, quick questions about using schema in Functions. Is there a way to let pulsar do the schema conversion? I am using a set of generic functions that select fields to perform operations based on settings passed through user config, somewhat like in SQL. Currently I am using Avro to manually decode each message and had to keep a redundant copy of schemas for each topic so that this can be done automatically. I also had to create each topic before the functions are deployed to make this work. I wonder if there are examples for how to utilize the built-in Schema Registry in functions? Thank you!
----
2019-10-08 16:43:43 UTC - Matteo Merli: When you specify a POJO as the in/out types for a Java function, the types will automatically be used to enforce the schema. The schema type (JSON, Avro, …) will be either inferred from the topic schema, specified in the function configuration or defaulted to JSON
----
2019-10-08 19:48:42 UTC - Xiaolei Zhu: Thank you. Is there a way to access this in the python api? I am trying to decouple the functions from schema, so it will be preferable not having to specify schema in function definition. For example I wrote a func to do mean and variance, and I can specify or modify on the fly which field it is computing from. This allows the same function to be applied to any topic. Ideally it would be fantastic to directly receive an Avro record object that is adapted to the schema of the topic. I did this some roundabout way but I’m wondering if there is an existing mechanism to do it.
----
2019-10-08 19:50:23 UTC - Xiaolei Zhu: Or perhaps if there is a way to receive specific field(s) as function inputs that be awesome
----
2019-10-08 20:45:21 UTC - Addison Higham: speaking of schema... if I have an avro schema already, I don't see an easy way to convert that to a pulsar avro schema...
----
2019-10-08 20:45:24 UTC - Addison Higham: am I missing something?
----
2019-10-08 20:47:29 UTC - Sijie Guo: Use SchemaDefinition to specify the avro schema json definition
----
2019-10-08 20:50:17 UTC - Addison Higham: okay, that makes sense... but I think I know where my problem is going to be... I am trying to use scala case classes without having to set `@BeanProperty` annotation everywhere...
----
2019-10-08 20:50:40 UTC - Sijie Guo: oh ok
----
2019-10-08 20:50:55 UTC - Raman Gupta: I had this question too... why wouldn't Pulsar just accept instances of `org.apache.avro.Schema` in `SchemaDefinition`?
----
2019-10-08 20:52:49 UTC - Addison Higham: oh maybe this will work... it takes the json def and doesn't use the pojo if you have the json definitions and instead just parses that, but trying to see where it does the actual serialization
----
2019-10-08 21:00:46 UTC - Dragos: @Dragos has joined the channel
----
2019-10-08 21:10:49 UTC - Matteo Merli: > Is there a way to access this in the python api
The functions Python API is not yet integrated with the schema
----
2019-10-08 21:21:00 UTC - Sijie Guo: because we don’t want to depend on the external library on the api module
----
2019-10-08 21:22:09 UTC - Raman Gupta: You could make it an optional (in gradle terms, an implementation) dependency.
----
2019-10-08 21:22:46 UTC - Sijie Guo: well if you use it in interface, it is a compile dependency not a provided dependency
----
2019-10-08 21:23:41 UTC - Sijie Guo: also Pulsar shades its dependencies as well, the avro library in pulsar-client module is also being shaded.
----
2019-10-08 21:24:03 UTC - Raman Gupta: Yeah, that's causing problems for me too, as the shaded avro version in Pulsar is too old.
----
2019-10-08 21:24:58 UTC - Sijie Guo: You can use the unshaded version pulsar-client-original. Although you might have to deal with dependencies conflicts with other your libraries.
----
2019-10-08 21:25:27 UTC - Raman Gupta: > well if you use it in interface, it is a compile dependency not a provided dependency
It wouldn't be part of the `SchemaDefinition` interface -- only the `AvroSchema` implementation.
----
2019-10-08 21:26:15 UTC - Raman Gupta: As an aside, I saw `pulsar-client-original`, yes, and that was my plan. Would be nice not to have to do that extra work though.
----
2019-10-08 21:26:41 UTC - Sijie Guo: if you are talking about AvroSchema, we can expose a constructor for AvroSchema with avro.Schema if needed.
----
2019-10-08 21:28:37 UTC - Sijie Guo: > it takes the json def and doesn’t use the pojo if you have the json definitions and instead just parses that, but trying to see where it does the actual serialization
@Addison Higham: if you supply the json def, Pulsar schema will not use POJO to generate the schema definition. If you don’t supply the json def, Pulsar schema will generate the schema definition for the POJO. Avro serializes the POJO based on the json definition.
----
2019-10-08 21:29:06 UTC - Raman Gupta: Yeah that would be nice. I've already got all the compiled Avro files lying around. Better yet, allow to passin a SpecificRecord class, and extract the schema from it directly.
----
2019-10-08 21:29:28 UTC - Raman Gupta: Or even any `GenericContainer`
----
2019-10-08 21:30:21 UTC - Sijie Guo: Can you file a GitHub issue for that?
----
2019-10-08 21:30:31 UTC - Raman Gupta: Yep
----
2019-10-08 21:33:00 UTC - Addison Higham: @Sijie Guo I think the problem is though that the default Avro serializers don't know what to do with case classes (but maybe I am wrong?) as they aren't POJOs, so I could do generic records that way, but if I want it to just take case classes, I need to implement my own SchemaReader and SchemaWriter that is smart about case classes
----
2019-10-08 21:33:16 UTC - Addison Higham: avro4s does all that... just need to plumb it in
----
2019-10-08 21:36:19 UTC - Raman Gupta: Does avro4s generate case classes that implement `SpecificRecord`? Because if so, then the default Avro serializer should know how to deal with it, IIUC.
----
2019-10-08 21:36:59 UTC - Addison Higham: na, it is more of "I have case classes that I want to make go in and out of avro" less of code generation from avro defs
----
2019-10-08 21:37:11 UTC - Raman Gupta: Ah
----
2019-10-08 21:43:35 UTC - Sijie Guo: Ah I see
----
2019-10-08 21:44:04 UTC - Sijie Guo: Maybe check out how pulsar4s handles the case classes for Avro schema
----
2019-10-08 21:44:45 UTC - Addison Higham: it just uses `@BeanProperty` for everything
----
2019-10-08 21:44:57 UTC - Addison Higham: which I have a few hundred props... sorta annoying to add :stuck_out_tongue:
----
2019-10-08 22:04:39 UTC - kelv: @kelv has joined the channel
----
2019-10-09 00:50:24 UTC - Nicolas Ha: <https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerInterceptor.html>
`onAcknowledge` - where can I find what kind of Exceptions, and how to distinguish a “good” acknowledge, a timeout or a nack?
----
2019-10-09 00:59:08 UTC - Nicolas Ha: also, given a ` onAcknowledgeCumulative` `MessageId` - is there a way to find the non-acked `messagesId`s that will be acked?
----
2019-10-09 02:46:32 UTC - yici: @yici has joined the channel
----