You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/01/18 09:11:02 UTC
Slack digest for #general - 2019-01-18
2019-01-17 09:25:25 UTC - Vincent Ngan: What is the maximum size of a message allowed by Pulsar?
----
2019-01-17 09:25:48 UTC - Ali Ahmed: 5 mega bytes
----
2019-01-17 09:26:48 UTC - Vincent Ngan: Is this a hard limit? Can I configure it to a larger value?
----
2019-01-17 10:47:42 UTC - Ganga Lakshmanasamy: @Karthik Ramasamy @Ravi Said that you have some reference for real-time consumption of logs from a https endpoint and ingesting into S3/ elastic using streamlio. Can you please share the details.
----
2019-01-17 11:08:54 UTC - jia zhai: This is limited by BookKeeper. and it is hard coded in pulsar currently.
----
2019-01-17 11:51:27 UTC - Harald Gustafsson: When deleting a namespace all topics need to be deleted first, that can be done with force even when producers or consumers are connected. The problem is that producers will reconnect after 0.1 seconds, and then the namespace is still not empty when trying to delete it. Is there a way of informing producers and consumers that when forcefully disconnected they should not reconnect or alternatively is there some broker admin interface to inform all clients on a topic to disconnect?
----
2019-01-17 12:17:33 UTC - jia zhai: @Harald Gustafsson Currently, seems there is no such interface.
----
2019-01-17 12:20:15 UTC - Harald Gustafsson: @jia zhai Was thinking if I removed access right from namespace first, would that prevent clients from reconnecting?
----
2019-01-17 12:25:15 UTC - jia zhai: :+1: you could have a try.
----
2019-01-17 12:25:47 UTC - jia zhai: In the code, when do reconnection, it will verify the authorize
----
2019-01-17 13:00:54 UTC - Vincent Ngan: Thx
----
2019-01-17 14:26:33 UTC - Gofu: @Gofu has joined the channel
----
2019-01-17 14:29:39 UTC - Gofu: Helo guys, can someone tell me if there is some documentation of the REST admin api, that shows the schema of the http body? Here <https://pulsar.apache.org/docs/latest/reference/RestApi/#/admin/persistent/:property/:cluster/:namespace/:destination/partitions>, I can't see the body schema. I am assuming that's missing because when I do the request the response message is: The request entity cannot be empty.
----
2019-01-17 14:52:33 UTC - Paul van der Linden: I might be missing something
----
2019-01-17 14:52:43 UTC - Paul van der Linden: But I deployed a pulsar cluster to kubernetes
----
2019-01-17 14:52:54 UTC - Paul van der Linden: in a different namespace then "default"
----
2019-01-17 14:53:36 UTC - Paul van der Linden: somehow the broker reports it's url as if it is running in the default namespace: `broker.default.svc.cluster.local` instead of `broker.test-pulsar.svc.cluster.local`
----
2019-01-17 15:10:22 UTC - Paul van der Linden: NVM, found it in the metadata init manifest
+1 : jia zhai
----
2019-01-17 15:24:50 UTC - Sijie Guo: @Gofu you are accessing the old website. you can find the rest api documentation here : <https://pulsar.apache.org/en/admin-rest-api/>
----
2019-01-17 15:44:43 UTC - Matteo Merli: Yes, removing the permissions will immediately disconnect existing producers/consumers that have no more access
----
2019-01-17 15:55:24 UTC - Gofu: thank you so much
----
2019-01-17 16:27:48 UTC - Gofu: @Sijie Guo even in <https://pulsar.apache.org/en/admin-rest-api/> the endpoint doesnt have body specification in the endpoint "Create a partitioned topic" per example.
----
2019-01-17 16:32:10 UTC - Ganga Lakshmanasamy: We are having problem in sending and receiving JSONObjects. The json values are converted into objects from string/array. is it something expected? Here is an example. This is the data sent {"id":"Thgkkg"} , data received {"id":"java.lang.Object@69cccc31"} Here is the consumer creation code
consumer = client.newConsumer(Schema.AVRO(JSONObject.class))
.topic(TOPIC_NAME)
.subscriptionType(SubscriptionType.Shared)
.subscriptionName(SUBSCRIPTION_NAME)
.subscribe();
----
2019-01-17 16:34:23 UTC - Sijie Guo: ah, i see. I think swagger annotation doesn’t render the rest api page correctly.
the post body should be an integer - the number of the partition.
----
2019-01-17 16:35:36 UTC - Sijie Guo: if your payloads are json, you should use json schema not avro schema, no?
----
2019-01-17 16:36:52 UTC - Gofu: @Sijie Guo I have tried without success
{
"partitions": 2
}
----
2019-01-17 17:02:38 UTC - Sijie Guo: @Gofu just post with ‘2’
----
2019-01-17 17:08:54 UTC - Gofu: @Sijie Guo thank you very much, the only way I can see it is looking into pulsar api code?
----
2019-01-17 17:14:25 UTC - Sijie Guo: @Gofu - most of the rest apis might already have documented the post body. however I think swagger annotation doesn’t handle well (or we misuse it), this rest api doesn’t seem to have primitive post payload documented. for such case, pulsar api code might be an easier way for you to figure out. or ping us on slack channel :slightly_smiling_face:
for this particular issue, do you mind creating a github issue for us? we will follow up to address the documentation issue.
----
2019-01-17 17:15:24 UTC - Gofu: Yes sure @Sijie Guo, thanks for the support
----
2019-01-17 18:52:17 UTC - Ravi: any luck on the GUI for pulsar in a container ?
----
2019-01-17 18:59:49 UTC - chris: @Ravi there is an image with the pulsar gui in it (apachepulsar/pulsar-dashboard). here is an example docker compose file to run it locally <https://github.com/apache/pulsar/blob/master/docker-compose/standalone-dashboard/docker-compose.yml>
----
2019-01-17 19:32:08 UTC - Louis Lapierre: @Louis Lapierre has joined the channel
----
2019-01-17 20:41:15 UTC - Emma Pollum: Currently there are some topic stats that are not available through prometheus, only the command line. Is there anything in progress to combine these?
----
2019-01-17 20:51:56 UTC - Ivan Kelly: @Emma Pollum which stats in particular? some stats changes have gone in recently
----
2019-01-17 20:55:45 UTC - Emma Pollum: I'm looking for average message size and message rate expired
----
2019-01-17 21:07:12 UTC - Ivan Kelly: hmm, looks like the recent changes were all around functions
----
2019-01-17 22:00:30 UTC - Emma Pollum: @Ivan Kelly Are there any plans to put these into prometheus?
----
2019-01-17 23:38:32 UTC - Jerry Peng: @Grant Wu after this PR gets merged, the issue you described in regards to getting the actual topic name of a message when subscribing to a regex topic will be fix:
<https://github.com/apache/pulsar/pull/3384>
----
2019-01-17 23:41:07 UTC - Grant Wu: I saw that, great! :clap:
----
2019-01-18 00:16:13 UTC - Grant Wu: So I’m reading <https://pulsar.apache.org/docs/en/functions-guarantees/> and I was wondering if I could get a little more clarity on how exactly we know when a function has successfully processed an input
----
2019-01-18 00:17:58 UTC - Grant Wu: @Jerry Peng I’ve skimmed through <https://streaml.io/blog/exactly-once> and it seems like a great overview of the concepts of effectively once processing
----
2019-01-18 00:18:17 UTC - Grant Wu: But I think “how do we know a Pulsar Function succeeded” isn’t really described here
----
2019-01-18 00:18:24 UTC - Grant Wu: Is it that no exception was raised?
----
2019-01-18 00:18:28 UTC - Grant Wu: (For Python, at least)
----
2019-01-18 00:18:32 UTC - Grant Wu: cc @Jonathan Budd
----
2019-01-18 00:30:48 UTC - Emma Pollum: When setting cluster properties, what is the difference between --url and --broker-service-url
----
2019-01-18 00:41:32 UTC - Emma Pollum: And what is a peer cluster?
----
2019-01-18 01:09:27 UTC - Sijie Guo: you can check this out - <https://github.com/apache/pulsar/wiki/PIP-8:-Pulsar-beyond-1M-topics>
----
2019-01-18 01:09:55 UTC - Sijie Guo: --broker-service-url is the pulsar binary protocol : pulsar://<dns>:6650
----
2019-01-18 01:10:08 UTC - Sijie Guo: --url is the http rest endpoint: http://<dns>:8080
----
2019-01-18 04:51:32 UTC - bossbaby: how prevent pulsar client subcribe subcription that they not own it. I think i use cert but im not found document about it
----
2019-01-18 05:59:07 UTC - jia zhai: @bossbaby FYI： <http://pulsar.apache.org/docs/en/security-overview.html#role-tokens>
----
2019-01-18 07:21:29 UTC - Ganga Lakshmanasamy: Thanks. Will check this. @Ravi FYI
----
2019-01-18 08:09:42 UTC - xiaolong.ran: @xiaolong.ran has joined the channel
----