You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/10/18 09:11:02 UTC
Slack digest for #general - 2018-10-18
2018-10-17 15:21:51 UTC - Ivan Kelly: @Martin Svensson what issue are you seeing?
----
2018-10-17 15:21:58 UTC - Ivan Kelly: there should be no configuration required
----
2018-10-17 15:42:56 UTC - Nicolas Ha: well it looks like I won't be able to get to the pulsar helm daemonset soon so if someone else wants to contribute it I'd be very glad :smile:
----
2018-10-17 15:54:24 UTC - Karthik Palanivelu: Team, I am trying to test the Reader functionality. Following are the outputs from my producer in Cluster 1. I am having the reader in Cluster 2. Clustering is fine, I tested it. But not able to read using reader. I need to read from Id-74. Consumer works fine. Please let me know what I am doing wrong. Below are the info from Producer:
```
Sending Message ---> messages-East-100 Entry Id -> 70 Ledger Id -> 0 Partition Index -->-1
Sending Message ---> messages-East-101 Entry Id -> 71 Ledger Id -> 0 Partition Index -->-1
Sending Message ---> messages-East-102 Entry Id -> 72 Ledger Id -> 0 Partition Index -->-1
Sending Message ---> messages-East-103 Entry Id -> 73 Ledger Id -> 0 Partition Index -->-1
Sending Message ---> messages-East-104 Entry Id -> 74 Ledger Id -> 0 Partition Index -->-1
Sending Message ---> messages-East-105 Entry Id -> 75 Ledger Id -> 0 Partition Index -->-1
Sending Message ---> messages-East-106 Entry Id -> 76 Ledger Id -> 0 Partition Index -->-1
Sending Message ---> messages-East-107 Entry Id -> 77 Ledger Id -> 0 Partition Index -->-1
Sending Message ---> messages-East-108 Entry Id -> 78 Ledger Id -> 0 Partition Index -->-1
Sending Message ---> messages-East-109 Entry Id -> 79 Ledger Id -> 0 Partition Index -->-1
```
Reader Code:
```
MessageId idImpl = new MessageIdImpl(0, 74, -1);
Reader reader = client.newReader().topic(topic).startMessageId(idImpl)
.create();
while (true) {
// Wait for a message
Message msg = reader.readNext();
System.out.println("Received message: " + StringUtils.newStringUtf8(msg.getData()));
}
```
----
2018-10-17 16:02:31 UTC - Martin Svensson: @Ivan Kelly I am now using 2.2 and gcp storage and no longer have that problem. If it’s any help I saved these lines:
I 13:34:34.830 [bookkeeper-ml-workers-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/test/persistent/t4] Error opening ledger for reading at position 1574:213552 - org.apache.bookkeeper.mledger.ManagedLedgerException: Unknown exception
I 13:34:34.830 [bookkeeper-ml-workers-OrderedExecutor-1-0] WARN org.apache.bookkeeper.mledger.impl.OpReadEntry - [public/test/persistent/t4][s1] read failed from ledger at position:1574:213552 : Unknown exception
I 13:34:34.831 [broker-topic-workers-OrderedScheduler-2-0] ERROR org.apache.pulsar.broker.service.persistent.PersistentDispatcherSingleActiveConsumer - [<persistent://public/test/t4> / s1-Consumer{subscription=PersistentSubscription{topic=<persistent://public/test/t4>, name=s1}, consumerId=0, consumerName=d34dd, address=/10.36.9.58:34328}] Error reading entries at 1574:213552 : Unknown exception - Retrying to read in 57.205 seconds
----
2018-10-17 16:04:44 UTC - Ivan Kelly: during read of offload?
----
2018-10-17 16:05:32 UTC - Martin Svensson: I do however find the replay from offloaded ledgers to be really slow (500 msg/s instead of 50000 msg/s from my local dev box connected via proxy in kubernetes)
----
2018-10-17 16:06:34 UTC - Martin Svensson: yes, during read of offload (the errors you see are when reading from S3)
----
2018-10-17 16:12:23 UTC - Ivan Kelly: ok, I've added a task for look into what the expected performance should be
----
2018-10-17 16:18:14 UTC - Ryan Samo: @Ryan Samo has joined the channel
----
2018-10-17 16:50:16 UTC - Ryan Samo: Hey guys and gals! I have 2 questions if anyone has ran across this.
1. Is there a good example out on the interwebs of a Pulsar producer and consumer both using WSS secure websockets? I have normal websockets working beautifully but the next step is to secure it down and move to port 8443.
2. Does the Pulsar websocket API support end to end encryption (AES)? The only example of this in the docs is with the Java API. If so is there an example for this as well?
Thanks!
----
2018-10-17 17:12:26 UTC - Martin Svensson: thanks @Ivan Kelly
I see that my brokers all jump from half gb of memory to around 10 gb just at the time it starts reading from gcp
----
2018-10-17 17:46:09 UTC - Matteo Merli: @Karthik Palanivelu The reader will start on the next message id. This is because typically you store the message id of something that was already processed.
We were thinking of adding another option to let application specify whether that is included/excluded
----
2018-10-17 17:50:41 UTC - Matteo Merli: @Ryan Samo
1. The configuriguration for HTTPS in web socket proxy is very similar to the conf in broker. Follow the steps for broker (<http://pulsar.apache.org/docs/en/security-tls-authentication.html#tls-authentication-overview>) and apply same config in `websocket.conf` (or if you’re enabling websocket inside broker, directly in `broker.conf`).
2. Right now, the end-to-end encryption is not available through WebSocket. The main problem is how to pass the keys to the websocket proxy in a secure manner.
----
2018-10-17 17:50:56 UTC - Karthik Palanivelu: @Matteo Merli My use case is I produce messages on Cluster 1 which is set to replicate at Cluster 2. If I have the consumers on Cluster 1 and 2; they both read the same set of messages on a Async replication. I believe you can recollect this scenario, I discussed this earlier. In this case, I am having consumer on Cluster 1 alone and Reader in Cluster 2 which should read the messages after MessageId 74; but it is not doing any. If I start a consumer at Cluster 2, it reads all the messages. I need help to setup reader correctly because Replication works.
----
2018-10-17 17:58:38 UTC - Matteo Merli: The problem is that the message ids will be different across the 2 clusters
----
2018-10-17 17:59:41 UTC - Karthik Palanivelu: Oh, then how I get to read it from where I left off. Reader takes only Id and not Timestamp. Correct?
----
2018-10-17 18:03:12 UTC - Matteo Merli: Yes, correct. Though is possible to use a subscription and roll it back on a particular timestamp
----
2018-10-17 18:03:57 UTC - Matteo Merli: (through admin API/ cli)
----
2018-10-17 18:05:00 UTC - Ryan Samo: Thanks for the quick response @Matteo Merli ! I’ll give it a shot.
----
2018-10-17 18:05:29 UTC - Karthik Palanivelu: How Could I do in Reader?
----
2018-10-17 18:06:27 UTC - Karthik Palanivelu: Any Sample on using a Subscription to use a TimeStamp in Java API?
----
2018-10-17 18:06:28 UTC - Matteo Merli: I guess we would have to extend Reader to take a “start-timeestamp)
----
2018-10-17 18:07:00 UTC - Matteo Merli: <http://pulsar.apache.org/docs/en/admin-api-persistent-topics/#reset-cursor>
----
2018-10-17 18:07:50 UTC - Matteo Merli: ```
reset-cursor Reset position for subscription to position closest to timestamp
Usage: reset-cursor [options] <persistent://property/cluster/namespace/destination>
Options:
* -s, --subscription
Subscription to reset position on
* --time, -t
time in minutes to reset back to (or minutes, hours,days,weeks eg:
100m, 3h, 2d, 5w)
```
----
2018-10-17 18:08:42 UTC - Matteo Merli: :+1: Let me know if there issues in setting that up
----
2018-10-17 18:09:19 UTC - Karthik Palanivelu: Yes...I remember you saying that in our conversation of Failover Subscription....any closer on that request?
----
2018-10-17 18:09:33 UTC - Ryan Samo: I sure will! :+1:
----
2018-10-17 18:13:00 UTC - Ryan Samo: Oh sorry one more thing, on the websocket client side I will need to pass the cert, key, and ca in order for authentication and authorization to work. Is there an example of this in nodeJs or python that you know of?
----
2018-10-17 18:22:40 UTC - Matteo Merli: These would be depending on how you establish the HTTPS session. I haven’t actually tried that, but there should be a `sslopt` config map when you create the websocket client in Python
----
2018-10-17 18:41:24 UTC - Ryan Samo: Thanks, I'll look into that. I coded up a quick test with Node.js and it seems to be close to working, but I always wind up with a "bad_certificate" 500 error. It seems the handshake from client to server is not happy. The tlsOptions portion of this snippet is part of the underlying Node.js tcp.connect() method which is called from the Node.js HTTPS library. The websocket package utilizes this code path so I would think it should work, might just be a self signed cert issue.
----
2018-10-17 19:02:07 UTC - sfescape: btw, I use package reconnecting-websocket to handle failure modes. This is particularly helpful with long running consumers.
----
2018-10-17 19:03:25 UTC - Ryan Samo: Nice tip @sfescape , I’ll check that out too
+1 : sfescape
----
2018-10-17 20:06:20 UTC - Ivan Kelly: yes, when reading from long term storage, you need to stage the data in memory before reading it in
----
2018-10-17 20:06:33 UTC - Ivan Kelly: not 100% on what the buffer sizes are with GCP
----
2018-10-17 20:47:30 UTC - Paul Wistrand: Hi All, I'm trying out pulsar on minikube and following the example to log using a python function (see <http://pulsar.apache.org/docs/en/functions-api/#python-logging>). If I don't deploy with the option to make the function log to a topic shouldn't I be seeing a log statement somewhere? I was expecting the function to execute on one of the broker pods. I can see its been executed via `pulsar-admin functions getstatus`. /thanks
----
2018-10-17 20:51:39 UTC - Sanjeev Kulkarni: @Paul Wistrand In the broker/worker which executed the function, you should see a logs directory. The function logs will be kept in that directory
+1 : Paul Wistrand, Ali Ahmed
----
2018-10-17 21:00:40 UTC - Paul Wistrand: @Sanjeev Kulkarni thank you. do you know if its possible to have this redirected to stdout so I wouldn't need to run logging sidecar to get this into log management system?
----
2018-10-17 21:05:07 UTC - Sanjeev Kulkarni: we are working on making this redirect to stdout for both java/python functions. If you are using java, easiest would be to change the logging yaml thats inside the broker docker container
----
2018-10-17 21:05:16 UTC - Sanjeev Kulkarni: for python, not sure,
----
2018-10-17 21:10:33 UTC - Paul Wistrand: ok thx
----
2018-10-17 22:25:59 UTC - Martin Svensson: the offloaded data wasn’t more than a couple of hundred mb though
----
2018-10-18 04:04:33 UTC - pierrexiong: @pierrexiong has joined the channel
----
2018-10-18 07:54:52 UTC - Ivan Kelly: Ok, will take a look. This was all with GCP, not S3?
----