You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/01/22 09:11:03 UTC
Slack digest for #general - 2020-01-22
2020-01-21 09:23:31 UTC - orly priev: @orly priev has joined the channel
----
2020-01-21 11:03:20 UTC - Fernando: Does anyone know if there’s a connector example that supports mTLS currently? I’d like to add it to <https://github.com/apache/pulsar/blob/master/pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/ElasticSearchSink.java> But I’m not too familiar with Java libraries. I need to be able to pass a CA, and an SSL cert/key pair.
----
2020-01-21 11:41:00 UTC - Swaroop Kumar: Hello everyone, here is my usecase.
My producer will send out messages for multiple users in varying numbers. My consumers (count 30) are listening to these messages. As the message count per user is varying, I would like to process fair share of messages per user (say 5) at a given time. Is it possible to achieve this behaviour using Pulsar?
----
2020-01-21 15:34:15 UTC - Filippo Fadda: @Filippo Fadda has joined the channel
----
2020-01-21 16:27:54 UTC - Filippo Fadda: If I have understood your problem, you can use batching. If batching is enabled, the producer will accumulate and send a batch of messages in a single request. Batching size is defined by the maximum number of messages and maximum publish latency.
----
2020-01-21 16:32:04 UTC - David Kjerrumgaard: @Swaroop Kumar Perhaps you could use the Key-shared subscription mode to achieve your desired outcome. If you know the number of consumers up front, then you could assign each of them a "key" on the producer side. So if you wanted 5 messages to go to consumer 1, you would set the key value of the next 5 messages produced to "1", then the next 3 messages would have a key value of "4", etc.
----
2020-01-21 16:33:10 UTC - Devin G. Bost: I’m getting a strange exception in one of the tests in my PR: <https://github.com/apache/pulsar/pull/6031/checks?check_run_id=399755355>
It’s only this one test that’s failing in: `unit-test-flaky`
```[ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 8.773 s <<< FAILURE! - in org.apache.pulsar.proxy.server.ProxyTest
20
[ERROR] testRegexSubscription(org.apache.pulsar.proxy.server.ProxyTest) Time elapsed: 0.155 s <<< FAILURE!```
None of my code in the PR touched anything related to regex, and I can’t reproduce the error locally.
----
2020-01-21 16:33:59 UTC - David Kjerrumgaard: @Fernando So you are asking for a TLS Auth based CredentialsProvider inside the Elastic Search connector, and are looking for an example ?
----
2020-01-21 16:57:18 UTC - Fernando: Yes, as far as I can see from the Java implementation it doesn't take mTLS into account. I'm wondering if there's another connector that has mTLS so I can reference it to implement something similar in the elastic search case
----
2020-01-21 19:11:01 UTC - David Kjerrumgaard: You will have to extend the existing connector to include the necessary properties required for mTLS (trust certs file, client certificate and any associated private key, etc) and then use those to establish the connection to the secured ES server
----
2020-01-21 19:11:43 UTC - David Kjerrumgaard: The AWS connectors might have a good example since they require AIM credentials to be passed in
----
2020-01-21 21:44:25 UTC - Devin G. Bost: Sure, there are ways to do that.
However, without more detail, it’s hard to comment on specifics.
----
2020-01-22 04:27:11 UTC - Matt Hino: @Matt Hino has joined the channel
----