You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/05/01 09:11:02 UTC
Slack digest for #general - 2018-05-01
2018-04-30 10:23:13 UTC - jim.pauley: @jim.pauley has joined the channel
----
2018-04-30 10:42:53 UTC - jim.pauley: anyone have a link to where the javadoc MessageToValuesMapper in pulsar-storm 1.22 can be found? All I can find from <https://pulsar.incubator.apache.org/docs/v1.22.0-incubating/getting-started/LocalCluster/> is client-api not pulsar-storm api
----
2018-04-30 11:14:49 UTC - Harald Gustafsson: Anyone that is building a pulsar docker image from master branch, and publish it? Like this one but for latest master branch: <https://hub.docker.com/r/apachepulsar/pulsar/> Wanted to test some of the soon 2.0 features.
----
2018-04-30 14:25:03 UTC - Byron: So it turns out the WS interface works when connecting to the broker nodes, but I get a 404 when connecting to a proxy
----
2018-04-30 15:18:59 UTC - Karthik Palanivelu: Hi All and @Matteo Merli, I am trying to enable TLS to test the Multi-Tenant feature as we discussed last week. I generated the certs based on the doc on <https://pulsar.apache.org/docs/v1.22.0-incubating/admin/Authz/#tls-client-auth>. I generated Server and Client Certs with *.<http://pulsarapp.com|pulsarapp.com>. I configured the Standalone.conf, discovery.conf and client.conf as mentioned. I replaced broker.conf with standalone.conf as I am using the same instance. But I am getting the following error:
```
2018-04-30 11:12:46,971 - WARN - [pulsar-web-76-5:AuthenticationFilter@71] - [10.120.145.72] Failed to authenticate HTTP request: Authentication required
2018-04-30 11:12:46,973 - INFO - [pulsar-web-76-5:Slf4jRequestLog@60] - 10.120.145.72 - - [30/Apr/2018:11:12:46 -0400] "GET //<HOST_NAME>:8080/admin/clusters HTTP/1.1" 401 345 "-" "Jersey/2.23.2 (HttpUrlConnection 1.8.0_161)" 11
2018-04-30 11:12:46,983 - INFO - [main:PulsarStandaloneStarter@200] - HTTP 401 Authentication required
```
Can you please help me with what changes I am missing?
----
2018-04-30 15:33:53 UTC - Sijie Guo: > Anyone that is building a pulsar docker image from master branch, and publish it?
@Harald Gustafsson: we don’t have a nightly job on publishing the latest image from master, we only have a jenkins job for publishing the image for releases. we will be starting 2.0 release this week (with the voting process, one at pulsar community and one at apache incubator), expected to have 2.0 release general available late next week. Let me know if this timeline works for you or not.
----
2018-04-30 15:35:21 UTC - Sijie Guo: > So it turns out the WS interface works when connecting to the broker nodes, but I get a 404 when connecting to a proxy
@Byron: which version are you using? 1.22.0 or latest master?
yeah current http related (both admin and ws) are not proxied at proxies. there is an improvement in 2.0 release addressed that.
----
2018-04-30 15:38:26 UTC - Byron: 1.22
----
2018-04-30 15:39:28 UTC - Byron: Ok, I can create a service for the brokers directly. Re: proxies, since I am using k8s with an ingress point, is a proxy needed in this context?
----
2018-04-30 15:42:07 UTC - Byron: Since incoming connections will be load balanced anyway
----
2018-04-30 15:42:11 UTC - Byron: across the brokers
----
2018-04-30 15:57:06 UTC - Sijie Guo: > is a proxy needed in this context
good question.
if you are ingesting within same k8s network, you don’t need to proxy. however if you are ingesting outside of k8s network, you need proxy depends on how do you setup the network.
why is that? because most of the binary protocol and http rest api will have a “redirect” request, a broker will redirect the clients to the broker that owns the topic (http requests are redirected via 307). on redirection, the clients are basically told “you should talk to that broker at this ‘location’“. the broker location is the hostname (or ip) or that broker. typically the hostname/ip are internal to k8s network, so your clients can’t actually be redirected to that broker.
the only exception is if you enable nodePort for broker port and http rest port when you running the brokers on k8s, and set the advertisedAddress to node. in this way, the port is exported and node address will be used for redirection, your client can actually work. This is doable if you have your own physical cluster and manage the hosts. However it is tricky when running at the k8s service in cloud.
a proxy here is essentially a “broker” but it handles finding the topic owners and proxying the connections to the right broker.
hope this explain why proxy is needed in k8s.
----
2018-04-30 16:06:48 UTC - Byron: @Sijie Guo yes that makes perfect sense. thank you
----
2018-04-30 16:06:50 UTC - Harald Gustafsson: @Sijie Guo Good to know that 2.0 is coming soon. Maybe I could build it myself, which docker file is it that generates it. I'm running in standalone mode to begin with to test out my setup locally first.
----
2018-04-30 16:07:04 UTC - Sijie Guo: @Harald Gustafsson one second
----
2018-04-30 16:07:25 UTC - Matteo Merli: @Harald Gustafsson `docker/build.sh`
----
2018-04-30 16:07:35 UTC - Sijie Guo: <https://github.com/apache/incubator-pulsar/tree/master/docker>
----
2018-04-30 16:07:47 UTC - Sijie Guo: oh @Matteo Merli is fast :slightly_smiling_face:
----
2018-04-30 16:34:18 UTC - Harald Gustafsson: Thanks, I'll try that.
----
2018-04-30 17:56:55 UTC - Karthik Palanivelu: Hi All, Is there any specific setting for generating the client cert for pulsar-admin to use that is used to configured in client.conf? Like Common name as admin?
----
2018-04-30 22:11:52 UTC - Rob V: @Rob V has joined the channel
----
2018-04-30 22:16:01 UTC - Rob V: hi there
----
2018-04-30 22:16:19 UTC - Rob V: is there any support for message schemas, like Kafka supports Avro messages?
----
2018-04-30 22:16:28 UTC - Rob V: using the Confluent schema registry?
----
2018-04-30 22:31:42 UTC - Ali Ahmed: pulsar 2.0 which is releasing soon , will have a pluggable schema registry and type safe java clients
----
2018-05-01 00:45:10 UTC - Byron: any planned support for atomic batch writes? i.e. an ack only if N messages have been stored
----
2018-05-01 00:56:28 UTC - Ali Ahmed: batching is already supported
----
2018-05-01 01:00:35 UTC - Byron: The way I am interpreting the batching support is for performance reasons, like flushing periodically rather than on each write. i mean `publisher.send([msg1, msg2, msg3])`
----
2018-05-01 01:01:41 UTC - Byron: the use case being committing sets of messages together atomically
----
2018-05-01 01:01:42 UTC - Ali Ahmed: I recommend you use the async api with batching , that way multiple calls are internally batched together based on the config
----
2018-05-01 01:04:36 UTC - Sijie Guo: @Byron yes there was a working item supporting unlimited-size messages / atomic batch writes . but it is not included in 2.0, we prioritized other features for 2.0. After 2.0, we might pickup this work.
----
2018-05-01 01:06:15 UTC - Byron: Great, thanks for understanding
----
2018-05-01 01:06:54 UTC - Byron: I really dig the simplicity of the API and administrative controls are very good
+1 : Sijie Guo
----
2018-05-01 01:07:33 UTC - Byron: The real use I am thinking about multi-message writes is for event sourcing use cases, where multiple events may need/want to be published together for consumers
----
2018-05-01 01:08:10 UTC - Byron: of course messages could be kept track of individually and retried, but the batch is definitely a nice to have
----
2018-05-01 01:09:17 UTC - Byron: out of curiosity, what is the use case for unlimited size messages? i suppose a generalization of any size, so the message itself would be streamed and chunked transparently?
----
2018-05-01 01:12:18 UTC - Sijie Guo: yes. atomic batch writes definitely provide simplicity to a lot of applications. “atomic writes” is kind of the mechanism we are thinking, it can be exposed directly to users, which users can use that directly. “atomic writes” can become the underlying mechanism for chunking and streaming large messages, for “unlimited-size messages” use cases. “unlimited-size messages” use cases - e.g. if you want to stream images/checks or even video frames.
----
2018-05-01 01:13:10 UTC - Byron: right, that makes sense. good abstraction for that
----
2018-05-01 01:14:15 UTC - Byron: thanks for answering my questions!
----
2018-05-01 01:15:04 UTC - Sijie Guo: no problem. will also update you when we pick this work up.
----
2018-05-01 01:16:12 UTC - Byron: i am going to probably going to write a Go client around the WS interface for the time being to act as a bridge. i was told/heard someone is working on a Go client using the binary protocol
----
2018-05-01 01:16:32 UTC - Byron: i wouldn’t mind reviewing it, testing it out, etc.
----
2018-05-01 01:16:42 UTC - Byron: if the author(s) are interested in that
----
2018-05-01 01:17:17 UTC - Byron: i wasn’t sure _who_ was working on it
----
2018-05-01 01:17:47 UTC - Byron: and i know the wrapper around the C client is an option as well
----
2018-05-01 01:18:12 UTC - Byron: albeit introducing the cgo dependency
----
2018-05-01 01:20:44 UTC - Sijie Guo: yeah, there are two groups of people writing go clients. both are around binary protocol. one is still closed source, we have been talking to the authors to open source it, but it might take a while; and this is the other one : <https://github.com/t2y/go-pulsar>
----
2018-05-01 01:21:28 UTC - Sijie Guo: yeah, the other approach is write a go client wrapping the c++ client. that is probably more doable to make sure the client catching up with features
----
2018-05-01 01:22:41 UTC - Matteo Merli: Tentatively, the target for cgo based implementation is 2.1
----
2018-05-01 01:23:29 UTC - Byron: thanks for the info. i need to take a look at that client again
----