You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/11/22 09:11:04 UTC
Slack digest for #general - 2018-11-22
2018-11-21 11:47:23 UTC - Ivan Kelly: have you any documentation on using this method? I sounds like each cert has to be publicly downloadable by the broker or something
----
2018-11-21 11:48:28 UTC - Ivan Kelly: ah, we don't support that now I don't think. Once the message leaves the client, it doesn't get modified by the broker
----
2018-11-21 12:45:10 UTC - Ganga Lakshmanasamy: So any suggestions here?
----
2018-11-21 12:46:16 UTC - Ganga Lakshmanasamy: @Ivan Kelly So is it mandatory that the message to be converted to byte array while sent through producer?
----
2018-11-21 12:47:43 UTC - Ivan Kelly: no, so if you use schema you can pass in objects
----
2018-11-21 12:48:08 UTC - Ivan Kelly: are you using java ?
----
2018-11-21 12:48:26 UTC - Ganga Lakshmanasamy: ok. I am looking up for some code samples/ documentation in converting current messagebuilder to typed builder
----
2018-11-21 12:48:31 UTC - Ganga Lakshmanasamy: yes, I am using java
----
2018-11-21 12:49:43 UTC - Ganga Lakshmanasamy: The main problem is we have a map object which needs to be sent and parsed back to map or jsonobject to pass it on to one of our rest api, which is failing during parse.
----
2018-11-21 12:50:36 UTC - Ivan Kelly: so, it's something like
try (Producer<MyObject> p = client.newProducer(Schema.AVRO(MyObject.class).topic("my-topic").create()) {
p.newMessage().value(new MyObject()).send();
----
2018-11-21 12:51:10 UTC - Ivan Kelly: or, you can skip the builder completely, and just use p.send(new MyObject());
----
2018-11-21 13:07:12 UTC - Ganga Lakshmanasamy: ok Thanks, let me check this out.
----
2018-11-21 15:12:23 UTC - Ryan Samo: I’m struggling to find the exact example but this link shows the use of a certificate chain with Jetty. It explains sort of what I’m talking about. Basically everything relies on the root like normal but the clients would rely on the intermediate layer allowing you to revoke the intermediate without changing the root.
<https://support.cloudbees.com/hc/en-us/articles/222829768-How-to-setup-HTTPS-within-Jetty-with-Intermediate-certificates-?mobile_site=true>
----
2018-11-21 15:40:39 UTC - Ivan Kelly: well, how many requests do you think you'll have per second?
----
2018-11-21 15:40:49 UTC - Ivan Kelly: what size machine is this? sounds massive
----
2018-11-21 16:15:45 UTC - Christophe Bornet: Small question about sync replication. Suppose I have 1 cluster over 2 datacenters, E=4, Qw=2, Qa=2 and region aware policy. In nominal mode, Pulsar will distribute the data over the 2 datacenters, is that correct ? Now if one of the DCs crashes, can I assume that Pulsar will redistribute the fragments over the remaining DC ? Will I lose messages in the process ? If the failed DC comes back, are the fragment redistributed over the 2 DCs immediately ?
----
2018-11-21 16:22:10 UTC - Matteo Merli: > In nominal mode, Pulsar will distribute the data over the 2 datacenters, is that correct ?
That’s correct, each DC is viewed as a rack and the replicas will be placed in order to maximize the distribution.
> Now if one of the DCs crashes, can I assume that Pulsar will redistribute the fragments over the remaining DC ?
Are you referring to the existing data or new data published after 1 DC is offline?
* Existing data : BK auto-recovery will re-establish the configured replication factor (in this case `Qw=2`) and, since DC1 is down, it will place the 2 copies in DC2
* New data will have both copies be placed in DC2 immediately
> Will I lose messages in the process ?
No, unless you loose 2 machines (not only they’re offline, but either “gone” or disk broke down)
> If the failed DC comes back, are the fragment redistributed over the 2 DCs immediately ?
Yes, existing data from failed DC will be immediately available and new data will start to be written there again
----
2018-11-21 16:28:56 UTC - Christophe Bornet: > No, unless you loose 2 machines (not only they’re offline, but either “gone” or disk broke down)
You mean loose 2 machines from DC2 in addition to the DC1 failure ? Or the 2 machines from DC1 (I'm trying to imagine, DC1 got nuked)
----
2018-11-21 16:34:11 UTC - Matteo Merli: In normal condition, with `Qw=2`, the 2 replicas will be place 1 in DC1 and the other in DC2.
If DC1 is offline, data will still be there, just unavailable (for some extended amount of time). If DC1 is nuked.. well we assume the disk is broken.
Now we have 1 copy in DC2.
* If this bookie goes down --> Data will be temporarily unavailable
* If this bookie disk crashes --> Data will be either unavail or lost (depending on DC1)
BK auto-recovery will start to copy data to have 2 copies again, in this case by writing to another bookie in DC2. This process will start immediately after DC1 goes offline (or with a configurable delay) .
----
2018-11-21 16:36:18 UTC - Christophe Bornet: This is just so cool :sunglasses: !
----
2018-11-21 16:37:02 UTC - Christophe Bornet: > new data will start to be written there again
----
2018-11-21 16:37:39 UTC - Christophe Bornet: you mean that new data is again distributed automatically on both DCs, right ?
----
2018-11-21 16:38:40 UTC - Matteo Merli: That’s correct, new ledgers will always consider all the available bookies when forming an ensemble (and then applying the placement policy).
----
2018-11-21 16:43:07 UTC - Christophe Bornet: Awesome ! You can't do such things with monolithic partitions if you see what I mean :smile:
----
2018-11-21 16:43:27 UTC - Matteo Merli: :smile:
----
2018-11-21 18:33:46 UTC - Adam Kazberuk: @Adam Kazberuk has joined the channel
----
2018-11-21 19:35:54 UTC - Adam Kazberuk: This is likely a configurational oversight on my end (still new & ignorant), hoping someone can help me out!
Trying to set up a DeadLetterQueue on a websocket consumer, looks like the functionality was added fairly recently: <https://github.com/apache/pulsar/pull/2968>
I've got a consumer connecting and receiving messages (with DLQ params on connect), and the un-acked messages are re-sent after 10s. However, the messages never stop being retried.
I see this in the logs for the consumer connection:
`11:04:34.411 [pulsar-web-357] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [21/Nov/2018:11:04:34 -0800] "GET /ws/v2/consumer/persistent/public/default/my-topic/subscription-name?ackTimeoutMillis=10000&maxRedeliverCount=1&deadLetterTopic=dead HTTP/1.1" 101 0 "-" "-" 85`
with `maxRedeliverCount=1&deadLetterTopic=dead` which appears to be the correct parameters (I've tried various params, i.e. 'maxRedeliverCount=5'): <https://github.com/apache/pulsar/pull/2968/files#diff-9de5d3e81a341a0af69fd50e24aea480R316>
This appears on each message retry:
```
11:14:14.420 [pulsar-timer-74-1] WARN org.apache.pulsar.client.impl.UnAckedMessageTracker - [ConsumerBase{subscription='subscription-name', consumerName='c2e41', topic='<persistent://public/default/my-topic>'}] 1 messages have timed-out
11:14:14.420 [broker-topic-workers-OrderedScheduler-7-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/default/persistent/my-topic-subscription-name] Rewind from 659:1 to 659:0
```
----
2018-11-21 19:50:10 UTC - Matteo Merli: @Adam Kazberuk Support for DLQ in websocket proxy was just added recently <https://github.com/apache/pulsar/pull/2968> and will be out on 2.3.0
----
2018-11-21 19:54:22 UTC - Adam Kazberuk: @Matteo Merli, solid, thank you!
----
2018-11-21 22:40:39 UTC - Byron: Hi folks, is there a download available for a pre-built C++ library (with headers) so I can link against it for the Go library?
----
2018-11-21 22:42:03 UTC - Ali Ahmed: no prebuilt but a brew formulae to install the C++ lib is present
----
2018-11-21 22:42:43 UTC - Byron: My macOS version isn’t supported anymore so I would like to build a linux Docker image to build my Go code
----
2018-11-21 22:44:00 UTC - Byron: does the pulsar docker image contain the C++ library? an ideal image would be all stripped down with just that and the dependencies
----
2018-11-21 22:44:06 UTC - Matteo Merli: @Byron you can use the formula to build it yourself. Have you tried: <http://pulsar.apache.org/docs/en/client-libraries-cpp/#macos>
----
2018-11-21 22:45:31 UTC - Byron: I did, but my macOS version is too old 10.11 according to Homebrew
----
2018-11-21 22:45:59 UTC - Byron: Failing on this
```
==> ./bootstrap --prefix=/usr/local/Cellar/cmake/3.12.4 --no-system-libs --parallel=4 --datadir=/share/cm
==> make
Last 15 lines from /Users/ruthb/Library/Logs/Homebrew/cmake/02.make:
[ 25%] Building C object Utilities/cmliblzma/CMakeFiles/cmliblzma.dir/liblzma/simple/simple_decoder.c.o
[ 25%] Building C object Utilities/cmlibuv/CMakeFiles/cmlibuv.dir/src/unix/proctitle.c.o
[ 25%] Building C object Source/CursesDialog/form/CMakeFiles/cmForm.dir/fty_ipv4.c.o
[ 26%] Building C object Utilities/cmliblzma/CMakeFiles/cmliblzma.dir/liblzma/simple/simple_encoder.c.o
[ 26%] Building C object Source/CursesDialog/form/CMakeFiles/cmForm.dir/fty_num.c.o
[ 26%] Building C object Source/CursesDialog/form/CMakeFiles/cmForm.dir/fty_regex.c.o
[ 26%] Linking C static library libcmlibuv.a
[ 26%] Building C object Utilities/cmliblzma/CMakeFiles/cmliblzma.dir/liblzma/simple/sparc.c.o
[ 26%] Built target cmlibuv
[ 26%] Building C object Utilities/cmliblzma/CMakeFiles/cmliblzma.dir/liblzma/simple/x86.c.o
[ 27%] Linking C static library libcmForm.a
[ 27%] Linking C static library libcmliblzma.a
[ 27%] Built target cmForm
[ 27%] Built target cmliblzma
make: *** [all] Error 2
```
----
2018-11-21 22:46:17 UTC - Byron: And then Homebrew outputs this:
```
Do not report this issue to Homebrew/brew or Homebrew/core!
These open issues may also help:
cmake 3.13.0 <https://github.com/Homebrew/homebrew-core/pull/34361>
swi-prolog 7.7.20 cmake based build <https://github.com/Homebrew/homebrew-core/pull/33706>
Error: You are using macOS 10.11.
We (and Apple) do not provide support for this old version.
```
----
2018-11-21 22:47:16 UTC - Matteo Merli: Got it
----
2018-11-21 22:47:26 UTC - Byron: However I see packages for Linux are available now: <http://pulsar.apache.org/docs/en/client-libraries-cpp/#linux>
----
2018-11-21 22:47:32 UTC - Ali Ahmed: @Byron just follow the shell commands in the brew itself
<https://raw.githubusercontent.com/apache/pulsar/master/pulsar-client-cpp/homebrew/libpulsar.rb>
----
2018-11-21 22:48:09 UTC - Byron: So I can create a Docker image installing those packages with header to be able to link the Go library.
----
2018-11-21 22:48:24 UTC - Byron: Thanks @Ali Ahmed I will try it manually to see if that changes anything
----
2018-11-21 22:48:44 UTC - Ali Ahmed: were are targeting to run osx or linux ?
----
2018-11-21 22:49:29 UTC - Ali Ahmed: the python packages should install the native c++ libs if you use pip or easy install
----
2018-11-21 22:49:45 UTC - Byron: Oh ok, good to know. I am targeting linux, but developing on a mac
----
2018-11-21 22:51:08 UTC - Matteo Merli: @Ali Ahmed python are self-contained, but Go needs the library to be there
----
2018-11-21 22:51:08 UTC - Byron: So that is why I can do a Docker-based workflow if need be
----
2018-11-21 22:51:24 UTC - Byron: I will install the client-devel debian package?
----
2018-11-21 22:51:27 UTC - Byron: That should be good?
----
2018-11-21 22:51:30 UTC - Matteo Merli: yes
----
2018-11-21 22:51:35 UTC - Byron: :+1:
----
2018-11-21 22:52:52 UTC - Byron: Thank you all
+1 : Matteo Merli
----
2018-11-21 23:00:03 UTC - Byron: @Matteo Merli Sorry one more issue if you are available
----
2018-11-21 23:00:28 UTC - Byron: Here is my Docker image:
```
FROM golang:1.11
COPY apache-pulsar-client-2.2.0.deb /opt/apache-pulsar-client.deb
COPY apache-pulsar-client-dev-2.2.0.deb /opt/apache-pulsar-client-dev.deb
RUN dpkg -i /opt/apache-pulsar-client.deb && \
dpkg -i /opt/apache-pulsar-client-dev.deb && \
apt-get install -f
ENV CGO_ENABLED=1
RUN go get <http://github.com/apache/pulsar/pulsar-client-go/pulsar|github.com/apache/pulsar/pulsar-client-go/pulsar>
```
----
2018-11-21 23:00:47 UTC - Byron: Getting this error on build:
```
# <http://github.com/apache/pulsar/pulsar-client-go/pulsar|github.com/apache/pulsar/pulsar-client-go/pulsar>
src/github.com/apache/pulsar/pulsar-client-go/pulsar/c_client.go:232:3: could not determine kind of name for C.pulsar_string_list_free
src/github.com/apache/pulsar/pulsar-client-go/pulsar/c_client.go:229:31: could not determine kind of name for C.pulsar_string_list_get
src/github.com/apache/pulsar/pulsar-client-go/pulsar/c_client.go:226:24: could not determine kind of name for C.pulsar_string_list_size
src/github.com/apache/pulsar/pulsar-client-go/pulsar/c_client.go:220:78: could not determine kind of name for C.pulsar_string_list_t
cgo:
gcc errors for preamble:
In file included from src/github.com/apache/pulsar/pulsar-client-go/pulsar/c_client.go:24:0:
./c_go_pulsar.h:133:66: error: unknown type name 'pulsar_string_list_t'
void pulsarGetTopicPartitionsCallbackProxy(pulsar_result result, pulsar_string_list_t* partitions, void *ctx);
^~~~~~~~~~~~~~~~~~~~
./c_go_pulsar.h: In function '_pulsar_client_get_topic_partitions':
./c_go_pulsar.h:137:61: error: 'pulsarGetTopicPartitionsCallbackProxy' undeclared (first use in this function)
pulsar_client_get_topic_partitions_async(client, topic, pulsarGetTopicPartitionsCallbackProxy, ctx);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./c_go_pulsar.h:137:61: note: each undeclared identifier is reported only once for each function it appears in
```
----
2018-11-21 23:01:03 UTC - Matteo Merli: You should get the 2.2.20 tag
----
2018-11-21 23:01:27 UTC - Matteo Merli: master code has already some changes for 2.3 release (you’d need to compile the master c++ lib)
----
2018-11-21 23:01:35 UTC - Matteo Merli: (2.2.0 tag)
----
2018-11-21 23:01:57 UTC - Byron: For the Go library. Ok
----
2018-11-21 23:02:28 UTC - Matteo Merli: Go tag needs to be <= C++ tag
----
2018-11-21 23:02:38 UTC - Byron: Ok
----
2018-11-21 23:02:40 UTC - Byron: Thanks
----
2018-11-21 23:03:47 UTC - Matteo Merli: (I wonder is there’s a way to give more meaningful errors there)
----
2018-11-22 03:37:04 UTC - Zheng Dayu: @Zheng Dayu has joined the channel
----
2018-11-22 04:45:45 UTC - Abhishek Mulay: @Abhishek Mulay has joined the channel
----
2018-11-22 08:13:12 UTC - zero: @zero has joined the channel
----