You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/08/03 09:11:03 UTC
Slack digest for #general - 2020-08-03

2020-08-02 11:09:37 UTC - Allen ONeill: Thanks @Jerry Peng that looks very promising .... I understand the function code part and how to upload/deploy that - where/how do we upload/deploy the YAML file that defines the window itself?
----
2020-08-02 11:22:01 UTC - Allen ONeill: Also, my understanding is that the function operates as normal, except, because its flagged as a window function (to pulsar internals), then its output is stored and only returns (its output) when the window criteria is met - is this correct?
----
2020-08-02 17:37:46 UTC - Brandon Whitehead: @Brandon Whitehead has joined the channel
----
2020-08-02 17:38:07 UTC - Brandon Whitehead: hello
----
2020-08-02 17:40:37 UTC - Brandon Whitehead: I'm trying to download pulsar through docker, but for some reason the docker install isn't working cause I don't have windows 10 home prerequisite. But I checked my settings and I actually do, so I'm at a loss as to what to do
----
2020-08-02 17:51:48 UTC - vipul12: @vipul12 has joined the channel
----
2020-08-02 18:09:31 UTC - Brandon Whitehead: this is the error I get on the docker installer
----
2020-08-02 18:09:33 UTC - Brandon Whitehead: Docker Desktop requires Windows 10 Pro/Enterprise (15063+) or Windows 10 Home (19018+).
----
2020-08-02 18:33:57 UTC - Adriaan de Haan: This is starting to get me down... I can't find what is causing this failure in ACK's - and it seems to be very predictable. Running at 800 messages per second, it's increasing at about 1 every 3 seconds.
----
2020-08-02 18:50:58 UTC - Adriaan de Haan: Actually, irrespective of speed - if there is any messages being sent, the backlog grows at about that that rate - 1 ever 3 seconds.
----
2020-08-02 18:50:59 UTC - Adriaan de Haan: 
----
2020-08-02 19:53:45 UTC - Brandon Whitehead: okay, so I finally installed docker on my windows
----
2020-08-02 19:53:57 UTC - Brandon Whitehead: how do I install the pulsar in docker?
----
2020-08-02 19:54:13 UTC - Brandon Whitehead: I'm using the instructions from here
----
2020-08-02 19:54:14 UTC - Brandon Whitehead: <https://pulsar.apache.org/docs/ja/standalone-docker/>
----
2020-08-02 19:54:28 UTC - Brandon Whitehead: but the terminal wont take this format
----
2020-08-02 21:04:14 UTC - Jerry Peng: &gt;  how to upload/deploy that - where/how do we upload/deploy the YAML file that defines the window itself?
It is part of the normal function config file that can be submit like the following:
```./bin/pulsar-admin functions create --function-config-file my-function.yaml```
&gt; Also, my understanding is that the function operates as normal, except, because its flagged as a window function (to pulsar internals), then its output is stored and only returns (its output) when the window criteria is met - is this correct?
Yes, the function is triggered (i.e. process()) is called) when the window criteria is met.  Here is also another resource that explains window functions:

<https://www.splunk.com/en_us/blog/it/event-processing-design-patterns-with-pulsar-functions.html>
----
2020-08-02 21:25:54 UTC - Anticia: @Anticia has joined the channel
----
2020-08-02 23:35:42 UTC - Sijie Guo: Yes. ackTimeout will result in duplicates. If the consumer is not crash, consumer maintains ack state so it will use it to ignore already acked messages. If the consumer crashes, the messages will be delivered and you might process duplicated messages.
----
2020-08-03 00:24:32 UTC - Axel Sirota: Hey all! Anyone knows how to get over this error in pulsar sql?

```2020-08-03T00:18:48.088Z	ERROR	bookkeeper-io-7-1	org.apache.bookkeeper.proto.PerChannelBookieClient	Could not connect to bookie: [id: 0xd1a4ab96, L:/127.0.0.1:41112]/127.0.0.1:3181, current state CONNECTING :
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /127.0.0.1:3181
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused

2020-08-03T00:18:48.093Z	ERROR	BookKeeperClientWorker-OrderedExecutor-3-0	org.apache.bookkeeper.client.ReadLastConfirmedOp	While readLastConfirmed ledger: 939 did not hear success responses from all quorums
2020-08-03T00:18:48.103Z	ERROR	Query-20200803_001847_00001_ejpqg-179	org.apache.pulsar.sql.presto.PulsarSplitManager	Failed to get splits
org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.apache.bookkeeper.client.BKException$BKReadException: Error while reading ledger
Caused by: java.util.concurrent.CompletionException: org.apache.bookkeeper.client.BKException$BKReadException: Error while reading```
I am trying to run Pulsar SQL in a separate container from the Pulsar container and both are attached to the same docker network :slightly_smiling_face: Should I expose the Bookeeper port from the pulsar container so SQL can read it? Maybe it appears to be it, but I don’t know which Bk is talking about since <https://pulsar.apache.org/docs/en/sql-deployment-configurations/|in these docs> there is a reference to zookeeper and broker uris but not BK uris hahah

Any help is welcome and deeply appreciated! :slightly_smiling_face:
----
2020-08-03 00:36:42 UTC - Axel Sirota: I think the issue is that Pulsar SQL is assuming that the bookkeeper Uri is `127.0.0.1:3181`  while, in my deployment is `pulsar-host:3181` . If only I could find what configuration to change to set this up as I have with zookeeper
----
2020-08-03 01:29:12 UTC - xue: <https://github.com/apache/pulsar/issues/7706>
Pulsar Proxy No active broker is available
----
2020-08-03 05:21:12 UTC - thinker0: pulsar-beat-output filebeat Skew ???

<https://github.com/streamnative/pulsar-beat-output/tree/master/filebeat>
----
2020-08-03 05:23:36 UTC - Sijie Guo: Hmm. Sound like a bug. Maybe create an issue in the pulsar-beat-output repo? So @tuteng can take a look.
slightly_smiling_face : thinker0
----
2020-08-03 05:25:50 UTC - thinker0: <https://github.com/streamnative/pulsar-beat-output/issues/24>
----
2020-08-03 05:38:29 UTC - Jerry Peng: Yes the zookeeper and bookkeeper profs need to be exposed
----
2020-08-03 05:38:41 UTC - Sijie Guo: @thinker0 thank you
----
2020-08-03 05:40:47 UTC - Jerry Peng: The configurations for the connector are here:

<https://github.com/apache/pulsar/blob/master/conf/presto/catalog/pulsar.properties|https://github.com/apache/pulsar/blob/master/conf/presto/catalog/pulsar.properties>
----