You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/05/05 09:11:03 UTC
Slack digest for #general - 2019-05-05
2019-05-04 22:03:00 UTC - Nicolas Ha: Hello everyone :slightly_smiling_face: long time no see - I am reading this: <https://pulsar.apache.org/docs/en/sql-getting-started/>
Is it accessible from Java too?
----
2019-05-04 22:04:32 UTC - Ali Ahmed: you can use the presto jdbc client in java to run the queries
----
2019-05-04 22:05:00 UTC - Nicolas Ha: Oh great news - what should I put for the connection?
----
2019-05-04 22:05:45 UTC - Nicolas Ha: I mean - is there a default port? Auth?
----
2019-05-04 22:06:28 UTC - Ali Ahmed: it will be something like this
```jdbc:<presto://192.168.203.141:8080>```
----
2019-05-04 22:06:48 UTC - Ali Ahmed: there should presto jbbc doc that you can reference
----
2019-05-04 22:12:28 UTC - Nicolas Ha: thanks :slightly_smiling_face:
----
2019-05-04 22:12:52 UTC - Nicolas Ha: (I was looking for something in pulsar - that’s why I couldn’t find it :slightly_smiling_face: )
----
2019-05-04 22:34:42 UTC - Nicolas Ha: anyone knows if presto runs be default there: <https://hub.docker.com/r/apachepulsar/pulsar/tags> ?
(can’t find the source of that)
----
2019-05-04 22:35:29 UTC - Nicolas Ha: ah - spoke too soon, looks like <https://github.com/apache/pulsar/blob/master/docker/pulsar/Dockerfile> (which doesn’t seem to have it)
----
2019-05-04 22:41:07 UTC - Matteo Merli: @Nicolas Ha It’s included in that image, you just need to start it with `bin/pulsar sql-worker start`
----
2019-05-04 22:41:23 UTC - Matteo Merli: that will start a single node presto instance
----
2019-05-04 22:43:16 UTC - Nicolas Ha: I am trying to do that:```FROM apachepulsar/pulsar:latest
RUN bin/pulsar sql-worker start
```
=>```docker build .
Sending build context to Docker daemon 2.048kB
Step 1/2 : FROM apachepulsar/pulsar:latest
---> fd323a11901e
Step 2/2 : RUN bin/pulsar sql-worker start
---> Running in c35536078dde
-- Invalid command ‘sql-worker’ -- Use ‘bin/pulsar help’ to get a list of valid commands
The command ‘/bin/sh -c bin/pulsar sql-worker start’ returned a non-zero code: 1```
----
2019-05-04 22:44:00 UTC - Nicolas Ha: oh - I need to do it in the `command` perhaps
----
2019-05-04 22:44:44 UTC - Matteo Merli: can you try with `apachepulsar/pulsar:2.3.1` ?
----
2019-05-04 22:45:15 UTC - Nicolas Ha: 2.3.1 worked
----
2019-05-04 22:46:29 UTC - Nicolas Ha: ```docker build .
Sending build context to Docker daemon 2.048kB
Step 1/2 : FROM apachepulsar/pulsar:2.3.1
---> 30fb009acc89
Step 2/2 : RUN bin/pulsar sql-worker start
---> Running in f67d7d9ca1ac
Started as 23
Removing intermediate container f67d7d9ca1ac
---> 01f08af2b851
Successfully built 01f08af2b851```
----
2019-05-04 23:32:10 UTC - Nicolas Ha: ```presto> show catalogs;
ERROR: failed to open pager: Cannot run program "less": error=2, No such file or directory
Catalog
---------
pulsar
system
(2 rows)
Query 20190504_233044_00001_75ftd, FINISHED, 1 node
Splits: 19 total, 19 done (100.00%)
0:03 [0 rows, 0B] [0 rows/s, 0B/s]
presto> show schemas in pulsar;
Query 20190504_233103_00002_75ftd failed: Failed to get schemas from pulsar: Connection refused
```
^is the Connection refused expected?
----
2019-05-04 23:32:51 UTC - Nicolas Ha: (I ran the docker image, `bin/pulsar sql-worker run` in one terminal and `bin/pulsar sql` in the above)
----
2019-05-04 23:34:15 UTC - Nicolas Ha: I did it so because it seems I need to wait for the broker before starting `bin/pulsar sql-worker run` (not sure about that at the moment)
----
2019-05-04 23:35:21 UTC - Nicolas Ha: `show schemas in system;` works though, so it may (?) be expected?
----
2019-05-05 03:53:00 UTC - Jianfeng Qiao: Anyone ever tried valgrind on Pulsar broker?
----
2019-05-05 04:17:56 UTC - Sijie Guo: you mean running broker on valgrind?
----
2019-05-05 04:18:33 UTC - Sijie Guo: did you expose the port to outside of the docker container?
----
2019-05-05 06:10:04 UTC - Jianfeng Qiao: Yes
----
2019-05-05 06:10:37 UTC - Sijie Guo: I am not aware of anyone doing that yet.
----
2019-05-05 06:13:58 UTC - Jianfeng Qiao: I'm running openmessaging benchmark tool against pulsar. I find a lot of memory will be used by broker and in the end I get ' java.lang.OutOfMemoryError: Java heap space' error.
----
2019-05-05 06:14:37 UTC - Jianfeng Qiao: I suspect there might be memory leak somewhere.
----
2019-05-05 06:21:10 UTC - Sijie Guo: did you check if you give enough memory to broker?
----
2019-05-05 06:31:49 UTC - Jianfeng Qiao: -Djute.maxbuffer=10485760 -Djava.net.preferIPv4Stack=true -Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -Dio.netty.leakDetectionLevel=advanced -Dio.netty.recycler.maxCapacity.default=1000 -Dio.netty.recycler.linkCapacity=1024
----
2019-05-05 06:32:45 UTC - Jianfeng Qiao: All these configuration are by default, I do not change them.
----
2019-05-05 06:36:53 UTC - Sijie Guo: I see. it would be good to check if your bookie write latency. If you have a slow bookie, the add entries might be accumulating at your broker.
----
2019-05-05 06:44:53 UTC - Jianfeng Qiao: There is no thottling for add entries?
----
2019-05-05 07:00:53 UTC - Sijie Guo: there are settings at bookie side of throttling the add entries. I guess the throttling doesn’t kick in before broker OOM.
I am not sure about openmessaging client and how it configures the client. It might be worth checking what is the maxPendingRequests.
Alternatively, I would suggest u using pulsar-perf to get some baseline latency number.
----
2019-05-05 07:31:37 UTC - chenyongxi: @chenyongxi has joined the channel
----