You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/02/13 09:11:04 UTC
Slack digest for #general - 2020-02-13
2020-02-12 10:33:32 UTC - Sijie Guo: maybe create an issue and write down the exact command that you know. Otherwise it is hard for people in the community to help.
----
2020-02-12 10:37:03 UTC - Sijie Guo: > ny lines besides authenticationEnabled, authorizationEnabled, and authenticationProviders?
it depends on which authentication and authorization providers are you using.
You can check out the security documentation: <http://pulsar.apache.org/docs/en/security-overview/>
> when I set those options, pulsar just shuts down, before I have a chance to run any pulsar-admin commands
if you are running standalone, you can checkout the log to see what are the errors and you can report the error to github. so people in the community can help.
----
2020-02-12 10:54:41 UTC - rmb: both log files are empty
----
2020-02-12 11:08:59 UTC - rmb: the command I'm running is
`docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:2.5.0 bin/pulsar standalone`
----
2020-02-12 11:42:09 UTC - Abhilash Mandaliya: well, it didn’t work properly. Now I am trying with uber jar option. If we fo with that way, do we need to change anything in config? Because connector is not starting even after uber jar being present under connectors directory
@Ali Ahmed or anyone else can you put some light on this?
----
2020-02-12 13:12:34 UTC - Naby: Thank you @Ming. Thank you @Sijie Guo.
----
2020-02-12 15:31:41 UTC - Ruslan Sheremet: @rmb You are have replaced all `/pulsar/conf` with your local version dir in `pulsarconf` volume. Need to replace only specific configs, not all dir.
----
2020-02-12 15:45:48 UTC - rmb: I don't follow; what docker command should I be running? that command was taken from <https://pulsar.apache.org/docs/en/standalone-docker/>; I didn't alter any config files before running it.
----
2020-02-12 17:08:52 UTC - tcourdy: anyone know off the top of their head which is correct in terms of the kafka connnector config property name the doc states `ack` the example config shows `acks` . You can see the discrepancy here: <https://pulsar.apache.org/docs/en/io-kafka-sink/#configuration>
----
2020-02-12 17:12:59 UTC - Ruslan Sheremet: @rmb I mean if you have already configs in your local volume `pulsarconf` , this configs will override configs in container. So if you want override some of the config, you should define exect path to it like:
----
2020-02-12 17:13:56 UTC - Ruslan Sheremet: ```-v ./pulsar/conf/standalone.conf:/pulsar/conf/standalone.conf```
----
2020-02-12 17:15:09 UTC - Jordan Pilat: Looks like `acks`, which would match the standard name for it in Kafka itself:
<https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaSinkConfig.java#L52>
----
2020-02-12 17:22:21 UTC - tcourdy: cool thanks
----
2020-02-12 17:22:50 UTC - Sijie Guo: `acks`. I think that is a typo in the configuration section. Anyone interested in submitting a pull request?
heavy_check_mark : Jordan Pilat
----
2020-02-12 17:59:52 UTC - Neng Lu: Hi, does anyone have problem running `mvn test` for the pulsar repo? Mine constantly failed..
----
2020-02-12 18:07:24 UTC - Neng Lu: The error message are as follows:
```[INFO] Apache Pulsar :: Tiered Storage :: JCloud .......... FAILURE [ 5.348 s]
...
...
[ERROR] An error occurred while instantiating class org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloaderTest: Cannot create a new instance of test class class org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloaderTest
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException There was an error in the forked process
[ERROR]
...
...```
----
2020-02-12 18:07:40 UTC - Neng Lu: does anyone happen to know the problem?
----
2020-02-12 18:09:12 UTC - Jordan Pilat: I can today, seems like a good first contribution :stuck_out_tongue_winking_eye:
----
2020-02-12 18:42:26 UTC - Jordan Pilat: Is:
<https://github.com/apache/pulsar/blob/master/site2/website/versioned_docs/version-2.5.0/io-kafka-sink.md>
generated from:
<https://github.com/apache/pulsar/blob/master/site2/docs/io-kafka-sink.md>
...or are they 2 separate changes?
----
2020-02-12 18:46:48 UTC - Jordan Pilat: Never mind, I see at <https://github.com/jrpilat/pulsar/tree/master/site2> they are indeed generated
----
2020-02-12 18:59:19 UTC - Jordan Pilat: Though, since this is a 'bug fix' to the documentation, should I edit the 2.5.0 versioned copy as well?
----
2020-02-12 19:03:19 UTC - Aditya Vishwakarma: We are doing a POC with 1000 msgs/sec to check for any issues. You suggestions helpful. Thanks.
I was seeing segmentation faults while trying to out a python app. I was wondering if these are isolated issues just for me, or is the python client is mature enough.
----
2020-02-12 19:07:41 UTC - Jordan Pilat: Is that the full error message?
----
2020-02-12 19:27:58 UTC - Sijie Guo: yes. it is a ‘bug fix’. so you might fix the typo in the old versions documentation as well.
----
2020-02-12 19:37:28 UTC - Joshua Dunham: Hey Sijieg, I've looked at the k8s behavior of the system. Is it possible to use the bookkeeper container instead of the apachepulsar/pulsar to start a bookkeeper quorum?
----
2020-02-12 19:45:15 UTC - Jordan Pilat: OK to assign <https://github.com/apache/pulsar/issues/6312> to myself and submit a PR?
----
2020-02-12 19:45:44 UTC - Sijie Guo: done
thanks : Jordan Pilat
----
2020-02-12 20:01:09 UTC - Sijie Guo: Sorry that I missed this thread before.
> A BK cluster on a remote machine accessible over network. How do I configure pulsar to use that?
Yes. If the bookkeeper started using the same zookeeper cluster as pulsar broker, and the ledgers root path is “/ledgers”. You don’t need special configurations.
If you using a different zookeeper cluster or different ledgers path, it requires some code changes. I believe that change was merged to master and will be released in future releases.
> I’ve looked at the k8s behavior of the system. Is it possible to use the bookkeeper container instead of the apachepulsar/pulsar to start a bookkeeper quorum?
yes you can change the container to use bookkeeper container.
----
2020-02-12 20:19:02 UTC - Sijie Guo: Are you able to run `mvn -pl tiered-storage/jcloud/ test` ?
----
2020-02-12 20:21:39 UTC - Neng Lu: @Jordan Pilat the full error message is very long, I extracted the most informative one.
----
2020-02-12 20:21:51 UTC - Neng Lu: @Sijie Guo Thanks, let me try the cmd
----
2020-02-12 20:23:54 UTC - Neng Lu: Yes, the cmd works for me
----
2020-02-12 20:34:12 UTC - Joshua Dunham: Is it possible to change the ZK path that pulsar looks for ~ /bookkeeper/ledgers. We have a enterprise managed zookeeper ensemble. Not one dedicated..although that's not unreasonable to stand up a pulsar specific zookeeper.
----
2020-02-12 20:45:29 UTC - David Kjerrumgaard: @rmb To enable JWT authentication, you will need at add the following properties in the conf/standalone.conf file: `authenticationEnabled=true` , `authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken`, and `tokenPublicKey=file:///PATH TO /my-public.key`
----
2020-02-12 20:46:34 UTC - David Kjerrumgaard: The last one allows the broker to validate the JWT tokens it receives.
----
2020-02-12 20:53:01 UTC - Sijie Guo: > both log files are empty
How to you start the standalone? Are you starting it in foreground?
----
2020-02-12 20:53:05 UTC - Joshua Dunham: Just reread your comment above. Looking forward to customizable ZK Ledgers path. Thank you!
----
2020-02-12 20:54:43 UTC - David Kjerrumgaard: I have a working Docker image that you can use to test JWT authentication with. In fact, it supports BOTH TLS and JWT. Anyways, it can serve as a good reference. <https://github.com/david-streamlio/pulsar-in-action/tree/master/docker-images/pulsar-standalone>
----
2020-02-12 23:46:57 UTC - rmb: Hi David, those are the options I added; pulsar shuts down immediately.
I started standalone as a daemon and got a log file; I posted an issue at <https://github.com/apache/pulsar/issues/6309> with a copy of the log file, but the errors are things like
> [main] ERROR org.apache.pulsar.functions.worker.WorkerService - Error Starting up in worker
> org.apache.pulsar.client.admin.PulsarAdminException$NotAuthorizedException : HTTP 401 Unauthorized
and
> java.lang.NoClassDefFoundError: Could not initialize class io.netty.channel.epoll.EpollEventLoop
----
2020-02-13 00:32:21 UTC - Ming: Python lib uses C++ library underneath. You might want to raise a GH issue with this segmentation fault. It should not have crashed. Even with 1000 msg/s with 5MB, the throughput is still significant but definitely doable in Pulsar.
----
2020-02-13 01:08:08 UTC - Sijie Guo: @rmb: I see. It failed related to function worker authentication. there are two options:
a) disable function worker in your standalone first. You can do it by running `bin/pulsar standalone -nfw -nss`.
b) you can configure conf/standalone.conf to enable authentication for broker client (<https://github.com/apache/pulsar/blob/master/conf/broker.conf#L405>). That is the one used by function worker contacting to brokers.
----
2020-02-13 03:07:59 UTC - Eugen: If anyone has experience with autoscaling Pulsar, I'd like to hear! How did you do it, and how did it work out for you?
----
2020-02-13 03:34:24 UTC - Addison Higham: We haven't done it yet, but the broker tier on top of k8s with pod autoscaler and the built in metrics seems pretty straight forward and a bit of research we have done seems to support it. Bookkeeper is a bit trickier, at least scale down actions
thanks : Eugen, Sijie Guo
+1 : David Kjerrumgaard, Ali Ahmed, Sijie Guo
----
2020-02-13 03:35:20 UTC - Addison Higham: Our plan on BK for now is to give us really monitoring and over provision a bit so we can hopefully capacity plan around it
----
2020-02-13 03:40:15 UTC - Sylvia: @Sylvia has joined the channel
----
2020-02-13 07:41:14 UTC - Kelvin Sajere: Any ideas ?
If i open Apache pulsar manager /management/clusters. It keeps loading forever. This is the error am getting in my console
```{"timestamp":"2020-02-13T07:33:32.180+0000","status":500,"error":"Internal Server Error","message":"No message available","path":"/pulsar-manager/admin/v2/clusters"}```
Apache pulsar manager logs:
----
2020-02-13 07:41:14 UTC - Kelvin Sajere: ``` docker logs -f hungry_matsumoto
Starting PostGreSQL Server
The files belonging to this database system will be owned by user "pulsar".
This user must also own the server process.
The database cluster will be initialized with locales
COLLATE: C
CTYPE: C.UTF-8
MESSAGES: C
MONETARY: C
NUMERIC: C
TIME: C
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /data/postgresql ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... sh: locale: not found
2020-02-13 07:26:57.117 UTC [20] WARNING: no usable system locales were found
ok
syncing data to disk ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success.
waiting for server to start....2020-02-13 07:26:57.834 UTC [24] LOG: listening on IPv4 address "127.0.0.1", port 5432
2020-02-13 07:26:57.834 UTC [24] LOG: could not bind IPv6 address "::1": Address not available
2020-02-13 07:26:57.834 UTC [24] HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2020-02-13 07:26:57.836 UTC [24] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432"
2020-02-13 07:26:57.838 UTC [24] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2020-02-13 07:26:57.849 UTC [25] LOG: database system was shut down at 2020-02-13 07:26:57 UTC
2020-02-13 07:26:57.850 UTC [24] LOG: database system is ready to accept connections
done
server started
ALTER ROLE
CREATE DATABASE
GRANT
You are now connected to database "pulsar_manager" as user "pulsar".
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
Starting Pulsar Manager Front end
Starting Pulsar Manager Back end
Start servie no enable JWT.
2020-02-13 07:26:58,249 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2020-02-13 07:26:58,252 INFO supervisord started with pid 43
2020-02-13 07:26:59,258 INFO spawned: 'pulsar-manager-backend' with pid 45
2020-02-13 07:27:00,335 INFO success: pulsar-manager-backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-13 07:30:28.371 UTC [115] ERROR: LIMIT #,# syntax is not supported at character 38
2020-02-13 07:30:28.371 UTC [115] HINT: Use separate LIMIT and OFFSET clauses.
2020-02-13 07:30:28.371 UTC [115] STATEMENT: SELECT name,broker FROM environments LIMIT $1, $2
2020-02-13 07:31:28.636 UTC [115] ERROR: LIMIT #,# syntax is not supported at character 38
2020-02-13 07:31:28.636 UTC [115] HINT: Use separate LIMIT and OFFSET clauses.
2020-02-13 07:31:28.636 UTC [115] STATEMENT: SELECT name,broker FROM environments LIMIT $1, $2
2020-02-13 07:32:28.712 UTC [115] ERROR: LIMIT #,# syntax is not supported at character 38
2020-02-13 07:32:28.712 UTC [115] HINT: Use separate LIMIT and OFFSET clauses.
2020-02-13 07:32:28.712 UTC [115] STATEMENT: SELECT name,broker FROM environments LIMIT $1, $2
2020-02-13 07:33:28.783 UTC [115] ERROR: LIMIT #,# syntax is not supported at character 38
2020-02-13 07:33:28.783 UTC [115] HINT: Use separate LIMIT and OFFSET clauses.
2020-02-13 07:33:28.783 UTC [115] STATEMENT: SELECT name,broker FROM environments LIMIT $1, $2
2020-02-13 07:34:28.835 UTC [115] ERROR: LIMIT #,# syntax is not supported at character 38
2020-02-13 07:34:28.835 UTC [115] HINT: Use separate LIMIT and OFFSET clauses.
2020-02-13 07:34:28.835 UTC [115] STATEMENT: SELECT name,broker FROM environments LIMIT $1, $2
2020-02-13 07:35:28.909 UTC [115] ERROR: LIMIT #,# syntax is not supported at character 38
2020-02-13 07:35:28.909 UTC [115] HINT: Use separate LIMIT and OFFSET clauses.
2020-02-13 07:35:28.909 UTC [115] STATEMENT: SELECT name,broker FROM environments LIMIT $1, $2
2020-02-13 07:36:29.010 UTC [115] ERROR: LIMIT #,# syntax is not supported at character 38
2020-02-13 07:36:29.010 UTC [115] HINT: Use separate LIMIT and OFFSET clauses.
2020-02-13 07:36:29.010 UTC [115] STATEMENT: SELECT name,broker FROM environments LIMIT $1, $2
2020-02-13 07:37:29.072 UTC [115] ERROR: LIMIT #,# syntax is not supported at character 38
2020-02-13 07:37:29.072 UTC [115] HINT: Use separate LIMIT and OFFSET clauses.
2020-02-13 07:37:29.072 UTC [115] STATEMENT: SELECT name,broker FROM environments LIMIT $1, $2
2020-02-13 07:38:29.111 UTC [115] ERROR: LIMIT #,# syntax is not supported at character 38
2020-02-13 07:38:29.111 UTC [115] HINT: Use separate LIMIT and OFFSET clauses.
2020-02-13 07:38:29.111 UTC [115] STATEMENT: SELECT name,broker FROM environments LIMIT $1, $2
2020-02-13 07:39:29.169 UTC [115] ERROR: LIMIT #,# syntax is not supported at character 38
2020-02-13 07:39:29.169 UTC [115] HINT: Use separate LIMIT and OFFSET clauses.
2020-02-13 07:39:29.169 UTC [115] STATEMENT: SELECT name,broker FROM environments LIMIT $1, $2
2020-02-13 07:40:29.224 UTC [115] ERROR: LIMIT #,# syntax is not supported at character 38
2020-02-13 07:40:29.224 UTC [115] HINT: Use separate LIMIT and OFFSET clauses.
2020-02-13 07:40:29.224 UTC [115] STATEMENT: SELECT name,broker FROM environments LIMIT $1, $2```
----
2020-02-13 07:41:14 UTC - Kelvin Sajere:
Apache pulsar : 2.4.2
Apache Pulsar Manager : v0.1.0
----
2020-02-13 08:43:50 UTC - Eugen: Haven't looked at the error, but the clusters tab keeps spinning for me (and my coworker as well) - we started in "standalone" mode and thought, that's probably why
----