You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/12/09 09:11:02 UTC
Slack digest for #general - 2019-12-09
2019-12-08 12:16:16 UTC - Nicolas Ha: Thanks :slightly_smiling_face: I don’t see a way to change the pulsar API port at the moment in a docker-compose setup, it seems hardcoded?
<https://github.com/apache/pulsar-manager/blob/713802827eb00214f295195de16f1c82351d0499/src/main/resources/application.properties#L59-L75>
----
2019-12-08 12:20:29 UTC - Nicolas Ha: for instance I use the port `8090` - I can’t seem to get the environment working
----
2019-12-08 12:24:14 UTC - tuteng: You can try test use command curl -v <http://127.0.0.1:8090/metrics/>
----
2019-12-08 15:58:59 UTC - Roman Popenov: I have run with additional debugging traces while running `cmake .` command
----
2019-12-08 16:00:32 UTC - Roman Popenov: Output
----
2019-12-08 16:19:38 UTC - Sijie Guo: @Roman Popenov can you file a github issue for this? @xiaolong.ran can help you with the c++ client build issue.
----
2019-12-08 17:10:55 UTC - Roman Popenov: Seems like I found a workaround
----
2019-12-08 17:12:32 UTC - Roman Popenov: I’ve added `python3` and `python37` in the `CMakeLists.txt` under `pulsar/pulsar-client-cpp`:
```if (PYTHONLIBS_VERSION_STRING MATCHES "^3.+$")
MESSAGE(STATUS "DETECTED Python 3")
string(REPLACE "." ";" PYTHONLIBS_VERSION_NO_LIST ${PYTHONLIBS_VERSION_STRING})
list(GET PYTHONLIBS_VERSION_NO_LIST 0 PYTHONLIBS_VERSION_MAJOR)
list(GET PYTHONLIBS_VERSION_NO_LIST 1 PYTHONLIBS_VERSION_MINOR)
set(BOOST_PYTHON_NAME_POSTFIX ${PYTHONLIBS_VERSION_MAJOR}${PYTHONLIBS_VERSION_MINOR})
# For python3 the lib name is boost_python3
set(BOOST_PYTHON_NAME_LIST python3;python3-mt;python-py${BOOST_PYTHON_NAME_POSTFIX};python${BOOST_PYTHON_NAME_POSTFIX}-mt;python${BOOST_PYTHON_NAME_POSTFIX})
else ()
# Regular boost_python
set(BOOST_PYTHON_NAME_LIST python3;python37;python;python-mt;python-py27;python27-mt;python27)
endif ()```
----
2019-12-08 17:13:48 UTC - Roman Popenov: It does require using `brew install boost-python3`
+1 : xiaolong.ran
----
2019-12-08 18:07:43 UTC - David Kjerrumgaard: What version of Python are you using?
----
2019-12-08 18:08:28 UTC - Roman Popenov: By default, python is 2.7, but I also have python3 installed throught brew
----
2019-12-08 18:09:12 UTC - Roman Popenov: For some reason, boost-python cannot be found found by cmake
----
2019-12-08 18:20:27 UTC - David Kjerrumgaard: I think your workaround is what I was going to suggest. :smiley:
----
2019-12-08 18:20:46 UTC - David Kjerrumgaard: Use Python 3 and use brew to install the missing lib
----
2019-12-08 18:23:38 UTC - Roman Popenov: Now I am off to the next issue:
```/usr/local/bin/python2 /Users/rp/PycharmProjects/PulsarDemo/Producer.py
Traceback (most recent call last):
File "/Users/rp/PycharmProjects/PulsarDemo/Producer.py", line 1, in <module>
import pulsar
File "build/bdist.macosx-10.14-x86_64/egg/pulsar/__init__.py", line 102, in <module>
File "build/bdist.macosx-10.14-x86_64/egg/_pulsar.py", line 7, in <module>
File "build/bdist.macosx-10.14-x86_64/egg/_pulsar.py", line 6, in __bootstrap__
ImportError: dynamic module does not define init function (init_pulsar)```
----
2019-12-08 18:28:43 UTC - David Kjerrumgaard: Can you share the code?
----
2019-12-08 18:29:25 UTC - David Kjerrumgaard: Does this code block work for you?
----
2019-12-08 18:29:33 UTC - Roman Popenov: I took it from here and modified slightly:
<https://github.com/streamlio/pulsar-python-tutorial/blob/master/README.md>
```import pulsar
TOPIC = '<persistent://test-tenant/test-ns/python-topic>'
PULSAR_SERVICE_URL = '<pulsar://localhost:6650>'
SUBSCRIPTION = 'my-subscription'
# Create a Pulsar client instance. The instance can be shared across multiple
# producers and consumers
client = pulsar.Client(PULSAR_SERVICE_URL)
# Create a producer on the topic. If the topic doesn't exist
# it will be automatically created
producer = client.create_producer(TOPIC)
for i in range(10):
content = 'hello-pulsar-%d' % i
# Publish a message and wait until it is persisted
producer.send(content)
print('Published message: "%s"' % content)
client.close()```
----
2019-12-08 18:29:48 UTC - Roman Popenov: It seems to be failing at the import statement
----
2019-12-08 18:30:35 UTC - David Kjerrumgaard: have you installed the client? `pip install pulsar-client==2.4.1`
----
2019-12-08 18:31:03 UTC - Roman Popenov: yes I have
----
2019-12-08 18:56:02 UTC - David Kjerrumgaard: weird. Can you confirm that the pip install was for your Python3 instance and not 2.7?
----
2019-12-08 18:56:24 UTC - Roman Popenov: It was for python27
----
2019-12-08 19:00:05 UTC - David Kjerrumgaard: Can you try running the code again with Python 3?
----
2019-12-08 19:00:24 UTC - David Kjerrumgaard: and using pip to install the lib in Python3
----
2019-12-08 20:48:10 UTC - Roman Popenov: Ok, so I am back to what I saw yesterday:
```python3 Producer.py
2019-12-08 15:47:28.440 INFO ConnectionPool:85 | Created connection for <pulsar://localhost:6650>
2019-12-08 15:47:28.447 ERROR ClientConnection:374 | [<none> -> <pulsar://localhost:6650>] Failed to establish connection: Connection refused
2019-12-08 15:47:28.447 INFO ClientConnection:1343 | [<none> -> <pulsar://localhost:6650>] Connection closed
2019-12-08 15:47:28.447 ERROR ClientImpl:182 | Error Checking/Getting Partition Metadata while creating producer on <persistent://test-tenant/test-ns/demo-topic> -- 5
2019-12-08 15:47:28.447 INFO ClientConnection:229 | [<none> -> <pulsar://localhost:6650>] Destroyed connection
Traceback (most recent call last):
File "Producer.py", line 13, in <module>
producer = client.create_producer(TOPIC)
File "/usr/local/lib/python3.7/site-packages/pulsar/__init__.py", line 476, in create_producer
p._producer = self._client.create_producer(topic, conf)
Exception: Pulsar error: ConnectError```
----
2019-12-08 20:50:23 UTC - Endre Karlson: Anyone here running Pulsar on Azure and wanna share their setup ?
----
2019-12-08 20:52:29 UTC - David Kjerrumgaard: @Roman Popenov It looks like your Pulsar instance hasn't opened it's broker port, 6650. I am not sure if you are actually running Pulsar on the same host as your producer code or if you are using port forwarding, but I would check that port to see if anything is listening
----
2019-12-09 06:31:57 UTC - Sijie Guo: @Sijie Guo set the channel topic: - Pullsar release 2.4.2 released! <http://pulsar.apache.org/docs/en/2.4.2/concepts-overview/>
<http://pulsar.apache.org/release-notes/#2.4.2>
- Pulsar manager release 0.1.0 released! <http://pulsar.apache.org/docs/en/next/administration-pulsar-manager/>
tada : Ali Ahmed, Yong Zhang, xiaolong.ran, juraj
----
2019-12-09 06:48:34 UTC - Ali Ahmed: this is pretty good tool to query pulsar via presto
<https://github.com/wix/quix>
----
2019-12-09 08:24:24 UTC - juraj: Hi, has something changed between 2.4.1 and 2.4.2 so that after creating a tenant with `pulsar-admin tenants create mytenant`, the following will fail: `pulsar-admin namespaces create mytenant/myns` with `Cluster [mypulsar] is not in the list of allowed clusters list for tenant [mytenant]` ? This worked fine on 2.4.1.
----
2019-12-09 08:32:27 UTC - juraj: `pulsar-admin clusters list` returns no clusters, so the init didn't create a cluster at all
----
2019-12-09 08:33:05 UTC - juraj: changes like this break the tooling and should not occur in minor version upgrades IMO
----
2019-12-09 08:34:53 UTC - Sijie Guo: how did you upgrade the cluster?
----
2019-12-09 08:35:13 UTC - Sijie Guo: are you creating a brand new cluster with 2.4.2?
----
2019-12-09 08:35:40 UTC - juraj: did a clean install, undeployed the previous first, cleaned everything yes - was using the same procedure before when working with 2.4.1
----
2019-12-09 08:36:51 UTC - Sijie Guo: which instruction did you follow?
----
2019-12-09 08:37:45 UTC - juraj: i'm using the pulsar helm chart, with some very minor modifications, which worked on 2.4.1
----
2019-12-09 08:38:39 UTC - juraj: i also have it running on local for quick testing - but still haven't figured out how to build my own docker image - bc this is done in travis ci
----
2019-12-09 08:40:08 UTC - juraj: (the one i'm reporting now is running on aws EKS)
----
2019-12-09 08:41:42 UTC - Sijie Guo: what does `pulsar-admin tenants get mytenant` return?
----
2019-12-09 08:46:05 UTC - juraj: when i create it, it exists:
----
2019-12-09 08:47:29 UTC - juraj: but the namespace creation then fails bc there's no cluster:
----
2019-12-09 08:49:00 UTC - juraj: idk, i can try to create the cluster manually, just need to figure out the `--broker-url` param for `pulsar-admin clusters create`
----
2019-12-09 08:52:01 UTC - Sijie Guo: what is your local cluster name? is it `pulsar-dev`?
----
2019-12-09 08:52:21 UTC - Sijie Guo: the error indicates pulsar-dev is not in your allowed cluster list.
----
2019-12-09 08:53:09 UTC - Sijie Guo: `bin/pulsar-admin tenants update --allowed-cluster pulsar-dev mytenant`
+1 : xiaolong.ran
----
2019-12-09 08:53:25 UTC - Sijie Guo: you need to add `pulsar-dev` into the allowed cluster list of `mytenant`
----
2019-12-09 08:57:04 UTC - Sijie Guo: `pulsar-admin tenants create` always creates a tenant with zero allowed clusters. If you don’t specify the current cluster in the an allowed cluster list, you are not allowed to create a namespace in current cluster.
----
2019-12-09 08:57:22 UTC - Sijie Guo: Are you sure you running the same set of commands as in your previous 2.4.1 cluster?
----
2019-12-09 08:58:37 UTC - juraj: yes, same set
the problem is that the pulsar-dev cluster does not exist:
----
2019-12-09 08:59:19 UTC - juraj:
----
2019-12-09 08:59:49 UTC - juraj: (also, the `tenants create` command didn't require any options on 2.4.1 and all worked fine)
----
2019-12-09 09:03:35 UTC - Nicolas Ha: yes that curl works
----
2019-12-09 09:04:35 UTC - Nicolas Ha:
----
2019-12-09 09:06:23 UTC - juraj: if i'd figured out how to build the docker image locally, i could do a lot more in terms of bug fixing, as i'm a 20 year java dev myself
----
2019-12-09 09:06:31 UTC - Sijie Guo: can you get the log of the pod running cluster-metadata-setup?
----
2019-12-09 09:07:04 UTC - Sijie Guo: @juraj: you can run `mvn clean install -Pdocker` to build docker image if you wish
----
2019-12-09 09:07:28 UTC - Sijie Guo: although I don’t think it is a problem of 2.4.2 image
----
2019-12-09 09:08:13 UTC - Sijie Guo: if the cluster is not created, it looks more like a problem when setting up cluster metadata.
----
2019-12-09 09:08:22 UTC - Sijie Guo: so if you can get the log of that pod, it would be great.
----
2019-12-09 09:08:34 UTC - juraj:
----
2019-12-09 09:08:48 UTC - juraj: ^here; i'll try that mv cmd later thanks
----