You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/09/12 15:21:36 UTC
[GitHub] [pulsar] emanueledomingo opened a new issue, #17596: [Bug] ConnectError while using `pyarrow.fs`
emanueledomingo opened a new issue, #17596:
URL: https://github.com/apache/pulsar/issues/17596
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
OS: Ubuntu 22.04.1 LTS
Pulsar: 2.9.2
`python`: 3.10.6
`pulsar-client`: 2.9.3
`pyarrow`: 7.0.0
### Minimal reproduce step
The client doesn't connect to the Broker if i import the `pyarrow.fs` library before.
```python
from pyarrow import fs
import pulsar
c = pulsar.Client("pulsar://localhost:6650")
p = c.create_producer("test")
```
If i just comment or import the `pyarrow.fs` library after pulsar, it works.
### What did you expect to see?
```
2022-09-12 10:37:42.449 INFO [140586932254528] ClientConnection:189 | [<none> -> pulsar://localhost:6650] Create ClientConnection, timeout=10000
2022-09-12 10:37:42.449 INFO [140586932254528] ConnectionPool:96 | Created connection for pulsar://localhost:6650
2022-09-12 10:37:42.450 INFO [140586872165952] ClientConnection:375 | [127.0.0.1:51696 -> 127.0.0.1:6650] Connected to broker
2022-09-12 10:37:42.462 INFO [140586872165952] HandlerBase:64 | [persistent://public/default/test, ] Getting connection from pool
2022-09-12 10:37:42.465 INFO [140586872165952] ClientConnection:189 | [<none> -> pulsar://localhost:6650] Create ClientConnection, timeout=10000
2022-09-12 10:37:42.466 INFO [140586872165952] ConnectionPool:96 | Created connection for pulsar://ac3b9ea4f607:6650
2022-09-12 10:37:42.466 INFO [140586872165952] ClientConnection:377 | [127.0.0.1:51698 -> 127.0.0.1:6650] Connected to broker through proxy. Logical broker: pulsar://ac3b9ea4f607:6650
2022-09-12 10:37:42.472 INFO [140586872165952] ProducerImpl:189 | [persistent://public/default/test, ] Created producer on broker [127.0.0.1:51698 -> 127.0.0.1:6650]
```
### What did you see instead?
```
0000-00-00 00:00:00.000 INFO [0000-00-00 00:00:00.000 INFO [0000-00-00 00:00:00.000 INFO [0000-00-00 00:00:00.000 INFO [0000-00-00 00:00:00.000 INFO [0000-00-00 00:00:00.000 INFO [0000-00-00 00:00:00.000 INFO [0000-00-00 00:00:00.000 INFO [0000-00-00 00:00:00.000 INFO [0000-00-00 00:00:00.000 ERROR [0000-00-00 00:00:00.000 INFO [---------------------------------------------------------------------------
ConnectError Traceback (most recent call last)
Cell In [4], line 1
----> 1 p = c.create_producer("test")
File ~/mambaforge/envs/xxx/lib/python3.10/site-packages/pulsar/__init__.py:642, in Client.create_producer(self, topic, producer_name, schema, initial_sequence_id, send_timeout_millis, compression_type, max_pending_messages, max_pending_messages_across_partitions, block_if_queue_full, batching_enabled, batching_max_messages, batching_max_allowed_size_in_bytes, batching_max_publish_delay_ms, message_routing_mode, lazy_start_partitioned_producers, properties, batching_type, encryption_key, crypto_key_reader)
639 conf.crypto_key_reader(crypto_key_reader.cryptoKeyReader)
641 p = Producer()
--> 642 p._producer = self._client.create_producer(topic, conf)
643 p._schema = schema
644 p._client = self._client
ConnectError: Pulsar error: ConnectError
```
### Anything else?
The same error occurs using the [`dagster`](https://dagster.io/) library. I noticed that both pyarrow and dagster use `grpcio` under the hood.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] laurent-chriqui commented on issue #17596: [Bug] ConnectError while using `pyarrow.fs`
Posted by GitBox <gi...@apache.org>.
laurent-chriqui commented on issue #17596:
URL: https://github.com/apache/pulsar/issues/17596#issuecomment-1285500127
Hello,
I have the same issue with the pyproj library using Python>=3.9.
When I import pyproj first, I have the exact same issue with Pulsar.
If I import pulsar first, I have a bug in pyproj when I try to do this for example:
```python
import pulsar
import pyproj
pyproj.Transformer.from_crs('epsg:4326', 'epsg:3035')
```
`Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.9/site-packages/pyproj/transformer.py", line 600, in from_crs
cstrencode(CRS.from_user_input(crs_from).srs),
File "/usr/local/lib/python3.9/site-packages/pyproj/crs/crs.py", line 501, in from_user_input
return cls(value, **kwargs)
File "/usr/local/lib/python3.9/site-packages/pyproj/crs/crs.py", line 348, in __init__
self._local.crs = _CRS(self.srs)
File "pyproj/_crs.pyx", line 2352, in pyproj._crs._CRS.__init__
pyproj.exceptions.CRSError: Invalid projection: epsg:4326: (Internal Proj Error: proj_create: cannot build geodeticCRS 4326: cannot build unit of measure 9122: non double value)`
So there seems to be some kind of conflict between cpp libraries since pyproj is also a wrapper for the PROJ cpp library and python 3.9+
I do not need to install any version of grpcio for this to happen though.
Thank you for looking into this because we are stuck at python 3.8 for now...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #17596: [Bug] ConnectError while using `pyarrow.fs`
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #17596:
URL: https://github.com/apache/pulsar/issues/17596#issuecomment-1278382328
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] emanueledomingo commented on issue #17596: [Bug] ConnectError while using `pyarrow.fs`
Posted by GitBox <gi...@apache.org>.
emanueledomingo commented on issue #17596:
URL: https://github.com/apache/pulsar/issues/17596#issuecomment-1245141692
Hi guys, i made more tests today at office. I found out that the problem is the combination of `python` and `grpcio` libraries. I tried the connection script with
```yaml
- python==3.9
- pulsar-client==2.10.1
- grpcio==1.48.0
```
and it doesn't work. The same script with
```yaml
- python<3.9
- pulsar-client==2.10.1
- grpcio==1.48.0
```
works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org