You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/03/14 10:40:27 UTC
[GitHub] [pulsar] Syphixs opened a new issue #9906: Python Client segmentation fault (core dumped) or free(): invalid pointer Errors
Syphixs opened a new issue #9906:
URL: https://github.com/apache/pulsar/issues/9906
**Describe the bug**
I wrote a very simple test method cause I do get some very hard to debug interactions between pulsar client 2.8 and 2.7 too with cv2 (all last 4 versions tested and same problem)
This is the Testprogram:
If I import cv2 first the core gets dumped at the line it wants to create the test producer:
If I import pulsar first the producer gets created successfully ( no suspicious logs at broker either) but as soon as cv2 gets to the second imread -> garbage collecting happens before I guess the core gets dumped again. so now the error is in cv2?!?
**To Reproduce**
Steps to reproduce the behavior:
Build pulsar-client-cpp and python client from source ( either 2.7 or newest master 2.8)
Try test program.
**Desktop (please complete the following information):**
- OS: Pop OS 20.10 but tried with Ubuntu 19.10 as well
- python 3.9.1 and 3.9
- boost and python boost 1.75 but tried with 1.71 as well
- gcc/++ 10.2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9906: Python Client segmentation fault (core dumped) or free(): invalid pointer Errors
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9906:
URL: https://github.com/apache/pulsar/issues/9906#issuecomment-1058892558
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Syphixs commented on issue #9906: Python Client segmentation fault (core dumped) or free(): invalid pointer Errors
Posted by GitBox <gi...@apache.org>.
Syphixs commented on issue #9906:
URL: https://github.com/apache/pulsar/issues/9906#issuecomment-799224333
You are correct I just have tested it on my mac and no problems there though I use the exact same libraries? boost 1.75 and pulsar client 2.8 ( 84cfb3affa1ce5bc1db29baab5498205049019dd ). I also tried the backtrace with your suggestion , thanks for that.
First output with cv2 import first. Hangs again at second imread
Second output with pulsar import first. Error at creating producer.
I have now tested it on 3 different ubuntu machines (19.10, 20.04. and 20.10) and with boost 1.75 built from source or 1.71 with apt package manager and opencv-python or opencv-python-headless ( version 4.4.0.40 till 4.5.1.48 ) and not a single combination works. I have also simplified the code that just cv2,pulsar and json have to be imported.
`
#!/usr/bin/env python3
import cv2
import pulsar
import json
def main():
client = {}
client["audience"] = "pulsar_service"
client["issuer_url"] = "https://example.com/auth/realms/test_company"
client["client_id"] = "pulsar-coco.detection.images"
client["client_secret"] = "secret"
client["grant_type"] = "client_credentials"
temp_client = pulsar.Client(
"pulsar+ssl://127.0.0.1:6651",
use_tls=True,
tls_trust_certs_file_path="./pulsar-test/ca.crt",
authentication=pulsar.AuthenticationOauth2(json.dumps(client)),
)
test_topic = "persistent://ml/test_company/test"
print(f"Testing client {client} in Realm on topic {test_topic}")
test_producer = temp_client.create_producer(
topic=test_topic, send_timeout_millis=5000
)
img_list = [
"./pulsar-test/images/8375743808_a43f504e44_b.jpg",
"./pulsar-test/images/8886531759_033e06ea68_b.jpg",
]
for x in range(len(img_list)):
print(f"{x} - Image")
rnd_img = cv2.imread(img_list[x])
print("after imread")
_, image_np_array = cv2.imencode(".jpg", rnd_img)
print("after imencode")
image_byte_array = image_np_array.tobytes()
print("after tobytes")
# img_h, img_w, channels = rnd_img.shape
# send to pulsar producer here...
if __name__ == "__main__":
main()
`
Any advice on further debugging would be highly appreciated! thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #9906: Python Client segmentation fault (core dumped) or free(): invalid pointer Errors
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #9906:
URL: https://github.com/apache/pulsar/issues/9906#issuecomment-799049682
I tried with OpenCV 4.5 that is installed by `pip3 install opencv-python` and Pulsar client that is built with latest master in MacOS Big Sur 11.2.
Then tried the command line with a Pulsar standalone, there's no error.
```
>>> import cv2
>>> import pulsar
>>> client = pulsar.Client(service_url='pulsar://localhost:6650')
>>> producer = client.create_producer(topic='my-topic')
2021-03-15 10:27:41.285 INFO [0x117ca8e00] ConnectionPool:85 | Created connection for pulsar://localhost:6650
2021-03-15 10:27:41.295 INFO [0x7000043a9000] ClientConnection:372 | [127.0.0.1:50925 -> 127.0.0.1:6650] Connected to broker
2021-03-15 10:27:41.301 INFO [0x7000043a9000] HandlerBase:54 | [persistent://public/default/my-topic, ] Getting connection from pool
2021-03-15 10:27:41.310 INFO [0x7000043a9000] ProducerImpl:176 | [persistent://public/default/my-topic, ] Created producer on broker [127.0.0.1:50925 -> 127.0.0.1:6650]
>>> producer.send('hello'.encode())
<_pulsar.MessageId object at 0x117b28190>
>>> img = cv2.imread('./images/lena.png')
>>> _, image_np_array = cv2.imencode('.jpg', img)
>>> print(image_np_array)
[[255]
[216]
[255]
...
[207]
[255]
[217]]
```
The segmentation fault might be something with the dependencies conflicts with Pulsar and OpenCV. But it's hard to debug. I suggest that you can simplify your code and watch the result again. BTW, you can also try to print the backtrace, see https://stackoverflow.com/questions/16731115/how-to-debug-a-python-segmentation-fault
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Syphixs commented on issue #9906: Python Client segmentation fault (core dumped) or free(): invalid pointer Errors
Posted by GitBox <gi...@apache.org>.
Syphixs commented on issue #9906:
URL: https://github.com/apache/pulsar/issues/9906#issuecomment-798884978
Maybe related to #9796
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Syphixs edited a comment on issue #9906: Python Client segmentation fault (core dumped) or free(): invalid pointer Errors
Posted by GitBox <gi...@apache.org>.
Syphixs edited a comment on issue #9906:
URL: https://github.com/apache/pulsar/issues/9906#issuecomment-799224333
You are correct I just have tested it on my mac and no problems there though I use the exact same libraries? boost 1.75 and pulsar client 2.8 ( 84cfb3affa1ce5bc1db29baab5498205049019dd ). I also tried the backtrace with your suggestion , thanks for that.
First output with cv2 import first. Hangs again at second imread
Second output with pulsar import first. Error at creating producer.
I have now tested it on 3 different ubuntu machines (19.10, 20.04. and 20.10) and with boost 1.75 built from source or 1.71 with apt package manager and opencv-python or opencv-python-headless ( version 4.4.0.40 till 4.5.1.48 ) and not a single combination works. I have also simplified the code that just cv2,pulsar and json have to be imported. Same errors occur
`
#!/usr/bin/env python3
import cv2
import pulsar
import json
def main():
client = {}
client["audience"] = "pulsar_service"
client["issuer_url"] = "https://example.com/auth/realms/test_company"
client["client_id"] = "pulsar-coco.detection.images"
client["client_secret"] = "secret"
client["grant_type"] = "client_credentials"
temp_client = pulsar.Client(
"pulsar+ssl://127.0.0.1:6651",
use_tls=True,
tls_trust_certs_file_path="./pulsar-test/ca.crt",
authentication=pulsar.AuthenticationOauth2(json.dumps(client)),
)
test_topic = "persistent://ml/test_company/test"
print(f"Testing client {client} in Realm on topic {test_topic}")
test_producer = temp_client.create_producer(
topic=test_topic, send_timeout_millis=5000
)
img_list = [
"./pulsar-test/images/8375743808_a43f504e44_b.jpg",
"./pulsar-test/images/8886531759_033e06ea68_b.jpg",
]
for x in range(len(img_list)):
print(f"{x} - Image")
rnd_img = cv2.imread(img_list[x])
print("after imread")
_, image_np_array = cv2.imencode(".jpg", rnd_img)
print("after imencode")
image_byte_array = image_np_array.tobytes()
print("after tobytes")
# img_h, img_w, channels = rnd_img.shape
# send to pulsar producer here...
if __name__ == "__main__":
main()
`
Any advice on further debugging would be highly appreciated! thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org