You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Yunze Xu <yz...@streamnative.io.INVALID> on 2023/03/01 06:57:30 UTC
Re: [VOTE] Pulsar Client Python Release 3.1.0 Candidate 3
Hi Zike,
I've reproduced this issue successfully with:
- Python 3.8 on Ubuntu 20.04
- Python 3.10 on macOS 12
It seems to be a serious bug and I'm going to figure out the reason ASAP.
Thanks,
Yunze
On Tue, Feb 28, 2023 at 5:17 PM Zike Yang <zi...@apache.org> wrote:
>
> Hi, Yunze
>
> It raises an exception when I run the consumer example.
>
> Here are my environments:
> * macos 12.06 x86_64
> * python 3.7
>
> Here are my reproduce steps:
> * Start the pulsar standalone
> * Start the consumer example
> * Start the producer example
> * The consumer can receive messages successfully.
> * Stop the consumer, then it throws some exception:
> ```
> ➜ examples git:(main) ✗ python consumer.py
> 2023-02-28 17:11:04.742 INFO [0x113b19600] Client:87 | Subscribing on
> Topic :my-topic
> 2023-02-28 17:11:04.742 INFO [0x113b19600] ClientConnection:190 |
> [<none> -> pulsar://localhost:6650] Create ClientConnection,
> timeout=10000
> 2023-02-28 17:11:04.742 INFO [0x113b19600] ConnectionPool:97 |
> Created connection for pulsar://localhost:6650
> 2023-02-28 17:11:04.745 INFO [0x700003606000] ClientConnection:388 |
> [127.0.0.1:49258 -> 127.0.0.1:6650] Connected to broker
> 2023-02-28 17:11:04.857 INFO [0x700003606000] HandlerBase:72 |
> [persistent://public/default/my-topic, my-subscription, 0] Getting
> connection from pool
> 2023-02-28 17:11:04.874 INFO [0x700003606000] ConsumerImpl:238 |
> [persistent://public/default/my-topic, my-subscription, 0] Created
> consumer on broker [127.0.0.1:49258 -> 127.0.0.1:6650]
> Received message 'hello' id='(132,30,-1,0)'
> Received message 'hello' id='(132,31,-1,0)'
> Received message 'hello' id='(132,32,-1,0)'
> Received message 'hello' id='(132,33,-1,0)'
> Received message 'hello' id='(132,34,-1,0)'
> Received message 'hello' id='(132,35,-1,0)'
> Received message 'hello' id='(132,36,-1,0)'
> Received message 'hello' id='(132,37,-1,0)'
> Received message 'hello' id='(132,38,-1,0)'
> Received message 'hello' id='(132,39,-1,0)'
> ^CKeyboardInterrupt
>
> The above exception was the direct cause of the following exception:
>
> Traceback (most recent call last):
> File "consumer.py", line 32, in <module>
> msg = consumer.receive()
> File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pulsar/__init__.py",
> line 1243, in receive
> msg = self._consumer.receive()
> SystemError
> 2023-02-28 17:11:07.367 WARN [0x113b19600] ConsumerImpl:126 |
> [persistent://public/default/my-topic, my-subscription, 0] Destroyed
> consumer which was not properly closed
> 2023-02-28 17:11:07.367 INFO [0x113b19600] ConsumerImpl:134 |
> [persistent://public/default/my-topic, my-subscription, 0] Closed
> consumer for race condition: 0
> libc++abi: terminating with uncaught exception of type
> std::__1::bad_weak_ptr: bad_weak_ptr
> [1] 56483 abort python consumer.py
> ```
> This causes the program to crash.
>
> I'm using the example codes here:
> https://github.com/apache/pulsar-client-python/tree/main/examples
>
> Could you take a look? Thanks.
>
> BR,
> Zike Yang
>
> On Tue, Feb 28, 2023 at 10:37 AM PengHui Li <pe...@apache.org> wrote:
> >
> > +1 (binding)
> >
> > - install on macos
> > - start a standalone (latest master)
> > - run the example
> > https://github.com/apache/pulsar-client-python#running-examples
> >
> > Regards,
> > Penghui
> >
> > On Thu, Feb 23, 2023 at 11:00 PM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > Thank you Yunze for double checking.
> > >
> > > I don't have time to test the release, so I am voting +0 (and not -1)
> > >
> > > Enrico
> > >
> > > Il giorno gio 23 feb 2023 alle ore 12:57 Yunze Xu
> > > <yz...@streamnative.io.invalid> ha scritto:
> > > >
> > > > Hi Enrico,
> > > >
> > > > I will test more operation systems and open a discussion soon. For
> > > > now, I just tested the TLS encryption and token authentication, and I
> > > > found this issue does not exist as expected for both Python and
> > > > Node.js clients.
> > > > - Windows: Only Python client works
> > > > - Ubuntu: Both work
> > > > - macOS: Not tested yet
> > > >
> > > > But the OAuth2 authentication case doesn't work. It's caused by
> > > > https://github.com/apache/pulsar/pull/16064 and fixed by
> > > > https://github.com/apache/pulsar-client-cpp/pull/190.
> > > >
> > > > When the protocol of the issuer URL is HTTPS:
> > > > - Before #16064, OAuth2 authentication works by skipping verifying the
> > > > peer, it's dangerous for security reasons
> > > > - After #16064 and before #190, there is no way to perform OAuth2
> > > authentication
> > > > - After #190, users can configure the `tls_trust_certs_file_path` to do
> > > that.
> > > >
> > > > So I think it should not be treated as a regression and should not
> > > > block this release. We can support automatically detecting the CA
> > > > certs for OAuth2 authentication later.
> > > >
> > > > The regression should be the case when the TLS encryption is enabled.
> > > > And as I've said at the beginning, I'm going to do more tests and open
> > > > a discussion.
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Thu, Feb 23, 2023 at 6:06 PM Enrico Olivelli <eo...@gmail.com>
> > > wrote:
> > > > >
> > > > > Does this version have the same problems about TLS CA Certs as the
> > > > > NodeJS client ?
> > > > > In that case I would cast a -1
> > > > >
> > > > > Enrico
> > > > >
> > > > > Il giorno gio 23 feb 2023 alle ore 01:21 Matteo Merli
> > > > > <ma...@gmail.com> ha scritto:
> > > > > >
> > > > > > +1 bindind
> > > > > > --
> > > > > > Matteo Merli
> > > > > > <ma...@gmail.com>
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 17, 2023 at 3:27 AM Yunze Xu
> > > <yz...@streamnative.io.invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > This is the 3rd release candidate for Apache Pulsar Client Python,
> > > > > > > version 3.1.0.
> > > > > > >
> > > > > > > It fixes the following issues:
> > > > > > >
> > > https://github.com/apache/pulsar-client-python/milestone/2?closed=1
> > > > > > >
> > > > > > > *** Please download, test and vote on this release. This vote will
> > > > > > > stay open for at least 72 hours ***
> > > > > > >
> > > > > > > Python wheels:
> > > > > > >
> > > > > > >
> > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-python-3.1.0-candidate-3/
> > > > > > >
> > > > > > > The supported python versions are 3.7, 3.8, 3.9, 3.10 and 3.11. The
> > > > > > > supported platforms and architectures are:
> > > > > > > - Windows x86_64 (windows/)
> > > > > > > - glibc-based Linux x86_64 (linux-glibc-x86_64/)
> > > > > > > - glibc-based Linux arm64 (linux-glibc-arm64/)
> > > > > > > - musl-based Linux x86_64 (linux-musl-x86_64/)
> > > > > > > - musl-based Linux arm64 (linux-musl-arm64/)
> > > > > > > - macOS universal 2 (macos/)
> > > > > > >
> > > > > > > You can download the wheel (the `.whl` file) according to your own
> > > OS
> > > > > > > and Python version
> > > > > > > and install the wheel:
> > > > > > > - Windows: `py -m pip install *.whl --force-reinstall`
> > > > > > > - Linux or macOS: `python3 -m pip install *.whl --force-reinstall`
> > > > > > >
> > > > > > > The tag to be voted upon: v3.1.0-candidate-3
> > > > > > > (9ed92ecee632c42b81a3198b8824d70d080af7f0)
> > > > > > >
> > > > > > >
> > > https://github.com/apache/pulsar-client-python/releases/tag/v3.1.0-candidate-3
> > > > > > >
> > > > > > > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > > > > > > https://dist.apache.org/repos/dist/release/pulsar/KEYS
> > > > > > >
> > > > > > > Please download the Python wheels and follow the README to test.
> > > > > > >
> > >
Re: [VOTE] Pulsar Client Python Release 3.1.0 Candidate 3
Posted by Yunze Xu <yz...@streamnative.io.INVALID>.
Hi Zike,
I've fixed this crash and showed some explanations in
https://github.com/apache/pulsar-client-python/pull/99. PTAL. After
it's cherry-picked to branch-3.1, I will open the candidate 4.
Thanks,
Yunze
On Wed, Mar 1, 2023 at 2:57 PM Yunze Xu <yz...@streamnative.io> wrote:
>
> Hi Zike,
>
> I've reproduced this issue successfully with:
> - Python 3.8 on Ubuntu 20.04
> - Python 3.10 on macOS 12
>
> It seems to be a serious bug and I'm going to figure out the reason ASAP.
>
> Thanks,
> Yunze
>
> On Tue, Feb 28, 2023 at 5:17 PM Zike Yang <zi...@apache.org> wrote:
> >
> > Hi, Yunze
> >
> > It raises an exception when I run the consumer example.
> >
> > Here are my environments:
> > * macos 12.06 x86_64
> > * python 3.7
> >
> > Here are my reproduce steps:
> > * Start the pulsar standalone
> > * Start the consumer example
> > * Start the producer example
> > * The consumer can receive messages successfully.
> > * Stop the consumer, then it throws some exception:
> > ```
> > ➜ examples git:(main) ✗ python consumer.py
> > 2023-02-28 17:11:04.742 INFO [0x113b19600] Client:87 | Subscribing on
> > Topic :my-topic
> > 2023-02-28 17:11:04.742 INFO [0x113b19600] ClientConnection:190 |
> > [<none> -> pulsar://localhost:6650] Create ClientConnection,
> > timeout=10000
> > 2023-02-28 17:11:04.742 INFO [0x113b19600] ConnectionPool:97 |
> > Created connection for pulsar://localhost:6650
> > 2023-02-28 17:11:04.745 INFO [0x700003606000] ClientConnection:388 |
> > [127.0.0.1:49258 -> 127.0.0.1:6650] Connected to broker
> > 2023-02-28 17:11:04.857 INFO [0x700003606000] HandlerBase:72 |
> > [persistent://public/default/my-topic, my-subscription, 0] Getting
> > connection from pool
> > 2023-02-28 17:11:04.874 INFO [0x700003606000] ConsumerImpl:238 |
> > [persistent://public/default/my-topic, my-subscription, 0] Created
> > consumer on broker [127.0.0.1:49258 -> 127.0.0.1:6650]
> > Received message 'hello' id='(132,30,-1,0)'
> > Received message 'hello' id='(132,31,-1,0)'
> > Received message 'hello' id='(132,32,-1,0)'
> > Received message 'hello' id='(132,33,-1,0)'
> > Received message 'hello' id='(132,34,-1,0)'
> > Received message 'hello' id='(132,35,-1,0)'
> > Received message 'hello' id='(132,36,-1,0)'
> > Received message 'hello' id='(132,37,-1,0)'
> > Received message 'hello' id='(132,38,-1,0)'
> > Received message 'hello' id='(132,39,-1,0)'
> > ^CKeyboardInterrupt
> >
> > The above exception was the direct cause of the following exception:
> >
> > Traceback (most recent call last):
> > File "consumer.py", line 32, in <module>
> > msg = consumer.receive()
> > File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pulsar/__init__.py",
> > line 1243, in receive
> > msg = self._consumer.receive()
> > SystemError
> > 2023-02-28 17:11:07.367 WARN [0x113b19600] ConsumerImpl:126 |
> > [persistent://public/default/my-topic, my-subscription, 0] Destroyed
> > consumer which was not properly closed
> > 2023-02-28 17:11:07.367 INFO [0x113b19600] ConsumerImpl:134 |
> > [persistent://public/default/my-topic, my-subscription, 0] Closed
> > consumer for race condition: 0
> > libc++abi: terminating with uncaught exception of type
> > std::__1::bad_weak_ptr: bad_weak_ptr
> > [1] 56483 abort python consumer.py
> > ```
> > This causes the program to crash.
> >
> > I'm using the example codes here:
> > https://github.com/apache/pulsar-client-python/tree/main/examples
> >
> > Could you take a look? Thanks.
> >
> > BR,
> > Zike Yang
> >
> > On Tue, Feb 28, 2023 at 10:37 AM PengHui Li <pe...@apache.org> wrote:
> > >
> > > +1 (binding)
> > >
> > > - install on macos
> > > - start a standalone (latest master)
> > > - run the example
> > > https://github.com/apache/pulsar-client-python#running-examples
> > >
> > > Regards,
> > > Penghui
> > >
> > > On Thu, Feb 23, 2023 at 11:00 PM Enrico Olivelli <eo...@gmail.com>
> > > wrote:
> > >
> > > > Thank you Yunze for double checking.
> > > >
> > > > I don't have time to test the release, so I am voting +0 (and not -1)
> > > >
> > > > Enrico
> > > >
> > > > Il giorno gio 23 feb 2023 alle ore 12:57 Yunze Xu
> > > > <yz...@streamnative.io.invalid> ha scritto:
> > > > >
> > > > > Hi Enrico,
> > > > >
> > > > > I will test more operation systems and open a discussion soon. For
> > > > > now, I just tested the TLS encryption and token authentication, and I
> > > > > found this issue does not exist as expected for both Python and
> > > > > Node.js clients.
> > > > > - Windows: Only Python client works
> > > > > - Ubuntu: Both work
> > > > > - macOS: Not tested yet
> > > > >
> > > > > But the OAuth2 authentication case doesn't work. It's caused by
> > > > > https://github.com/apache/pulsar/pull/16064 and fixed by
> > > > > https://github.com/apache/pulsar-client-cpp/pull/190.
> > > > >
> > > > > When the protocol of the issuer URL is HTTPS:
> > > > > - Before #16064, OAuth2 authentication works by skipping verifying the
> > > > > peer, it's dangerous for security reasons
> > > > > - After #16064 and before #190, there is no way to perform OAuth2
> > > > authentication
> > > > > - After #190, users can configure the `tls_trust_certs_file_path` to do
> > > > that.
> > > > >
> > > > > So I think it should not be treated as a regression and should not
> > > > > block this release. We can support automatically detecting the CA
> > > > > certs for OAuth2 authentication later.
> > > > >
> > > > > The regression should be the case when the TLS encryption is enabled.
> > > > > And as I've said at the beginning, I'm going to do more tests and open
> > > > > a discussion.
> > > > >
> > > > > Thanks,
> > > > > Yunze
> > > > >
> > > > > On Thu, Feb 23, 2023 at 6:06 PM Enrico Olivelli <eo...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > Does this version have the same problems about TLS CA Certs as the
> > > > > > NodeJS client ?
> > > > > > In that case I would cast a -1
> > > > > >
> > > > > > Enrico
> > > > > >
> > > > > > Il giorno gio 23 feb 2023 alle ore 01:21 Matteo Merli
> > > > > > <ma...@gmail.com> ha scritto:
> > > > > > >
> > > > > > > +1 bindind
> > > > > > > --
> > > > > > > Matteo Merli
> > > > > > > <ma...@gmail.com>
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Feb 17, 2023 at 3:27 AM Yunze Xu
> > > > <yz...@streamnative.io.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > This is the 3rd release candidate for Apache Pulsar Client Python,
> > > > > > > > version 3.1.0.
> > > > > > > >
> > > > > > > > It fixes the following issues:
> > > > > > > >
> > > > https://github.com/apache/pulsar-client-python/milestone/2?closed=1
> > > > > > > >
> > > > > > > > *** Please download, test and vote on this release. This vote will
> > > > > > > > stay open for at least 72 hours ***
> > > > > > > >
> > > > > > > > Python wheels:
> > > > > > > >
> > > > > > > >
> > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-python-3.1.0-candidate-3/
> > > > > > > >
> > > > > > > > The supported python versions are 3.7, 3.8, 3.9, 3.10 and 3.11. The
> > > > > > > > supported platforms and architectures are:
> > > > > > > > - Windows x86_64 (windows/)
> > > > > > > > - glibc-based Linux x86_64 (linux-glibc-x86_64/)
> > > > > > > > - glibc-based Linux arm64 (linux-glibc-arm64/)
> > > > > > > > - musl-based Linux x86_64 (linux-musl-x86_64/)
> > > > > > > > - musl-based Linux arm64 (linux-musl-arm64/)
> > > > > > > > - macOS universal 2 (macos/)
> > > > > > > >
> > > > > > > > You can download the wheel (the `.whl` file) according to your own
> > > > OS
> > > > > > > > and Python version
> > > > > > > > and install the wheel:
> > > > > > > > - Windows: `py -m pip install *.whl --force-reinstall`
> > > > > > > > - Linux or macOS: `python3 -m pip install *.whl --force-reinstall`
> > > > > > > >
> > > > > > > > The tag to be voted upon: v3.1.0-candidate-3
> > > > > > > > (9ed92ecee632c42b81a3198b8824d70d080af7f0)
> > > > > > > >
> > > > > > > >
> > > > https://github.com/apache/pulsar-client-python/releases/tag/v3.1.0-candidate-3
> > > > > > > >
> > > > > > > > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > > > > > > > https://dist.apache.org/repos/dist/release/pulsar/KEYS
> > > > > > > >
> > > > > > > > Please download the Python wheels and follow the README to test.
> > > > > > > >
> > > >