[GitHub] [pulsar] whisust opened a new issue, #18059: [Bug] Python pulsar-client producer does not produce messages with gunicorn `--preload` flag

[GitBox <gi...@apache.org>]

whisust opened a new issue, #18059:
URL: https://github.com/apache/pulsar/issues/18059

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
   
   
   ### Version
   
   Pulsar brokers version 2.9
   python 3.9
   Tested on macOS and in production (Linux)
   
   ### Minimal reproduce step
   
   ### 1. Python env setup
   
   Create a venv with the following packages installed:
   `requirements.txt`
   ```
   Flask==2.0.1
   gevent==21.8.0
   gunicorn==20.1.0
   pulsar-client==2.9.2
   ```
   
   ### 2. Pulsar setup
   
   Start a standalone pulsar broker compatible with a 2.9 client
   ```
   ~/dev/pulsar
   > bin/pulsar standalone -nss
   ```
   
   ### 3. Python script
   `test_pulsar.py`
   ```python
   import logging
   import uuid
   from datetime import datetime
   
   from _pulsar import Result
   from flask import Flask, make_response
   from pulsar import Client, MessageId, schema
   
   logging.basicConfig(level=logging.INFO, format='[%(asctime)s] [%(levelname)s] [%(name)s] %(message)s')
   logger = logging.getLogger(__name__)
   logger.info('Loading API')
   
   app = Flask(__name__)
   
   client = Client('pulsar://127.0.0.1:6650', authentication=None)
   
   producer = client.create_producer(
       'non-persistent://public/default/test-gunicorn',
       producer_name=f'my-producer-{uuid.uuid4()}',
       schema=schema.StringSchema()
   )
   
   
   def callback(res: Result, _msg_id: MessageId):
       logger.info(f'Callback result here! Event acknowledged by the broker.')
   
   
   @app.post('/post-message')
   def post_pulsar_message():
       logger.info('Calling producer.send_async now, in the next lines there should be the callback result')
       dt = datetime.now()
       producer.send_async(content=f'dt={dt.isoformat()}', callback=callback)
       logger.info('After producer.send_async, returning the http response')
       return '', 201
   
   
   @app.get('/')
   def healthcheck():
       logger.info('API Running fine')
       return make_response({'status': 'healthy'})
   
   
   logger.info('API started')
   ```
   
   
   ### 4. Run configurations
   
   In your virtual environment, run any of the following: 
   
   **valid configuration**
   ```
   gunicorn test_pulsar:app --workers=2
   ```
   
   **broken configuration**
   ```
   gunicorn test_pulsar:app --workers=2 --preload
   ```
   
   ### 5. Call the producing endpoint
   ```
   curl -X POST http://127.0.0.1:8000/post-message
   ```
   
   ### What did you expect to see?
   
   In the **valid configuration** , once the endpoint is called, the message is correctly produced and we can see the callback log line.
   
   ```
   [INFO] [test_pulsar] Calling producer.send_async now, in the next lines there should be the callback result
   [INFO] [test_pulsar] After producer.send_async, returning the http response
   [INFO] [test_pulsar] Callback result here! Event acknowledged by the broker.
   ```
   
   ### What did you see instead?
   
   In the **invalid configuration** the producer line is executed but no message is produced
   
   ```
   [INFO] [test_pulsar] Calling producer.send_async now, in the next lines there should be the callback result
   [INFO] [test_pulsar] After producer.send_async, returning the http response
   <nothing else>
   ```
   
   ### Anything else?
   
   I tried also with the `producer.send` version, but it blocks and times out.
   
   I also made sure that the monkey-patching expected in gunicorn runtimes does not affect the issue (tried with and without).
   
   It seems to be an issue with the execution context of the actual producer, being incompatible with a gunicorn runtime, at least with those params.
   
   This is an issue because using the `--preload` param is a must-have in production, as it generates only one of each static value (producer, db pool, caches and so on) instead of one per worker, and not having it is expensive in the compute / memory / operations side.
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #18059: [Bug] Python pulsar-client producer does not produce messages with gunicorn `--preload` flag

[GitBox <gi...@apache.org>]

github-actions[bot] commented on issue #18059:
URL: https://github.com/apache/pulsar/issues/18059#issuecomment-1316188998

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org