You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2022/11/10 05:39:03 UTC

[GitHub] [skywalking] Superskyyy opened a new issue, #9939: [Python] Python agent multiprocessing support enhancement

Superskyyy opened a new issue, #9939:
URL: https://github.com/apache/skywalking/issues/9939

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement.
   
   
   ### Description
   
   Making the Python agent work in a multiprocessing-enabled environment has been the main center of concern based on my observation of user feedback. This should be addressed before we release v1.0.0.
   
   To make it easier for our users and given the new meter reporter capability contributed by @jiang1997, we need this issue to track several important changes:
   
   The core issue to address: Separate instances when a new process is spawned/forked from the original process that initialized the Python agent, so that we don't report PVM metrics from multiple Python processes to a single instance. 
   
   Cases to provide solution for:
   1. Common case 1: Normal Python code with fork/spawn, we reinitialize Python agent in the child process and keep the parent process agent.
   2. Common case 2: Normal Python web application with pre-fork servers (Uwsgi, Gunicorn etc.), we may give an additional `sw-python -bla` CLI option to spawn only in child process (parent process does not take requests)
   3. Edge case: Multiprocessing pool /ProcessPoolExecutor: If we give each new process a new instance ID, we need to be careful not spawn new agents in pool-based child processes. Otherwise, we may end up with hundreds of thousands of instances (for example, in a data science application, we may frequently use pool-based processes to process data) 
   
   
   ### Use case
   
   Users could run Python agent smoothly in a multiprocessing Python project (which is rather common given the limitations of GIL)
   
   ### Related issues
   
   After the above changes, the following issues should all be addressed along the way.
   
   I will work with the OSPP student @jiang1997 to tackle these problems one by one.
   https://github.com/apache/skywalking/issues/9721
   
   https://github.com/apache/skywalking/issues/9071
   
   https://github.com/apache/skywalking/issues/8854
   
   https://github.com/apache/skywalking/issues/7319
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] Superskyyy commented on issue #9939: [Python] Python agent multiprocessing support enhancement

Posted by GitBox <gi...@apache.org>.
Superskyyy commented on issue #9939:
URL: https://github.com/apache/skywalking/issues/9939#issuecomment-1346940569

   Forking bugs (related to preforking server) is fixed on my test branch, corresponding E2E tests are refactored and passed. Opening first PR soon.
   
   Second PR will be about the real fork calls inside Python code, adding fork handlers to establish channels (needs to be done for gRPC and Kafka) should be sufficient.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] Superskyyy closed issue #9939: [Python] Python agent multiprocessing support enhancement

Posted by "Superskyyy (via GitHub)" <gi...@apache.org>.
Superskyyy closed issue #9939: [Python] Python agent multiprocessing support enhancement 
URL: https://github.com/apache/skywalking/issues/9939


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] Superskyyy commented on issue #9939: [Python] Python agent multiprocessing support enhancement

Posted by GitBox <gi...@apache.org>.
Superskyyy commented on issue #9939:
URL: https://github.com/apache/skywalking/issues/9939#issuecomment-1320467340

   checked case: FastAPI with Uvicorn on Gunicorn (multi-workers) with legacy setup, agent starts fine, fork happens before agent could start and threading is enabled by default.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org