You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2022/05/17 02:59:04 UTC

[GitHub] [skywalking] Superskyyy opened a new issue, #9071: [Feature] SkyWalking Python agent auto-startup in prefork servers

Superskyyy opened a new issue, #9071:
URL: https://github.com/apache/skywalking/issues/9071

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement.
   
   
   ### Description
   
   When we face a prefork server like Gunicorn, we will need to manually pass a postfork snippet ([example](https://skywalking.apache.org/docs/skywalking-python/latest/en/setup/faq/how-to-use-with-uwsgi/)) to spin up the agent in the forked process. Not every user is aware of this, which generally causes confusion. Fortunately, this can be automatically done starting Python 3.7 thanks to [this functionality](https://docs.python.org/3/library/os.html#os.register_at_fork). 
   
   For newcomers, this is a good first issue to start with.
   
   ### Use case
   
   Python agent should startup normally and automatically under prefork servers. 
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] jiang1997 commented on issue #9071: [Feature] SkyWalking Python agent auto-startup in prefork servers

Posted by GitBox <gi...@apache.org>.

jiang1997 commented on issue #9071:
URL: https://github.com/apache/skywalking/issues/9071#issuecomment-1308857658

   # Case
   https://skywalking.apache.org/docs/skywalking-python/next/en/setup/faq/how-to-use-with-uwsgi/
   But start the application by cli as showd below
   ```
   sw-python uwsgi --die-on-term \
       --http 0.0.0.0:5000 \
       --http-manage-expect \
       --master --workers 3 \
       --enable-threads \
       --threads 3 \
       --manage-script-name \
       --mount /=main:app
   ```
   I've done a small amount of research, and this might be caused by the inherent property of Linux's ``fork``.
   In Python Agent, we use a global variable [_started](https://github.com/apache/skywalking-python/blob/11a74be39b07d55290c2df3de2b499bb126de048/skywalking/agent/__init__.py#L39) to prevent redundant initialization. 
   
   In preforking mode of ``uWsgi``, 
   1. ```uWsgi``` spawns a master process in which the Agent will be initialized for the first time and ``_started`` will be set as ``True``. 
   2. ```uWsgi``` spawns n worker processes in which ``_started`` remains ``True``. So the necessary reporting threads will not be created here and nothing will be sent to ``oap``.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] Superskyyy commented on issue #9071: [Feature] SkyWalking Python agent auto-startup in prefork servers

Posted by GitBox <gi...@apache.org>.

Superskyyy commented on issue #9071:
URL: https://github.com/apache/skywalking/issues/9071#issuecomment-1146847660

   The current Python agent startup mechanism is very vulnerable to break since the background reporting threads were started whenever agent.start() is called (from code/CLI). As a solution, I will decouple the monkey_patch step and reporting_thread.start(), essentially delaying the first report after any worker fork, it should remove most of the weird behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] Superskyyy commented on issue #9071: [Feature] SkyWalking Python agent auto-startup in prefork servers

Posted by GitBox <gi...@apache.org>.

Superskyyy commented on issue #9071:
URL: https://github.com/apache/skywalking/issues/9071#issuecomment-1146957094

   > Is anything happening here?
   
   My bad, I spotted some major problems with the agent startup after process fork, but previous comment was wrong so I deleted it, I will update later when I figure it out.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] Superskyyy closed issue #9071: [Feature] SkyWalking Python agent auto-startup in prefork servers

Posted by "Superskyyy (via GitHub)" <gi...@apache.org>.

Superskyyy closed issue #9071: [Feature] SkyWalking Python agent auto-startup in prefork servers
URL: https://github.com/apache/skywalking/issues/9071


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] wu-sheng commented on issue #9071: [Feature] SkyWalking Python agent auto-startup in prefork servers

Posted by GitBox <gi...@apache.org>.

wu-sheng commented on issue #9071:
URL: https://github.com/apache/skywalking/issues/9071#issuecomment-1146907090

   Is anything happening here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] nonofroid commented on issue #9071: [Feature] SkyWalking Python agent auto-startup in prefork servers

Posted by GitBox <gi...@apache.org>.

nonofroid commented on issue #9071:
URL: https://github.com/apache/skywalking/issues/9071#issuecomment-1128355768

   😁


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] Superskyyy commented on issue #9071: [Feature] SkyWalking Python agent auto-startup in prefork servers

Posted by GitBox <gi...@apache.org>.

Superskyyy commented on issue #9071:
URL: https://github.com/apache/skywalking/issues/9071#issuecomment-1308948811

   > ## Use case
   > [skywalking.apache.org/docs/skywalking-python/next/en/setup/faq/how-to-use-with-uwsgi](https://skywalking.apache.org/docs/skywalking-python/next/en/setup/faq/how-to-use-with-uwsgi/) But start the application by cli as showd below
   > 
   > ```
   > sw-python uwsgi --die-on-term \
   >     --http 0.0.0.0:5000 \
   >     --http-manage-expect \
   >     --master --workers 3 \
   >     --enable-threads \
   >     --threads 3 \
   >     --manage-script-name \
   >     --mount /=main:app
   > ```
   > 
   > ## Potential cause
   > I've done a small amount of research, and this might be caused by the inherent property of Linux's `fork`. In Python Agent, we use a global variable [_started](https://github.com/apache/skywalking-python/blob/11a74be39b07d55290c2df3de2b499bb126de048/skywalking/agent/__init__.py#L39) to prevent redundant initialization.
   > 
   > In `preforking mode` of `uWsgi`,
   > 
   > 1. At first, `uWsgi` spawns a master process in which the Agent will be initialized for the first time and `_started` will be set as `True`.
   > 2. Then the master process spawns n worker processes in which `_started` remains `True`. So the necessary reporting threads will not be created here and nothing will be sent to `oap`.
   > 
   > Commenting this [line](https://github.com/apache/skywalking-python/blob/11a74be39b07d55290c2df3de2b499bb126de048/skywalking/bootstrap/loader/sitecustomize.py#L147) preventing initialization inside the master process, then each worker processes will have its own Agent initialized successfully.
   
   Thank you, I see where the problem is, maybe adding PID info to the `__started` flag so upon fork we can allow a safe restart.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org