You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "Superskyyy (via GitHub)" <gi...@apache.org> on 2023/02/17 21:16:24 UTC
[GitHub] [skywalking-python] Superskyyy opened a new pull request, #288: Add prefork support for uwsgi and gunicorn
Superskyyy opened a new pull request, #288:
URL: https://github.com/apache/skywalking-python/pull/288
This is the final PR of v1.0.0 and everything is complete. We can proceed to release once this is ok & merged.
This PR adds experimental prefork server automatic injection to `sw-python` CLI, making it production friendly. Two comon prefork servers are supported: uWSGI and Gunicorn, implmented in two differnet ways.
**uWSGI:** uses system level fork and not catchable by os.register_at_fork(), I used environment variables to inject uwsgi post_fork hooks to user command.
**Gunicorn:** uses os.fork() and catchable by os.register_at_fork(), I used the previous PR's new experimental fork support to handle Gunicorn workers. (That's why this prefork support is experimental)
**Other changes:**
Big chore to documentation mentioning related changes.
Add a performance doc on tuning agent.
**Breaking change:**
Modified entrypoint of docker image.
**E2E tests:**
Totally rewrite our E2E cases, services to cover Python 3.7 - 3.11 (Each tests Single process, Gunicorn, uWSGI, Threading Profiler, Greenlet Profiler) * (gRPC/HTTP/Kafka) = 55 test cases.
<!--
â ī¸ Please make sure to read this template first, pull requests that don't accord with this template
maybe closed without notice.
Texts surrounded by `<` and `>` are meant to be replaced by you, e.g. <framework name>, <issue number>.
Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]`
-->
<!-- ==== đ Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist đ ====
### Fix <bug description or bug issue link>
- [ ] Add a unit test to verify that the fix works.
- [ ] Explain briefly why the bug exists and how to fix it.
==== đ Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist đ ==== -->
<!-- ==== đ Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist đ ====
### Improve the performance of <class or module or ...>
- [ ] Add a benchmark for the improvement, refer to [the existing ones](https://github.com/apache/skywalking/blob/master/apm-commons/apm-datacarrier/src/test/java/org/apache/skywalking/apm/commons/datacarrier/LinkedArrayBenchmark.java)
- [ ] The benchmark result.
```text
<Paste the benchmark results here>
```
- [ ] Links/URLs to the theory proof or discussion articles/blogs. <links/URLs here>
==== đ Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist đ ==== -->
<!-- ==== đ Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist đ ====
### <Feature description>
- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
- [ ] Update the documentation to include this new feature.
- [ ] Tests(including UT, IT, E2E) are added to verify the new feature.
- [ ] If it's UI related, attach the screenshots below.
- [ ] I have rebuilt the `Configuration.md` documentation by running `make doc-gen`
==== đ Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist đ ==== -->
<!-- ==== đą Remove this line WHEN AND ONLY WHEN you're adding or modifying a plugin instrumentation, follow the checklist đ ====
### <Feature description>
- [ ] If adding a new plugin, add a component id in [the main repo](https://github.com/apache/skywalking/blob/master/oap-server/server-starter/src/main/resources/component-libraries.yml)
- [ ] If adding a new plugin, add a logo in [the UI repo](https://github.com/apache/skywalking-booster-ui/tree/main/src/assets/img/technologies)
- [ ] I have added the library to `pyproject.toml` (plugin group) by running `poetry add library --group plugins`
- [ ] I have rebuilt the `Plugins.md` documentation by running `make doc-gen`
==== đą Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist đ ==== -->
- [x] If this pull request closes/resolves/fixes an existing issue, replace the issue url.
Closes: https://github.com/apache/skywalking/issues/9939 Closes: https://github.com/apache/skywalking/issues/9071
Closes: https://github.com/apache/skywalking/issues/9721
- [x] Update the [`CHANGELOG.md`](https://github.com/apache/skywalking-python/blob/master/CHANGELOG.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] Superskyyy merged pull request #288: Add prefork support for uwsgi and gunicorn
Posted by "Superskyyy (via GitHub)" <gi...@apache.org>.
Superskyyy merged PR #288:
URL: https://github.com/apache/skywalking-python/pull/288
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] Superskyyy commented on a diff in pull request #288: Add prefork support for uwsgi and gunicorn
Posted by "Superskyyy (via GitHub)" <gi...@apache.org>.
Superskyyy commented on code in PR #288:
URL: https://github.com/apache/skywalking-python/pull/288#discussion_r1110457576
##########
demo/docker-compose.yaml:
##########
@@ -25,11 +25,11 @@ services:
- 12800 # HTTP
networks:
- manual
- environment:
- SW_KAFKA_FETCHER: default
- SW_KAFKA_FETCHER_SERVERS: kafka:9092
- SW_KAFKA_FETCHER_PARTITIONS: 2
- SW_KAFKA_FETCHER_PARTITIONS_FACTOR: 1
+# environment:
+# SW_KAFKA_FETCHER: default
+# SW_KAFKA_FETCHER_SERVERS: kafka:9092
+# SW_KAFKA_FETCHER_PARTITIONS: 2
+# SW_KAFKA_FETCHER_PARTITIONS_FACTOR: 1
Review Comment:
I intentionally left them here for developers to try out the agent, and since kafka is not the default so I commented them out. Should I simply uncomment them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] Superskyyy commented on a diff in pull request #288: Add prefork support for uwsgi and gunicorn
Posted by "Superskyyy (via GitHub)" <gi...@apache.org>.
Superskyyy commented on code in PR #288:
URL: https://github.com/apache/skywalking-python/pull/288#discussion_r1110457401
##########
CHANGELOG.md:
##########
@@ -8,7 +8,8 @@
please check with the latest official documentation before upgrading. (#273, #282)
https://skywalking.apache.org/docs/skywalking-python/v1.0.0/en/setup/configuration/
- **BREAKING**: All agent core capabilities are now covered by test cases and enabled by default (Trace, Log, PVM runtime metrics, Profiler)
-
+ - **BREAKING**: DockerHub Python agent images since v1.0.0 will no longer include the `run` part in `ENTRYPOINT ["sw-python", "run"]`,
+ user should prefix their command with `[-d/--debug] run [-p/--prefork] <Command>` for extra flexibility.
Review Comment:
hmm, I thought this is a true reflection of what the code does, did I understand something wrong?
-d is a global flag to sw-python and -p is a local flag to the `run` option. <command> is user's own application command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] kezhenxu94 commented on a diff in pull request #288: Add prefork support for uwsgi and gunicorn
Posted by "kezhenxu94 (via GitHub)" <gi...@apache.org>.
kezhenxu94 commented on code in PR #288:
URL: https://github.com/apache/skywalking-python/pull/288#discussion_r1110460057
##########
demo/docker-compose.yaml:
##########
@@ -25,11 +25,11 @@ services:
- 12800 # HTTP
networks:
- manual
- environment:
- SW_KAFKA_FETCHER: default
- SW_KAFKA_FETCHER_SERVERS: kafka:9092
- SW_KAFKA_FETCHER_PARTITIONS: 2
- SW_KAFKA_FETCHER_PARTITIONS_FACTOR: 1
+# environment:
+# SW_KAFKA_FETCHER: default
+# SW_KAFKA_FETCHER_SERVERS: kafka:9092
+# SW_KAFKA_FETCHER_PARTITIONS: 2
+# SW_KAFKA_FETCHER_PARTITIONS_FACTOR: 1
Review Comment:
Yes uncommenting them is preferable to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] Superskyyy commented on a diff in pull request #288: Add prefork support for uwsgi and gunicorn
Posted by "Superskyyy (via GitHub)" <gi...@apache.org>.
Superskyyy commented on code in PR #288:
URL: https://github.com/apache/skywalking-python/pull/288#discussion_r1110457401
##########
CHANGELOG.md:
##########
@@ -8,7 +8,8 @@
please check with the latest official documentation before upgrading. (#273, #282)
https://skywalking.apache.org/docs/skywalking-python/v1.0.0/en/setup/configuration/
- **BREAKING**: All agent core capabilities are now covered by test cases and enabled by default (Trace, Log, PVM runtime metrics, Profiler)
-
+ - **BREAKING**: DockerHub Python agent images since v1.0.0 will no longer include the `run` part in `ENTRYPOINT ["sw-python", "run"]`,
+ user should prefix their command with `[-d/--debug] run [-p/--prefork] <Command>` for extra flexibility.
Review Comment:
hmm, I thought this is a true reflection of what the code does, did I understand something wrong?
-d is a global flag to sw-python and -p is a local flag to the `run` option. `<command>` is user's own application command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] sonatype-lift[bot] commented on a diff in pull request #288: Add prefork support for uwsgi and gunicorn
Posted by "sonatype-lift[bot] (via GitHub)" <gi...@apache.org>.
sonatype-lift[bot] commented on code in PR #288:
URL: https://github.com/apache/skywalking-python/pull/288#discussion_r1110377975
##########
tests/e2e/base/consumer/flask_consumer.py:
##########
@@ -0,0 +1,43 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Used by uWSGI which doesn't support FASTAPI (ASGI).
+"""
+import time
+import random
+from flask import Flask, request
+import requests
+
+app = Flask(__name__)
+
+
+@app.route('/artist-consumer', methods=['POST', 'GET'])
+def artist():
+ try:
+ time.sleep(random.random())
Review Comment:
<picture><img alt="0% of developers fix this issue" src="https://lift.sonatype.com/api/commentimage/fixrate/0/display.svg"></picture>
<b>*[B311](https://bandit.readthedocs.io/en/1.7.4/blacklists/blacklist_calls.html#b311-random):</b>* Standard pseudo-random generators are not suitable for security/cryptographic purposes.
ââ <b>2 similar findings have been found in this PR</b>
<details><summary>đ Expand here to view all instances of this finding</summary><br/>
<div align=\"center\">
| **File Path** | **Line Number** |
| ------------- | ------------- |
| tests/e2e/base/provider/fastapi_provider.py | [57](https://github.com/apache/skywalking-python/blob/6c499a60cf8fa7a7f9aa055536e61b884ea9d9d1/tests/e2e/base/provider/fastapi_provider.py#L57) |
| tests/e2e/base/provider/flask_provider.py | [49](https://github.com/apache/skywalking-python/blob/6c499a60cf8fa7a7f9aa055536e61b884ea9d9d1/tests/e2e/base/provider/flask_provider.py#L49) |
<p><a href="https://lift.sonatype.com/results/github.com/apache/skywalking-python/01GSGKRFSXB0N4FF62BSVW83B8?t=Bandit|B311" target="_blank">Visit the Lift Web Console</a> to find more details in your report.</p></div></details>
---
<details><summary>âšī¸ Expand to see all <b>@sonatype-lift</b> commands</summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
<b>Help us improve LIFT! (<i>Sonatype LiftBot external survey</i>)</b>
Was this a good recommendation for you? <sub><small>Answering this survey will not impact your Lift settings.</small></sub>
[â[đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=397622792&lift_comment_rating=1)â] - [â[đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=397622792&lift_comment_rating=2)â] - [â[đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=397622792&lift_comment_rating=3)â] - [â[đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=397622792&lift_comment_rating=4)â] - [â[đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=397622792&lift_comment_rating=5)â]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] kezhenxu94 commented on a diff in pull request #288: Add prefork support for uwsgi and gunicorn
Posted by "kezhenxu94 (via GitHub)" <gi...@apache.org>.
kezhenxu94 commented on code in PR #288:
URL: https://github.com/apache/skywalking-python/pull/288#discussion_r1110455128
##########
demo/docker-compose.yaml:
##########
@@ -55,37 +55,37 @@ services:
environment:
SW_OAP_ADDRESS: "http://oap:12800"
- zookeeper:
- container_name: zk
- image: confluentinc/cp-zookeeper:latest
- ports:
- - "2181:2181"
- environment:
- ZOOKEEPER_CLIENT_PORT: 2181
- ZOOKEEPER_TICK_TIME: 2000
- networks:
- - manual
-
- kafka:
- container_name: kafka
- image: confluentinc/cp-kafka
- expose:
- - 9092
- - 9094
- ports:
- - 9092:9092
- - 9094:9094
- depends_on:
- - zookeeper
- environment:
- KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
- KAFKA_LISTENERS: INTERNAL://0.0.0.0:9092,OUTSIDE://0.0.0.0:9094
- KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://localhost:9094
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
- KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
- networks:
- - manual
+# zookeeper:
Review Comment:
And this. Please also check the other places, I didn't point out all commented places.
##########
demo/docker-compose.yaml:
##########
@@ -38,8 +38,8 @@ services:
ports:
- "12800:12800"
- "11800:11800"
- depends_on:
- - kafka
+# depends_on:
+# - kafka
Review Comment:
And this
##########
demo/docker-compose.yaml:
##########
@@ -25,11 +25,11 @@ services:
- 12800 # HTTP
networks:
- manual
- environment:
- SW_KAFKA_FETCHER: default
- SW_KAFKA_FETCHER_SERVERS: kafka:9092
- SW_KAFKA_FETCHER_PARTITIONS: 2
- SW_KAFKA_FETCHER_PARTITIONS_FACTOR: 1
+# environment:
+# SW_KAFKA_FETCHER: default
+# SW_KAFKA_FETCHER_SERVERS: kafka:9092
+# SW_KAFKA_FETCHER_PARTITIONS: 2
+# SW_KAFKA_FETCHER_PARTITIONS_FACTOR: 1
Review Comment:
Remove these?
##########
CHANGELOG.md:
##########
@@ -8,7 +8,8 @@
please check with the latest official documentation before upgrading. (#273, #282)
https://skywalking.apache.org/docs/skywalking-python/v1.0.0/en/setup/configuration/
- **BREAKING**: All agent core capabilities are now covered by test cases and enabled by default (Trace, Log, PVM runtime metrics, Profiler)
-
+ - **BREAKING**: DockerHub Python agent images since v1.0.0 will no longer include the `run` part in `ENTRYPOINT ["sw-python", "run"]`,
+ user should prefix their command with `[-d/--debug] run [-p/--prefork] <Command>` for extra flexibility.
Review Comment:
This is not true according to what we discussed yesterday and the code in this pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] Superskyyy commented on a diff in pull request #288: Add prefork support for uwsgi and gunicorn
Posted by "Superskyyy (via GitHub)" <gi...@apache.org>.
Superskyyy commented on code in PR #288:
URL: https://github.com/apache/skywalking-python/pull/288#discussion_r1110457401
##########
CHANGELOG.md:
##########
@@ -8,7 +8,8 @@
please check with the latest official documentation before upgrading. (#273, #282)
https://skywalking.apache.org/docs/skywalking-python/v1.0.0/en/setup/configuration/
- **BREAKING**: All agent core capabilities are now covered by test cases and enabled by default (Trace, Log, PVM runtime metrics, Profiler)
-
+ - **BREAKING**: DockerHub Python agent images since v1.0.0 will no longer include the `run` part in `ENTRYPOINT ["sw-python", "run"]`,
+ user should prefix their command with `[-d/--debug] run [-p/--prefork] <Command>` for extra flexibility.
Review Comment:
hmm, I thought this is a true reflection of what the code does, did I understand something wrong?
The said dockerfile change [here](https://github.com/Superskyyy/skywalking-python/blob/6c499a60cf8fa7a7f9aa055536e61b884ea9d9d1/docker/Dockerfile#LL27C2-L27C25)
-d is a global flag to sw-python and -p is a local flag to the `run` option. `<command>` is user's own application command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] kezhenxu94 commented on a diff in pull request #288: Add prefork support for uwsgi and gunicorn
Posted by "kezhenxu94 (via GitHub)" <gi...@apache.org>.
kezhenxu94 commented on code in PR #288:
URL: https://github.com/apache/skywalking-python/pull/288#discussion_r1110460449
##########
CHANGELOG.md:
##########
@@ -8,7 +8,8 @@
please check with the latest official documentation before upgrading. (#273, #282)
https://skywalking.apache.org/docs/skywalking-python/v1.0.0/en/setup/configuration/
- **BREAKING**: All agent core capabilities are now covered by test cases and enabled by default (Trace, Log, PVM runtime metrics, Profiler)
-
+ - **BREAKING**: DockerHub Python agent images since v1.0.0 will no longer include the `run` part in `ENTRYPOINT ["sw-python", "run"]`,
+ user should prefix their command with `[-d/--debug] run [-p/--prefork] <Command>` for extra flexibility.
Review Comment:
đ I read it wrong. NVM ignore this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org