You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2022/11/09 12:09:19 UTC
[GitHub] [skywalking-python] westarest opened a new pull request, #250: Doc for greenlet profiler
westarest opened a new pull request, #250:
URL: https://github.com/apache/skywalking-python/pull/250
This is a simple document describes how the threading profiler and greenlet profiler work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] wu-sheng merged pull request #250: Doc for greenlet profiler
Posted by GitBox <gi...@apache.org>.
wu-sheng merged PR #250:
URL: https://github.com/apache/skywalking-python/pull/250
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] wu-sheng commented on pull request #250: Doc for greenlet profiler
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #250:
URL: https://github.com/apache/skywalking-python/pull/250#issuecomment-1308694267
drawio files should be placed in the website, side by side with final png. @kezhenxu94 Is this good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] Superskyyy commented on a diff in pull request #250: Doc for greenlet profiler
Posted by GitBox <gi...@apache.org>.
Superskyyy commented on code in PR #250:
URL: https://github.com/apache/skywalking-python/pull/250#discussion_r1024803641
##########
docs/en/profiling/profiling.md:
##########
@@ -0,0 +1,83 @@
+# How does threading-profiler (the default mode) work
+
+These blogs [skywalking-profiling](https://skywalking.apache.org/blog/2020-04-13-apache-skywalking-profiling/#profiling-in-production) and [skywalking-python-profiling](https://skywalking.apache.org/blog/2021-09-12-skywalking-python-profiling/) described how the threading-profiler works
+
+And this figure demonstrates how the profiler works as well:
+
+
+```mermaid
+sequenceDiagram
+ API->>+working thread: get: /api/v1/user/
+ rect rgb(0,200,0)
+ API->>+profiling thread: start profiling
+ profiling thread->>working thread: snapshot
+ profiling thread->>working thread: snapshot
+ profiling thread->>working thread: snapshot
+ profiling thread->>-working thread: snapshot
+ end
+ working thread-->>-API: response
+```
+
+
+It works well with threading mode because the whole process will be executed in **the same** thread, so the profiling thread can fetch the complete profiling info of the process of the API request.
+
+# Why doesn't threading-profiler work in greenlet mode
+When the python program runs with gevent + greenlet, the process would be like this:
+
+
+
+```mermaid
+sequenceDiagram
+ API->>+working thread 1: get: /api/v1/user/
+ rect rgb(0,200,0)
+ greenlet.HUB-->>+working thread 1: swap in the profiled greenlet
+ API->>+profiling thread: start profiling
+ profiling thread->>working thread 1: snapshot
+ working thread 1-->>-greenlet.HUB : swap out the profiled greenlet
+ end
+ greenlet.HUB-->>+working thread 1: swap in the other greenlet
+ profiling thread->>working thread 1: snapshot
+ greenlet.HUB-->>+working thread 2: swap in the profiled greenlet
+ profiling thread->>working thread 1: snapshot
+ profiling thread->>working thread 1: snapshot
+ working thread 2-->-greenlet.HUB : swap out the profiled greenlet
+ profiling thread->>working thread 1: snapshot
+ profiling thread->>-working thread 1: snapshot
+ working thread 1-->>-greenlet.HUB : swap out the other greenlet
+ working thread 1-->>-API: response
+```
+
+
+In this circumstance, the snapshot of the working thread includes multi contexts of different greenlets, which will make skywalking confused to build the trace stack.
+
+
+Fortunately, greenlet has an API for profiling, the doc is [here](https://greenlet.readthedocs.io/en/latest/tracing.html). We can implement a greenlet profiler to solve this issue.
+
+
+# How the greenlet profiler works
+
+A greenlet profiler leverages the trace callback of greenlet, it works like this:
+
+
+```mermaid
+sequenceDiagram
+ API->>+working thread 1: get: /api/v1/user/
+ rect rgb(0,200,0)
+ greenlet.HUB-->>+working thread 1: swap in the profiled greenlet and snapshot
+ working thread 1-->>-greenlet.HUB : swap out the profiled greenlet and snapshot
+ end
+ greenlet.HUB-->>+working thread 1: swap in the other greenlet
+ rect rgb(0,200,0)
+ greenlet.HUB-->>+working thread 2: swap in the profiled greenlet and snapshot
+ working thread 2-->-greenlet.HUB : swap out the profiled greenlet and snapshot
+ end
+ working thread 1-->>-greenlet.HUB : swap out the other greenlet
+ working thread 1-->>-API: response
+```
+
+We can set a callback function to the greenlet that we need to profiling, then when the greenlet.HUB switches the context in/out to the working thread, the callback will build a snapshot of the greenlet's traceback and send it to skywalking.
+
+
+# the difference between these two profilers
Review Comment:
```suggestion
# The difference between these two profilers
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] kezhenxu94 commented on pull request #250: Doc for greenlet profiler
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on PR #250:
URL: https://github.com/apache/skywalking-python/pull/250#issuecomment-1308698234
Can you consider using [mermaid](https://mermaid-js.github.io/mermaid/#/) syntax to draw? GitHub natively supports this framework and our website also support that so it works both in GitHub website and our website. Also it's simple markdown language.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] Superskyyy commented on pull request #250: Doc for greenlet profiler
Posted by GitBox <gi...@apache.org>.
Superskyyy commented on PR #250:
URL: https://github.com/apache/skywalking-python/pull/250#issuecomment-1316224142
Sorry for the delay I will take a look soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] westarest commented on pull request #250: Doc for greenlet profiler
Posted by GitBox <gi...@apache.org>.
westarest commented on PR #250:
URL: https://github.com/apache/skywalking-python/pull/250#issuecomment-1308691319
> All graphics should be submitted to https://github.com/apache/skywalking-website/tree/master/static/doc-graph + `/python/1.0.0`. Then after the website is deployed, you could use the static links. Ref to what we did in other repositories. [apache/skywalking-banyandb#198 (comment)](https://github.com/apache/skywalking-banyandb/pull/198#discussion_r1017868651)
> All graphics should be submitted to https://github.com/apache/skywalking-website/tree/master/static/doc-graph + `/python/1.0.0`. Then after the website is deployed, you could use the static links. Ref to what we did in other repositories. [apache/skywalking-banyandb#198 (comment)](https://github.com/apache/skywalking-banyandb/pull/198#discussion_r1017868651)
Should I commit the .drawio files or only the PNG files into this repo?
And if we need the drawio files, what's the license format for the XML(drawio) file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] wu-sheng commented on pull request #250: Doc for greenlet profiler
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #250:
URL: https://github.com/apache/skywalking-python/pull/250#issuecomment-1308674471
All graphics should be submitted to https://github.com/apache/skywalking-website/tree/master/static/doc-graph + `/python/1.0.0`.
Then after the website is deployed, you could use the static links.
Ref to what we did in other repositories. https://github.com/apache/skywalking-banyandb/pull/198#discussion_r1017868651
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] wu-sheng commented on pull request #250: Doc for greenlet profiler
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #250:
URL: https://github.com/apache/skywalking-python/pull/250#issuecomment-1308684988
> ignore lic checks for .drawio
These files matter. They can't be ignored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] Superskyyy commented on pull request #250: Doc for greenlet profiler
Posted by GitBox <gi...@apache.org>.
Superskyyy commented on PR #250:
URL: https://github.com/apache/skywalking-python/pull/250#issuecomment-1318184696
Looking good to me, thank you for the important contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-python] kezhenxu94 commented on a diff in pull request #250: Doc for greenlet profiler
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #250:
URL: https://github.com/apache/skywalking-python/pull/250#discussion_r1020675932
##########
docs/en/profiling/profiling.md:
##########
@@ -0,0 +1,83 @@
+# how does threading-profiler (the default mode) work
Review Comment:
```suggestion
# How does threading-profiler (the default mode) work
```
##########
docs/en/profiling/profiling.md:
##########
@@ -0,0 +1,83 @@
+# how does threading-profiler (the default mode) work
+
+These [skywalking-profiling](https://skywalking.apache.org/blog/2020-04-13-apache-skywalking-profiling/#profiling-in-production) [skywalking-python-profiling](https://skywalking.apache.org/blog/2021-09-12-skywalking-python-profiling/) documents described how the threading-profiler works
+
+And this figure demonstrated how the profiler works as well:
+
+
+```mermaid
+sequenceDiagram
+ API->>+working thread: get: /api/v1/user/
+ rect rgb(0,200,0)
+ API->>+profiling thread: start profiling
+ profiling thread->>working thread: snapshot
+ profiling thread->>working thread: snapshot
+ profiling thread->>working thread: snapshot
+ profiling thread->>-working thread: snapshot
+ end
+ working thread-->>-API: response
+```
+
+
+It works well with threading mode because the whole process will be executed in **the same** thread, so the profiling thread can fetch the complete profiling info of the process of the API request.
+
+# why doesn't threading-profiler work in greenlet mode
Review Comment:
```suggestion
# Why doesn't threading-profiler work in greenlet mode
```
##########
docs/en/profiling/profiling.md:
##########
@@ -0,0 +1,83 @@
+# how does threading-profiler (the default mode) work
+
+These [skywalking-profiling](https://skywalking.apache.org/blog/2020-04-13-apache-skywalking-profiling/#profiling-in-production) [skywalking-python-profiling](https://skywalking.apache.org/blog/2021-09-12-skywalking-python-profiling/) documents described how the threading-profiler works
Review Comment:
```suggestion
These blogs [skywalking-profiling](https://skywalking.apache.org/blog/2020-04-13-apache-skywalking-profiling/#profiling-in-production) and [skywalking-python-profiling](https://skywalking.apache.org/blog/2021-09-12-skywalking-python-profiling/) described how the threading-profiler works
```
##########
docs/en/profiling/profiling.md:
##########
@@ -0,0 +1,83 @@
+# how does threading-profiler (the default mode) work
+
+These [skywalking-profiling](https://skywalking.apache.org/blog/2020-04-13-apache-skywalking-profiling/#profiling-in-production) [skywalking-python-profiling](https://skywalking.apache.org/blog/2021-09-12-skywalking-python-profiling/) documents described how the threading-profiler works
+
+And this figure demonstrated how the profiler works as well:
+
+
+```mermaid
+sequenceDiagram
+ API->>+working thread: get: /api/v1/user/
+ rect rgb(0,200,0)
+ API->>+profiling thread: start profiling
+ profiling thread->>working thread: snapshot
+ profiling thread->>working thread: snapshot
+ profiling thread->>working thread: snapshot
+ profiling thread->>-working thread: snapshot
+ end
+ working thread-->>-API: response
+```
+
+
+It works well with threading mode because the whole process will be executed in **the same** thread, so the profiling thread can fetch the complete profiling info of the process of the API request.
+
+# why doesn't threading-profiler work in greenlet mode
+When the python program runs with gevent + greenlet, the process would be like this:
+
+
+
+```mermaid
+sequenceDiagram
+ API->>+working thread 1: get: /api/v1/user/
+ rect rgb(0,200,0)
+ greenlet.HUB-->>+working thread 1: swap in the profiled greenlet
+ API->>+profiling thread: start profiling
+ profiling thread->>working thread 1: snapshot
+ working thread 1-->>-greenlet.HUB : swap out the profiled greenlet
+ end
+ greenlet.HUB-->>+working thread 1: swap in the other greenlet
+ profiling thread->>working thread 1: snapshot
+ greenlet.HUB-->>+working thread 2: swap in the profiled greenlet
+ profiling thread->>working thread 1: snapshot
+ profiling thread->>working thread 1: snapshot
+ working thread 2-->-greenlet.HUB : swap out the profiled greenlet
+ profiling thread->>working thread 1: snapshot
+ profiling thread->>-working thread 1: snapshot
+ working thread 1-->>-greenlet.HUB : swap out the other greenlet
+ working thread 1-->>-API: response
+```
+
+
+In this circumstance, the snapshot of the working thread includes multi contexts of different greenlets, which will make skywalking confusing to build the trace stack.
Review Comment:
```suggestion
In this circumstance, the snapshot of the working thread includes multi contexts of different greenlets, which will make skywalking confused to build the trace stack.
```
##########
docs/en/profiling/profiling.md:
##########
@@ -0,0 +1,83 @@
+# how does threading-profiler (the default mode) work
+
+These [skywalking-profiling](https://skywalking.apache.org/blog/2020-04-13-apache-skywalking-profiling/#profiling-in-production) [skywalking-python-profiling](https://skywalking.apache.org/blog/2021-09-12-skywalking-python-profiling/) documents described how the threading-profiler works
+
+And this figure demonstrated how the profiler works as well:
+
+
+```mermaid
+sequenceDiagram
+ API->>+working thread: get: /api/v1/user/
+ rect rgb(0,200,0)
+ API->>+profiling thread: start profiling
+ profiling thread->>working thread: snapshot
+ profiling thread->>working thread: snapshot
+ profiling thread->>working thread: snapshot
+ profiling thread->>-working thread: snapshot
+ end
+ working thread-->>-API: response
+```
+
+
+It works well with threading mode because the whole process will be executed in **the same** thread, so the profiling thread can fetch the complete profiling info of the process of the API request.
+
+# why doesn't threading-profiler work in greenlet mode
+When the python program runs with gevent + greenlet, the process would be like this:
+
+
+
+```mermaid
+sequenceDiagram
+ API->>+working thread 1: get: /api/v1/user/
+ rect rgb(0,200,0)
+ greenlet.HUB-->>+working thread 1: swap in the profiled greenlet
+ API->>+profiling thread: start profiling
+ profiling thread->>working thread 1: snapshot
+ working thread 1-->>-greenlet.HUB : swap out the profiled greenlet
+ end
+ greenlet.HUB-->>+working thread 1: swap in the other greenlet
+ profiling thread->>working thread 1: snapshot
+ greenlet.HUB-->>+working thread 2: swap in the profiled greenlet
+ profiling thread->>working thread 1: snapshot
+ profiling thread->>working thread 1: snapshot
+ working thread 2-->-greenlet.HUB : swap out the profiled greenlet
+ profiling thread->>working thread 1: snapshot
+ profiling thread->>-working thread 1: snapshot
+ working thread 1-->>-greenlet.HUB : swap out the other greenlet
+ working thread 1-->>-API: response
+```
+
+
+In this circumstance, the snapshot of the working thread includes multi contexts of different greenlets, which will make skywalking confusing to build the trace stack.
+
+
+Fortunately, greenlet has an API for profiling, the doc is [here](https://greenlet.readthedocs.io/en/latest/tracing.html). We can implement a greenlet profiler to solute this issue.
+
+
+# how the greenlet profiler works
Review Comment:
```suggestion
# How the greenlet profiler works
```
##########
docs/en/profiling/profiling.md:
##########
@@ -0,0 +1,83 @@
+# how does threading-profiler (the default mode) work
+
+These [skywalking-profiling](https://skywalking.apache.org/blog/2020-04-13-apache-skywalking-profiling/#profiling-in-production) [skywalking-python-profiling](https://skywalking.apache.org/blog/2021-09-12-skywalking-python-profiling/) documents described how the threading-profiler works
+
+And this figure demonstrated how the profiler works as well:
+
+
+```mermaid
+sequenceDiagram
+ API->>+working thread: get: /api/v1/user/
+ rect rgb(0,200,0)
+ API->>+profiling thread: start profiling
+ profiling thread->>working thread: snapshot
+ profiling thread->>working thread: snapshot
+ profiling thread->>working thread: snapshot
+ profiling thread->>-working thread: snapshot
+ end
+ working thread-->>-API: response
+```
+
+
+It works well with threading mode because the whole process will be executed in **the same** thread, so the profiling thread can fetch the complete profiling info of the process of the API request.
+
+# why doesn't threading-profiler work in greenlet mode
+When the python program runs with gevent + greenlet, the process would be like this:
+
+
+
+```mermaid
+sequenceDiagram
+ API->>+working thread 1: get: /api/v1/user/
+ rect rgb(0,200,0)
+ greenlet.HUB-->>+working thread 1: swap in the profiled greenlet
+ API->>+profiling thread: start profiling
+ profiling thread->>working thread 1: snapshot
+ working thread 1-->>-greenlet.HUB : swap out the profiled greenlet
+ end
+ greenlet.HUB-->>+working thread 1: swap in the other greenlet
+ profiling thread->>working thread 1: snapshot
+ greenlet.HUB-->>+working thread 2: swap in the profiled greenlet
+ profiling thread->>working thread 1: snapshot
+ profiling thread->>working thread 1: snapshot
+ working thread 2-->-greenlet.HUB : swap out the profiled greenlet
+ profiling thread->>working thread 1: snapshot
+ profiling thread->>-working thread 1: snapshot
+ working thread 1-->>-greenlet.HUB : swap out the other greenlet
+ working thread 1-->>-API: response
+```
+
+
+In this circumstance, the snapshot of the working thread includes multi contexts of different greenlets, which will make skywalking confusing to build the trace stack.
+
+
+Fortunately, greenlet has an API for profiling, the doc is [here](https://greenlet.readthedocs.io/en/latest/tracing.html). We can implement a greenlet profiler to solute this issue.
Review Comment:
```suggestion
Fortunately, greenlet has an API for profiling, the doc is [here](https://greenlet.readthedocs.io/en/latest/tracing.html). We can implement a greenlet profiler to solve this issue.
```
##########
docs/en/profiling/profiling.md:
##########
@@ -0,0 +1,83 @@
+# how does threading-profiler (the default mode) work
+
+These [skywalking-profiling](https://skywalking.apache.org/blog/2020-04-13-apache-skywalking-profiling/#profiling-in-production) [skywalking-python-profiling](https://skywalking.apache.org/blog/2021-09-12-skywalking-python-profiling/) documents described how the threading-profiler works
+
+And this figure demonstrated how the profiler works as well:
Review Comment:
```suggestion
And this figure demonstrates how the profiler works as well:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org