You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "dpgaspar (via GitHub)" <gi...@apache.org> on 2024/08/21 08:10:46 UTC
[PR] fix: trino thread app missing full context [superset]
dpgaspar opened a new pull request, #29981:
URL: https://github.com/apache/superset/pull/29981
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: trino thread app missing full context [superset]
Posted by "joaoferrao (via GitHub)" <gi...@apache.org>.
joaoferrao commented on code in PR #29981:
URL: https://github.com/apache/superset/pull/29981#discussion_r1732781224
##########
superset/db_engine_specs/trino.py:
##########
@@ -227,12 +227,22 @@ def execute_with_cursor(
execute_event = threading.Event()
def _execute(
- results: dict[str, Any], event: threading.Event, app: Flask
+ results: dict[str, Any],
+ event: threading.Event,
+ app: Flask,
+ g_copy: ctx._AppCtxGlobals,
) -> None:
logger.debug("Query %d: Running query: %s", query_id, sql)
try:
+ # Flask contexts are local to the thread that handles the request.
+ # When you spawn a new thread, it does not inherit the contexts
+ # from the parent thread,
+ # meaning the g object and other context-bound variables are not
+ # accessible
with app.app_context():
Review Comment:
Indeed **not** including `@ copy_current_request_context` makes the thread's downstream calls `url_for` function calls fail with the screenshot here https://github.com/apache/superset/pull/27631#issuecomment-2310854789
I found this out in the context adapting #27631 for Trino. My understanding is that, when using OAuth2 config, the `class OAuth2ClientConfigSchema` will be called in a separated thread and has this missing information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: trino thread app missing full context [superset]
Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar commented on PR #29981:
URL: https://github.com/apache/superset/pull/29981#issuecomment-2301973526
Thank you for the review @michael-s-molina
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: trino thread app missing full context [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #29981:
URL: https://github.com/apache/superset/pull/29981#discussion_r1724959807
##########
superset/db_engine_specs/trino.py:
##########
@@ -227,12 +227,22 @@ def execute_with_cursor(
execute_event = threading.Event()
def _execute(
- results: dict[str, Any], event: threading.Event, app: Flask
+ results: dict[str, Any],
+ event: threading.Event,
+ app: Flask,
+ g_copy: ctx._AppCtxGlobals,
) -> None:
logger.debug("Query %d: Running query: %s", query_id, sql)
try:
+ # Flask contexts are local to the thread that handles the request.
+ # When you spawn a new thread, it does not inherit the contexts
+ # from the parent thread,
+ # meaning the g object and other context-bound variables are not
+ # accessible
with app.app_context():
Review Comment:
Can't we use [copy_current_request_context](https://tedboy.github.io/flask/generated/flask.copy_current_request_context.html)?
```
@copy_current_request_context
def worker():
# Access Flask's g object here
# Your code goes here
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: trino thread app missing full context [superset]
Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar commented on code in PR #29981:
URL: https://github.com/apache/superset/pull/29981#discussion_r1724974750
##########
superset/db_engine_specs/trino.py:
##########
@@ -227,12 +227,22 @@ def execute_with_cursor(
execute_event = threading.Event()
def _execute(
- results: dict[str, Any], event: threading.Event, app: Flask
+ results: dict[str, Any],
+ event: threading.Event,
+ app: Flask,
+ g_copy: ctx._AppCtxGlobals,
) -> None:
logger.debug("Query %d: Running query: %s", query_id, sql)
try:
+ # Flask contexts are local to the thread that handles the request.
+ # When you spawn a new thread, it does not inherit the contexts
+ # from the parent thread,
+ # meaning the g object and other context-bound variables are not
+ # accessible
with app.app_context():
Review Comment:
only the request context is copied according to: https://github.com/pallets/flask/issues/3306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: trino thread app missing full context [superset]
Posted by "joaoferrao (via GitHub)" <gi...@apache.org>.
joaoferrao commented on code in PR #29981:
URL: https://github.com/apache/superset/pull/29981#discussion_r1732781224
##########
superset/db_engine_specs/trino.py:
##########
@@ -227,12 +227,22 @@ def execute_with_cursor(
execute_event = threading.Event()
def _execute(
- results: dict[str, Any], event: threading.Event, app: Flask
+ results: dict[str, Any],
+ event: threading.Event,
+ app: Flask,
+ g_copy: ctx._AppCtxGlobals,
) -> None:
logger.debug("Query %d: Running query: %s", query_id, sql)
try:
+ # Flask contexts are local to the thread that handles the request.
+ # When you spawn a new thread, it does not inherit the contexts
+ # from the parent thread,
+ # meaning the g object and other context-bound variables are not
+ # accessible
with app.app_context():
Review Comment:
Indeed **not** including `@ copy_current_request_context` makes the downstream `url_for` function calls fail with the screenshot here https://github.com/apache/superset/pull/27631#issuecomment-2310854789
I found this out in the context adapting #27631 for Trino. My understanding is that, when using OAuth2 config, the `class OAuth2ClientConfigSchema` will be called in a separated thread and has this missing information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: trino thread app missing full context [superset]
Posted by "joaoferrao (via GitHub)" <gi...@apache.org>.
joaoferrao commented on code in PR #29981:
URL: https://github.com/apache/superset/pull/29981#discussion_r1732781224
##########
superset/db_engine_specs/trino.py:
##########
@@ -227,12 +227,22 @@ def execute_with_cursor(
execute_event = threading.Event()
def _execute(
- results: dict[str, Any], event: threading.Event, app: Flask
+ results: dict[str, Any],
+ event: threading.Event,
+ app: Flask,
+ g_copy: ctx._AppCtxGlobals,
) -> None:
logger.debug("Query %d: Running query: %s", query_id, sql)
try:
+ # Flask contexts are local to the thread that handles the request.
+ # When you spawn a new thread, it does not inherit the contexts
+ # from the parent thread,
+ # meaning the g object and other context-bound variables are not
+ # accessible
with app.app_context():
Review Comment:
I'll try to open a PR soon, but turns out that indeed **not** including `@ copy_current_request_context` makes the downstream `url_for` function calls fail with the screenshot here https://github.com/apache/superset/pull/27631#issuecomment-2310854789
I found this out in the context adapting #27631 for Trino. My understanding is that, when using OAuth2 config, the `class OAuth2ClientConfigSchema` will be called in a separated thread and has this missing information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: trino thread app missing full context [superset]
Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar commented on code in PR #29981:
URL: https://github.com/apache/superset/pull/29981#discussion_r1724952403
##########
superset/db_engine_specs/trino.py:
##########
@@ -227,12 +227,22 @@ def execute_with_cursor(
execute_event = threading.Event()
def _execute(
- results: dict[str, Any], event: threading.Event, app: Flask
+ results: dict[str, Any],
+ event: threading.Event,
+ app: Flask,
+ g_copy: ctx._AppCtxGlobals,
) -> None:
logger.debug("Query %d: Running query: %s", query_id, sql)
try:
+ # Flask contexts are local to the thread that handles the request.
+ # When you spawn a new thread, it does not inherit the contexts
+ # from the parent thread,
+ # meaning the g object and other context-bound variables are not
+ # accessible
with app.app_context():
Review Comment:
seems basically similar, except I'm passing the current_app into the thread, also we need to recreate g inside the thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: trino thread app missing full context [superset]
Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar merged PR #29981:
URL: https://github.com/apache/superset/pull/29981
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: trino thread app missing full context [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #29981:
URL: https://github.com/apache/superset/pull/29981#issuecomment-2301440422
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/29981?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `50.00000%` with `1 line` in your changes missing coverage. Please review.
> Project coverage is 65.05%. Comparing base [(`76d897e`)](https://app.codecov.io/gh/apache/superset/commit/76d897eaa2f9e137102bc194c2e3109c29d0348f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`8c902ce`)](https://app.codecov.io/gh/apache/superset/commit/8c902ce8d712c1b001fc68c059c36de4ddec69c8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 605 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/superset/pull/29981?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/db\_engine\_specs/trino.py](https://app.codecov.io/gh/apache/superset/pull/29981?src=pr&el=tree&filepath=superset%2Fdb_engine_specs%2Ftrino.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | 50.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/29981?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #29981 +/- ##
==========================================
+ Coverage 60.48% 65.05% +4.56%
==========================================
Files 1931 528 -1403
Lines 76236 38150 -38086
Branches 8568 0 -8568
==========================================
- Hits 46114 24818 -21296
+ Misses 28017 13332 -14685
+ Partials 2105 0 -2105
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/29981/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/29981/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.95% <0.00%> (-0.22%)` | :arrow_down: |
| [javascript](https://app.codecov.io/gh/apache/superset/pull/29981/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/29981/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.50% <0.00%> (-0.31%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/superset/pull/29981/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.05% <50.00%> (+1.56%)` | :arrow_up: |
| [unit](https://app.codecov.io/gh/apache/superset/pull/29981/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.31% <50.00%> (+2.68%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/29981?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: trino thread app missing full context [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #29981:
URL: https://github.com/apache/superset/pull/29981#discussion_r1724949591
##########
superset/db_engine_specs/trino.py:
##########
@@ -227,12 +227,22 @@ def execute_with_cursor(
execute_event = threading.Event()
def _execute(
- results: dict[str, Any], event: threading.Event, app: Flask
+ results: dict[str, Any],
+ event: threading.Event,
+ app: Flask,
+ g_copy: ctx._AppCtxGlobals,
) -> None:
logger.debug("Query %d: Running query: %s", query_id, sql)
try:
+ # Flask contexts are local to the thread that handles the request.
+ # When you spawn a new thread, it does not inherit the contexts
+ # from the parent thread,
+ # meaning the g object and other context-bound variables are not
+ # accessible
with app.app_context():
Review Comment:
@dpgaspar I did a quick GPT search.
```
from flask import current_app
import threading
def my_thread():
with current_app.app_context():
# Access Flask context within the thread
app = current_app._get_current_object()
# Do something with the app
# Create and start the thread
thread = threading.Thread(target=my_thread)
thread.start()
```
Would something like this work for our use cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org