You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "dpgaspar (via GitHub)" <gi...@apache.org> on 2024/08/21 08:10:46 UTC

[PR] fix: trino thread app missing full context [superset]

dpgaspar opened a new pull request, #29981:
URL: https://github.com/apache/superset/pull/29981

   <!---
   Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
   Example:
   fix(dashboard): load charts correctly
   -->
   
   ### SUMMARY
   <!--- Describe the change below, including rationale and design decisions -->
   
   ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
   <!--- Skip this if not applicable -->
   
   ### TESTING INSTRUCTIONS
   <!--- Required! What steps can be taken to manually verify the changes? -->
   
   ### ADDITIONAL INFORMATION
   <!--- Check any relevant boxes with "x" -->
   <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
   - [ ] Has associated issue:
   - [ ] Required feature flags:
   - [ ] Changes UI
   - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
     - [ ] Migration is atomic, supports rollback & is backwards-compatible
     - [ ] Confirm DB migration upgrade and downgrade tested
     - [ ] Runtime estimates and downtime expectations provided
   - [ ] Introduces new feature or API
   - [ ] Removes existing feature or API
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix: trino thread app missing full context [superset]

Posted by "joaoferrao (via GitHub)" <gi...@apache.org>.
joaoferrao commented on code in PR #29981:
URL: https://github.com/apache/superset/pull/29981#discussion_r1732781224


##########
superset/db_engine_specs/trino.py:
##########
@@ -227,12 +227,22 @@ def execute_with_cursor(
         execute_event = threading.Event()
 
         def _execute(
-            results: dict[str, Any], event: threading.Event, app: Flask
+            results: dict[str, Any],
+            event: threading.Event,
+            app: Flask,
+            g_copy: ctx._AppCtxGlobals,
         ) -> None:
             logger.debug("Query %d: Running query: %s", query_id, sql)
 
             try:
+                # Flask contexts are local to the thread that handles the request.
+                # When you spawn a new thread, it does not inherit the contexts
+                # from the parent thread,
+                # meaning the g object and other context-bound variables are not
+                # accessible
                 with app.app_context():

Review Comment:
   Indeed **not** including `@ copy_current_request_context` makes the thread's downstream calls `url_for` function calls fail with the screenshot here https://github.com/apache/superset/pull/27631#issuecomment-2310854789
   
   I found this out in the context adapting #27631 for Trino. My understanding is that, when using OAuth2 config, the `class OAuth2ClientConfigSchema` will be called in a separated thread and has this missing information.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix: trino thread app missing full context [superset]

Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar commented on PR #29981:
URL: https://github.com/apache/superset/pull/29981#issuecomment-2301973526

   Thank you for the review @michael-s-molina 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix: trino thread app missing full context [superset]

Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #29981:
URL: https://github.com/apache/superset/pull/29981#discussion_r1724959807


##########
superset/db_engine_specs/trino.py:
##########
@@ -227,12 +227,22 @@ def execute_with_cursor(
         execute_event = threading.Event()
 
         def _execute(
-            results: dict[str, Any], event: threading.Event, app: Flask
+            results: dict[str, Any],
+            event: threading.Event,
+            app: Flask,
+            g_copy: ctx._AppCtxGlobals,
         ) -> None:
             logger.debug("Query %d: Running query: %s", query_id, sql)
 
             try:
+                # Flask contexts are local to the thread that handles the request.
+                # When you spawn a new thread, it does not inherit the contexts
+                # from the parent thread,
+                # meaning the g object and other context-bound variables are not
+                # accessible
                 with app.app_context():

Review Comment:
   Can't we use [copy_current_request_context](https://tedboy.github.io/flask/generated/flask.copy_current_request_context.html)?
   
   ```
   @copy_current_request_context
   def worker():
       # Access Flask's g object here
       # Your code goes here
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix: trino thread app missing full context [superset]

Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar commented on code in PR #29981:
URL: https://github.com/apache/superset/pull/29981#discussion_r1724974750


##########
superset/db_engine_specs/trino.py:
##########
@@ -227,12 +227,22 @@ def execute_with_cursor(
         execute_event = threading.Event()
 
         def _execute(
-            results: dict[str, Any], event: threading.Event, app: Flask
+            results: dict[str, Any],
+            event: threading.Event,
+            app: Flask,
+            g_copy: ctx._AppCtxGlobals,
         ) -> None:
             logger.debug("Query %d: Running query: %s", query_id, sql)
 
             try:
+                # Flask contexts are local to the thread that handles the request.
+                # When you spawn a new thread, it does not inherit the contexts
+                # from the parent thread,
+                # meaning the g object and other context-bound variables are not
+                # accessible
                 with app.app_context():

Review Comment:
   only the request context is copied according to: https://github.com/pallets/flask/issues/3306



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix: trino thread app missing full context [superset]

Posted by "joaoferrao (via GitHub)" <gi...@apache.org>.
joaoferrao commented on code in PR #29981:
URL: https://github.com/apache/superset/pull/29981#discussion_r1732781224


##########
superset/db_engine_specs/trino.py:
##########
@@ -227,12 +227,22 @@ def execute_with_cursor(
         execute_event = threading.Event()
 
         def _execute(
-            results: dict[str, Any], event: threading.Event, app: Flask
+            results: dict[str, Any],
+            event: threading.Event,
+            app: Flask,
+            g_copy: ctx._AppCtxGlobals,
         ) -> None:
             logger.debug("Query %d: Running query: %s", query_id, sql)
 
             try:
+                # Flask contexts are local to the thread that handles the request.
+                # When you spawn a new thread, it does not inherit the contexts
+                # from the parent thread,
+                # meaning the g object and other context-bound variables are not
+                # accessible
                 with app.app_context():

Review Comment:
   Indeed **not** including `@ copy_current_request_context` makes the downstream `url_for` function calls fail with the screenshot here https://github.com/apache/superset/pull/27631#issuecomment-2310854789
   
   I found this out in the context adapting #27631 for Trino. My understanding is that, when using OAuth2 config, the `class OAuth2ClientConfigSchema` will be called in a separated thread and has this missing information.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix: trino thread app missing full context [superset]

Posted by "joaoferrao (via GitHub)" <gi...@apache.org>.
joaoferrao commented on code in PR #29981:
URL: https://github.com/apache/superset/pull/29981#discussion_r1732781224


##########
superset/db_engine_specs/trino.py:
##########
@@ -227,12 +227,22 @@ def execute_with_cursor(
         execute_event = threading.Event()
 
         def _execute(
-            results: dict[str, Any], event: threading.Event, app: Flask
+            results: dict[str, Any],
+            event: threading.Event,
+            app: Flask,
+            g_copy: ctx._AppCtxGlobals,
         ) -> None:
             logger.debug("Query %d: Running query: %s", query_id, sql)
 
             try:
+                # Flask contexts are local to the thread that handles the request.
+                # When you spawn a new thread, it does not inherit the contexts
+                # from the parent thread,
+                # meaning the g object and other context-bound variables are not
+                # accessible
                 with app.app_context():

Review Comment:
   I'll try to open a PR soon, but turns out that indeed **not** including `@ copy_current_request_context` makes the downstream `url_for` function calls fail with the screenshot here https://github.com/apache/superset/pull/27631#issuecomment-2310854789
   
   I found this out in the context adapting #27631 for Trino. My understanding is that, when using OAuth2 config, the `class OAuth2ClientConfigSchema` will be called in a separated thread and has this missing information.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix: trino thread app missing full context [superset]

Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar commented on code in PR #29981:
URL: https://github.com/apache/superset/pull/29981#discussion_r1724952403


##########
superset/db_engine_specs/trino.py:
##########
@@ -227,12 +227,22 @@ def execute_with_cursor(
         execute_event = threading.Event()
 
         def _execute(
-            results: dict[str, Any], event: threading.Event, app: Flask
+            results: dict[str, Any],
+            event: threading.Event,
+            app: Flask,
+            g_copy: ctx._AppCtxGlobals,
         ) -> None:
             logger.debug("Query %d: Running query: %s", query_id, sql)
 
             try:
+                # Flask contexts are local to the thread that handles the request.
+                # When you spawn a new thread, it does not inherit the contexts
+                # from the parent thread,
+                # meaning the g object and other context-bound variables are not
+                # accessible
                 with app.app_context():

Review Comment:
   seems basically similar, except I'm passing the current_app into the thread, also we need to recreate g inside the thread



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix: trino thread app missing full context [superset]

Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar merged PR #29981:
URL: https://github.com/apache/superset/pull/29981


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix: trino thread app missing full context [superset]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #29981:
URL: https://github.com/apache/superset/pull/29981#issuecomment-2301440422

   ## [Codecov](https://app.codecov.io/gh/apache/superset/pull/29981?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `50.00000%` with `1 line` in your changes missing coverage. Please review.
   > Project coverage is 65.05%. Comparing base [(`76d897e`)](https://app.codecov.io/gh/apache/superset/commit/76d897eaa2f9e137102bc194c2e3109c29d0348f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`8c902ce`)](https://app.codecov.io/gh/apache/superset/commit/8c902ce8d712c1b001fc68c059c36de4ddec69c8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 605 commits behind head on master.
   
   | [Files](https://app.codecov.io/gh/apache/superset/pull/29981?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [superset/db\_engine\_specs/trino.py](https://app.codecov.io/gh/apache/superset/pull/29981?src=pr&el=tree&filepath=superset%2Fdb_engine_specs%2Ftrino.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | 50.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/29981?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #29981      +/-   ##
   ==========================================
   + Coverage   60.48%   65.05%   +4.56%     
   ==========================================
     Files        1931      528    -1403     
     Lines       76236    38150   -38086     
     Branches     8568        0    -8568     
   ==========================================
   - Hits        46114    24818   -21296     
   + Misses      28017    13332   -14685     
   + Partials     2105        0    -2105     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/superset/pull/29981/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hive](https://app.codecov.io/gh/apache/superset/pull/29981/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.95% <0.00%> (-0.22%)` | :arrow_down: |
   | [javascript](https://app.codecov.io/gh/apache/superset/pull/29981/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [presto](https://app.codecov.io/gh/apache/superset/pull/29981/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.50% <0.00%> (-0.31%)` | :arrow_down: |
   | [python](https://app.codecov.io/gh/apache/superset/pull/29981/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.05% <50.00%> (+1.56%)` | :arrow_up: |
   | [unit](https://app.codecov.io/gh/apache/superset/pull/29981/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.31% <50.00%> (+2.68%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/29981?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix: trino thread app missing full context [superset]

Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #29981:
URL: https://github.com/apache/superset/pull/29981#discussion_r1724949591


##########
superset/db_engine_specs/trino.py:
##########
@@ -227,12 +227,22 @@ def execute_with_cursor(
         execute_event = threading.Event()
 
         def _execute(
-            results: dict[str, Any], event: threading.Event, app: Flask
+            results: dict[str, Any],
+            event: threading.Event,
+            app: Flask,
+            g_copy: ctx._AppCtxGlobals,
         ) -> None:
             logger.debug("Query %d: Running query: %s", query_id, sql)
 
             try:
+                # Flask contexts are local to the thread that handles the request.
+                # When you spawn a new thread, it does not inherit the contexts
+                # from the parent thread,
+                # meaning the g object and other context-bound variables are not
+                # accessible
                 with app.app_context():

Review Comment:
   @dpgaspar I did a quick GPT search.
   
   ```
   from flask import current_app
   import threading
   
   def my_thread():
       with current_app.app_context():
           # Access Flask context within the thread
           app = current_app._get_current_object()
           # Do something with the app
   
   # Create and start the thread
   thread = threading.Thread(target=my_thread)
   thread.start()
   ```
   
   Would something like this work for our use cases?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org