You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2021/08/11 21:00:15 UTC

[GitHub] [superset] sabiroid opened a new issue #16209: [SIP-71] API to pass metadata over to Celery workers

sabiroid opened a new issue #16209:
URL: https://github.com/apache/superset/issues/16209


   ## [SIP-71] Proposal for API to pass metadata over to Celery workers
   
   ### Motivation
   
   There are several types of metadata required to run analytical queries. Among them are user-impersonating authentication tokens and granular source information (e.g. dashboard or chart IDs). Unfortunately, this "global" information is not available on the Celery workers since they run remotely in a different Flask context. This proposal suggests to add Superset config API to pass required "global" data over so user code on Celery worker is able to utilize it.
   
   ### Proposed Change
   
   Add  `CELERY_EXTRACT_METADATA` config function allowing to save all necessary data from `flask.g`, `flask.request` and `flask.context` on the request side and `CELERY_INIT_METADATA` config function restoring the data on the Celery worker side.
   
   ### New or Changed Public Interfaces
   
   Superset config will just have new `CELERY_EXTRACT_METADATA` and `CELERY_INIT_METADATA` config functions.
   
   ### New dependencies
   
   None
   
   ### Migration Plan and Compatibility
   
   The API is completely backward compatible
   
   ### Rejected Alternatives
   
   * I thought of just having `CELERY_PASSOVER_METADATA` config tuple with names of fields of `flask.g`, but it would not allow to populate `flask.g` with custom fields (mb derived from `flask.request` or `flask.session`), and adding a new functionality to populate `flask.g` in the first place would become as complex as the proposal while being way less intuitive.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] sabiroid commented on issue #16209: [SIP-71] API to pass metadata over to Celery workers

Posted by GitBox <gi...@apache.org>.
sabiroid commented on issue #16209:
URL: https://github.com/apache/superset/issues/16209#issuecomment-902117789


   Technically, all the necessary information (though maybe not in the most convenient form) can be found in `flask.g`, `flask.context`, and `flask.request`. At least, so far I haven't encountered any critical information truly missing there.
   
   I believe, we did not alter the Superset code too much (that's why I also want to eventually push this feature upstream so we have the minimal possible fork), we were mostly achieving our goals through passing over our own config file(s).
   
   In our case the authentication is done when the user logs in the Superset, so all synchronous requests are sorted out. But for asynchronous ones we need to pass the token over since the Celery worker wouldn't have any authorization otherwise.
   
   On top of that we also need Celery workers to have additional information about the query they are executing like user name and dashboard/chart ID, so we can set up proper resource management configuration in Presto (so, say, a single really bad dashboard with many unoptimized queries doesn't screw up the whole Presto cluster)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] craig-rueda commented on issue #16209: [SIP-71] API to pass metadata over to Celery workers

Posted by GitBox <gi...@apache.org>.
craig-rueda commented on issue #16209:
URL: https://github.com/apache/superset/issues/16209#issuecomment-902401717


   LGTM as long as the new config options default to a no-op, which I'm sure you were planning anyways. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] sabiroid edited a comment on issue #16209: [SIP-71] API to pass metadata over to Celery workers

Posted by GitBox <gi...@apache.org>.
sabiroid edited a comment on issue #16209:
URL: https://github.com/apache/superset/issues/16209#issuecomment-902121572


   @mistercrunch, when you say "make an inventory of all the places we read global context", do you mean to pass references to those places over to `CELERY_FLASK_METADATA_EXTRACTOR` and `CELERY_FLASK_METADATA_INITIALIZER`? That sounds like a really good idea, so the user setting Superset up won't need to import `flask` context-global variables themselves in their code.
   
   By the way, just to clarify, on the async workers we don't read the global context by `CELERY_FLASK_METADATA_INITIALIZER`, we write into it. But we read it by `CELERY_FLASK_METADATA_EXTRACTOR` on the main (i.e. "synchronous") side. ON the async side the global context is read by the `DB_CONNECTION_MUTATOR` to set up authorization parameters and granular source information.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] sabiroid commented on issue #16209: [SIP-71] API to pass metadata over to Celery workers

Posted by GitBox <gi...@apache.org>.
sabiroid commented on issue #16209:
URL: https://github.com/apache/superset/issues/16209#issuecomment-902121572


   @mistercrunch, when you say "make an inventory of all the places we read global context", do you mean to pass references to those places over to `CELERY_FLASK_METADATA_EXTRACTOR` and `CELERY_FLASK_METADATA_INITIALIZER`? That sounds looks like a really good idea, so the user setting Superset up won't need to import `flask` context-global variables themselves in their code.
   
   By the way, just to clarify, on the async workers we don't read the global context by `CELERY_FLASK_METADATA_INITIALIZER`, we write into it. But we read it by `CELERY_FLASK_METADATA_EXTRACTOR` on the main (i.e. "synchronous") side. ON the async side the global context is read by the `DB_CONNECTION_MUTATOR` to set up authorization parameters and granular source information.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] mistercrunch edited a comment on issue #16209: [SIP-71] API to pass metadata over to Celery workers

Posted by GitBox <gi...@apache.org>.
mistercrunch edited a comment on issue #16209:
URL: https://github.com/apache/superset/issues/16209#issuecomment-901606773


   > Among them are user-impersonating authentication tokens and granular source information (e.g. dashboard or chart IDs). 
   
   Should we make an inventory of all the places we read global context on the async workers? Did you alter any of that code? Is there some authentication-specific logic on your side of things?
   
   @craig-rueda @robdiciuccio thoughts?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] mistercrunch commented on issue #16209: [SIP-71] API to pass metadata over to Celery workers

Posted by GitBox <gi...@apache.org>.
mistercrunch commented on issue #16209:
URL: https://github.com/apache/superset/issues/16209#issuecomment-901606773


   > Among them are user-impersonating authentication tokens and granular source information (e.g. dashboard or chart IDs). 
   Should we make an inventory of all the places we read global context on the async workers? Did you alter any of that code? Is there some authentication-specific logic on your side of things?
   
   @craig-rueda @robdiciuccio thoughts?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org