You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/25 23:28:20 UTC

[GitHub] [airflow] o-nikolas opened a new issue, #27930: AIP-51 - Misc. Compatibility Checks

o-nikolas opened a new issue, #27930:
URL: https://github.com/apache/airflow/issues/27930

   ### Overview
   
   Coupling around other compatibility checks, similar to coupling 1 and 2, but which don't fall into either category.
   
   ### Examples
   - **3a**) Some Executors support pickling, some don’t, there is logic in both backfill and scheduler jobs that checks before submitting tasks to the Executor queue. Applies to DaskExecutor, LocalExecutor and SequentialExecutor as of now: https://github.com/apache/airflow/blob/26f94c5370587f73ebd935cecf208c6a36bdf9b6/airflow/jobs/backfill_job.py#L825-L829 https://github.com/apache/airflow/blob/26f94c5370587f73ebd935cecf208c6a36bdf9b6/airflow/jobs/scheduler_job.py#L703-L704
   - **3b**) Sentry has a celery integration, if the Executor is Celery, the Sentry integration is imported and configured: https://github.com/apache/airflow/blob/27e2101f6ee5567b2843cbccf1dca0b0e7c96186/airflow/sentry.py#L87
     - Sentry at the moment only supports Celery, so this particular coupling is low priority, but should still be fixed.
   
   ### Proposal
   A static method or attribute on the Executor class which can be checked by core code.
   
   There is a precedent already set with the `supports_ad_hoc_ti_run` attribute, see:
   https://github.com/apache/airflow/blob/fb741fd87254e235f99d7d67e558dafad601f253/airflow/executors/kubernetes_executor.py#L435 https://github.com/apache/airflow/blob/26f94c5370587f73ebd935cecf208c6a36bdf9b6/airflow/www/views.py#L1735-L1737


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] o-nikolas closed issue #27930: AIP-51 - Misc. Compatibility Checks

Posted by GitBox <gi...@apache.org>.
o-nikolas closed issue #27930: AIP-51 - Misc. Compatibility Checks
URL: https://github.com/apache/airflow/issues/27930


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] o-nikolas commented on issue #27930: AIP-51 - Misc. Compatibility Checks

Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #27930:
URL: https://github.com/apache/airflow/issues/27930#issuecomment-1352054608

   > The ticket is scoped out really well and I was planning on using pierrejeambrun's PR as inspiration for this one since it seems very similar.
   
   Glad to hear that! I tried to put a good amount of context in the issues to make them easy to start :smile:


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] rkarish commented on issue #27930: AIP-51 - Misc. Compatibility Checks

Posted by GitBox <gi...@apache.org>.
rkarish commented on issue #27930:
URL: https://github.com/apache/airflow/issues/27930#issuecomment-1353638034

   Hey Niko! I've opened a PR for this task that covers 3a and 3b. Looking forward to getting some feedback on this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] rkarish commented on issue #27930: AIP-51 - Misc. Compatibility Checks

Posted by GitBox <gi...@apache.org>.
rkarish commented on issue #27930:
URL: https://github.com/apache/airflow/issues/27930#issuecomment-1351867492

   Excited to get started on this tonight. The ticket is scoped out really well and I was planning on using pierrejeambrun's PR as inspiration for this one since it seems very similar. I will reach out if I need any guidance. Thanks Niko!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] o-nikolas commented on issue #27930: AIP-51 - Misc. Compatibility Checks

Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #27930:
URL: https://github.com/apache/airflow/issues/27930#issuecomment-1351861080

   Thanks for taking this one @rkarish! Ping me if you need any guidance :smile: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] rkarish commented on issue #27930: AIP-51 - Misc. Compatibility Checks

Posted by GitBox <gi...@apache.org>.
rkarish commented on issue #27930:
URL: https://github.com/apache/airflow/issues/27930#issuecomment-1350341087

   Hey Niko, I'll take this ticket if it's still available.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #27930: AIP-51 - Misc. Compatibility Checks

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27930:
URL: https://github.com/apache/airflow/issues/27930#issuecomment-1350657423

   Assigned you 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org