[GitHub] [airflow] potiuk commented on pull request #32876: enable db access for dag run listeners

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk commented on PR #32876:
URL: https://github.com/apache/airflow/pull/32876#issuecomment-1682774959

   I agree, this is a very bad idea to allow db access here. There are multiple reasons (HA locks is the most important one) why "user provided code" should not mess around the sessions and DB while  dag runs are processed by the scheduler. We heavily rely on those db operations to be performant, have very strictly demarcated session boundaries. Adding a DB code run by user in some crucial moments, might cause a number of subtle problems that might be extremely difficult to diagnose. Basicaly by opening this option you are shooting yourself in the foot (and actually more likely you will be passing the problems on maintainers when some unintended side-effects kick in).
   
   Alos listener by definition should avoid doing any "heavy" work. It just listens, by implementing some database/heavy processing in it, you are likely impacting scheduler's performance a lot. Don't do it.
   
   IMHO trying to run DB operations and reading Xcom here is doubly wrong:
   
   * You should use Airflow REST API to access XCom information
   *  You should not run any heavy workload in your listeners. Your listerners should at most trigger (for example) a lambda  function in AWS to perform such summary.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org