Re: [I] Implement new feature of refreshing token for long running tasks [airflow]

["jnstockley (via GitHub)" <gi...@apache.org>]

jnstockley commented on issue #53713:
URL: https://github.com/apache/airflow/issues/53713#issuecomment-3155935088

   @ashb Sorry for the delay, been on vacation the last week. I was able to reproduce the issue locally, and I think it's related to using AWS Secrets as the secrets backend. What I did was as follows
   1. Start one of our longer running jobs, pulls millions of rows from one DB to store into another DB
   2. Start another DAG, similar idea, but not nearly as many rows
   3. Noticed that the second DAG was queued, until the first DAG switched from the first task to the second task, and then the first DAG was also queued on the second task, until I manually marked the second DAG as failed. I did this because my AWS credentials expired on my local machine, i.e, needed to re-login.
   4. As soon as I marked the second DAG as failed, the first one failed with the follow logs
   
   In case it helps, the first DAG is called `SPOT-Extract-And-Load` and the second DAG is called `MacMillan-SFTP`
   
   [airflow.log](https://github.com/user-attachments/files/21604141/airflow.log)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org