[GitHub] [airflow] potiuk commented on issue #10790: Copy of [AIRFLOW-5071] JIRA: Thousands of Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?

[GitBox <gi...@apache.org>]

potiuk commented on issue #10790:
URL: https://github.com/apache/airflow/issues/10790#issuecomment-1254633385

   > Hello @karthik-raparthi, we also did experience similar issue with EFS. EFS is definitely not suited for a big airflow deployment, and we stopped having most of its issues when we moved to FSx File System. I therefore encourage you to move to this better solution :)
   > 
   > (we had EFS 100 Mo/s provisioned throughout and still experiencing this)
   
   Quite agree. There were multiple people reporting problems in huge airflow installation where EFS was used. I can also recommend (as usual) switching to Git Sync. I wrote an article about it https://medium.com/apache-airflow/shared-volumes-in-airflow-the-good-the-bad-and-the-ugly-22e9f681afca - especially when you are using Git to store your DAGs already, using shared volume is completely unnecessary and using Git Sync directly is far better solution.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org