You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Deebol (via GitHub)" <gi...@apache.org> on 2026/02/03 14:20:04 UTC

[I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Deebol opened a new issue, #61396:
URL: https://github.com/apache/airflow/issues/61396

   ### Apache Airflow version
   
   3.1.6
   
   ### If "Other Airflow 3 version" selected, which one?
   
   3.1.5
   
   ### What happened?
   
   Hi.
   
   In Airflow 3.1.6 (and 3.1.5, issue doesn't exist on 3.1.3), GitDagBundle with LocalExecutor performs a full git clone into a new directory (new Inode) and deletes the old one almost every time a task starts.
   
   This "Inode flipping" causes running tasks (e.g.: dbt via Cosmos) to lose their file descriptors to the DAG folder, resulting in FileNotFoundError or Directory not found errors.
   
   I executed such script inside airflow scheduler pod to detect ongoing git operations:
   ```
   python - <<'PY'
   import os, time, subprocess
   from datetime import datetime, timezone
   
   TARGET = "/tmp/airflow/dag_bundles/dh-pipeline-dags/versions/<provide sha>"
   INTERVAL = 0.05
   
   def now():
     return datetime.now(timezone.utc).strftime("%H:%M:%S.%f")[:-3]
   
   seen_pids = set()
   
   print(f"{now()} START monitoring GIT processes...")
   
   while True:
     try:
       pids = [pid for pid in os.listdir('/proc') if pid.isdigit()]
       for pid in pids:
         if pid not in seen_pids:
           try:
             with open(f"/proc/{pid}/cmdline", "rb") as f:
               cmd = f.read().replace(b"\x00", b" ").decode("utf-8", "ignore").strip()
   
             if "git" in cmd.lower():
               with open(f"/proc/{pid}/stat", "rb") as f:
                 stat_parts = f.read().split()
                 ppid = stat_parts[3].decode()
   
               inode_info = "N/A"
               if os.path.exists(TARGET):
                 try:
                   inode_info = os.stat(TARGET).st_ino
                 except: pass
   
               print(f"{now()} GIT DETECTED! PID={pid} PPID={ppid} INODE_BASE={inode_info}")
               print(f"   CMD: {cmd[:150]}")
   
               seen_pids.add(pid)
           except (FileNotFoundError, ProcessLookupError):
             continue
     except Exception as e:
       print(f"Error: {e}")
   
     if len(seen_pids) > 1000:
       seen_pids.clear()
   
     time.sleep(INTERVAL)
   PY
   
   ```
   And this is what I got on 3.1.6:
   ```
   ...
   13:33:06.182 GIT DETECTED! PID=16645 PPID=16640 INODE_BASE=139948383
      CMD: /usr/lib/git-core/git rev-list --objects --stdin --not --all --quiet --alternate-refs
   13:33:06.337 GIT DETECTED! PID=16648 PPID=16637 INODE_BASE=406416615
      CMD: git clone -v -- /tmp/airflow/dag_bundles/dh-pipeline-dags/bare /tmp/airflow/dag_bundles/dh-pipeline-dags/versions/1108133dd6b2afb806d3531e9a97f8b15994
   13:33:06.337 GIT DETECTED! PID=16649 PPID=16648 INODE_BASE=406416615
      CMD: /bin/sh -c git-upload-pack '/tmp/airflow/dag_bundles/dh-pipeline-dags/bare' git-upload-pack '/tmp/airflow/dag_bundles/dh-pipeline-dags/bare'
   13:33:06.337 GIT DETECTED! PID=16650 PPID=16649 INODE_BASE=406416615
      CMD: git-upload-pack /tmp/airflow/dag_bundles/dh-pipeline-dags/bare
   13:33:06.491 GIT DETECTED! PID=16652 PPID=16637 INODE_BASE=406416615
      CMD: git cat-file --batch-check
   13:33:06.491 GIT DETECTED! PID=16653 PPID=16637 INODE_BASE=406416615
      CMD: git reset --hard HEAD --
   13:33:09.825 GIT DETECTED! PID=16658 PPID=16655 INODE_BASE=406416615
      CMD: git fetch -v -- origin +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/*
   13:33:10.339 GIT DETECTED! PID=16666 PPID=16655 INODE_BASE=473240196
      CMD: git clone -v -- /tmp/airflow/dag_bundles/dh-pipeline-dags/bare /tmp/airflow/dag_bundles/dh-pipeline-dags/versions/1108133dd6b2afb806d3531e9a97f8b15994
   13:33:10.339 GIT DETECTED! PID=16667 PPID=16666 INODE_BASE=473240196
      CMD: /bin/sh -c git-upload-pack '/tmp/airflow/dag_bundles/dh-pipeline-dags/bare' git-upload-pack '/tmp/airflow/dag_bundles/dh-pipeline-dags/bare'
   13:33:10.339 GIT DETECTED! PID=16668 PPID=16667 INODE_BASE=473240196
      CMD: git-upload-pack /tmp/airflow/dag_bundles/dh-pipeline-dags/bare
   13:33:10.441 GIT DETECTED! PID=16669 PPID=16655 INODE_BASE=473240196
      CMD: git checkout qa/deployed
   ...
   ```
   
   git clone is happening and INODE_BASE is changing.
   
   On 3.1.3 I got:
   ```
   ...
   13:16:45.591 GIT DETECTED! PID=354362 PPID=354358 INODE_BASE=14307405
      CMD: git checkout dev/deployed
   13:16:46.580 GIT DETECTED! PID=354373 PPID=354366 INODE_BASE=14307405
      CMD: git cat-file --batch-check
   13:16:49.630 GIT DETECTED! PID=354397 PPID=354393 INODE_BASE=14307405
      CMD: git checkout dev/deployed
   13:16:49.990 GIT DETECTED! PID=354405 PPID=354400 INODE_BASE=14307405
      CMD: git cat-file --batch-check
   13:16:49.990 GIT DETECTED! PID=354406 PPID=354400 INODE_BASE=14307405
      CMD: git reset --hard HEAD --
   13:17:01.411 GIT DETECTED! PID=354439 PPID=354434 INODE_BASE=14307405
      CMD: git cat-file --batch-check
   13:17:01.411 GIT DETECTED! PID=354440 PPID=354434 INODE_BASE=14307405
      CMD: git reset --hard HEAD --
   13:17:04.472 GIT DETECTED! PID=354443 PPID=354441 INODE_BASE=14307405
      CMD: git version
   ...
   ```
   
   No git-clone's and INODE_BASE is not changing.
   
   ### What you think should happen instead?
   
   _No response_
   
   ### How to reproduce
   
   1. Setup Airflow 3.1.6 with LocalExecutor.
   2. Configure a GitDagBundle
   3. Run multiple tasks that reads files from the bundle directory (like a DbtSelectOperator from Cosmos).
   
   ### Operating System
   
   Debian GNU/Linux 12 (bookworm)
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow==3.1.6
   apache-airflow-core==3.1.6
   apache-airflow-providers-airbyte==5.3.1
   apache-airflow-providers-amazon==9.19.0
   apache-airflow-providers-common-compat==1.11.0
   apache-airflow-providers-common-io==1.7.0
   apache-airflow-providers-common-sql==1.30.2
   apache-airflow-providers-datadog==3.10.1
   apache-airflow-providers-http==5.6.2
   apache-airflow-providers-microsoft-azure==12.10.1
   apache-airflow-providers-mongo==5.3.1
   apache-airflow-providers-smtp==2.4.1
   apache-airflow-providers-standard==1.10.2
   apache-airflow-task-sdk==1.1.6
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   ```
       - name: AIRFLOW__DAG_PROCESSOR__DEFAULT_BUNDLE_NAME
         value: "dh-pipeline-dags"
       - name: AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST
         value: >
           [
             {
               "name": "dh-pipeline-dags",
               "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
               "kwargs": {
                 "tracking_ref": "qa/deployed",
                 "subdir": "dags",
                 "git_conn_id": "git_xc_dh_pipeline",
                 "refresh_interval": 300
               }
             }
           ]
   ```
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3895977798

   @Deebol can you test with this fix: https://github.com/apache/airflow/pull/61847


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3843909001

   cc @jedcunningham not sure if this is a GitDagBundle or airflow core issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3849405174

   Thanks @Deebol! I've reopened this, as that feature should not cause a full reclone.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3895980872

   > > After investigating this, I think we should remove the prune_dotgit_folder arg completely and users should depend on stale bundle cleanup for cleaning of the bundles. The gain we got from pruning dotgit for the version directly is washed off by constantly cloning of the version directory. cc [@ashb](https://github.com/ashb) [@jedcunningham](https://github.com/jedcunningham)
   > 
   > [@ephraimbuddy](https://github.com/ephraimbuddy) this saved 100s of Mbs from some users. So i dont think a straight revert is the right long term option. Though my removing the .git dir we loose the ability to tell if the checkout is clean or not (the re-clone issue we could fix in other ways)
   > 
   > Didn't we have some other ideas at the time? Worktrees?
   
   Managed to get it working with this fix: https://github.com/apache/airflow/pull/61847


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "Deebol (via GitHub)" <gi...@apache.org>.
Deebol commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3848887240

   `"prune_dotgit_folder": false` resolved the issue. Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3876818676

   After investigating this, I think we should remove the prune_dotgit_folder arg completely and users should depend on stale bundle cleanup for cleaning of the bundles. The gain we got from pruning dotgit for the version directly is washed off by constantly cloning of the version directory. cc @ashb 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3841618492

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3876822141

   Related: https://github.com/apache/airflow/issues/61716


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "Deebol (via GitHub)" <gi...@apache.org>.
Deebol commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3847780087

   On 3.1.6: `apache-airflow-providers-git==0.2.0`. 
   On 3.1.3: `apache-airflow-providers-git==0.0.9`.
   
   I explicitly provided `apache-airflow-providers-git==0.0.9` into my image based on 3.1.6 and it looks better. No more INODE flips.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "Deebol (via GitHub)" <gi...@apache.org>.
Deebol closed issue #61396: Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError)
URL: https://github.com/apache/airflow/issues/61396


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "anavrotski (via GitHub)" <gi...@apache.org>.
anavrotski commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3876872040

   Not sure if we are talking about the same executor, but in our deployment (3.1.6) we are using Celery + k8s executors, and these errors appear very often (FileNotFound, most often it is a json file from our DAGs, not python).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3845096365

   Not sure off the top of my head. @Deebol could you try using the git provider version you were using on 3.1.3 on 3.1.6 so we can narrow down which it is?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3847917671

   Gemini seems to think its #57069, and its rational seems reasonable:
   
   > Most likely culprit:
         • 16c544079d Remove '.git' folder from versions in GitDagBundle (#57069)
             •Reasoning: This commit introduces logic to delete the .git directory from the versioned checkout to save space (prune_dotgit_folder=True by default). The issue (#61396) describes "Inode flipping" where the directory is re-cloned every time. The GitDagBundle initialization logic checks if the repository exists and tries to open it using git.Repo(path). If the .git folder is missing, this raises an InvalidGitRepositoryError, which triggers a retry mechanism that deletes the existing directory and re-clones it. This cycle repeats on every initialization, causing the reported behavior.
   
   Can you add `prune_dotgit_folder=False` into your bundle config to see if that solves it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "Deebol (via GitHub)" <gi...@apache.org>.
Deebol commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3847303049

   On 3.1.6: `apache-airflow-providers-git==0.2.0`. 
   On 3.1.3: `apache-airflow-providers-git==0.0.9`.
   
   I explicitly provided `apache-airflow-providers-git==0.0.9` into my image based on 3.1.6 and the issue is still there. Although the error-log message is a bit different, it refers to `AirflowTaskTimeout` not to `FileNotFoundError`:
   ```
   Failed to import: /tmp/airflow/dag_bundles/dh-pipeline-dags/versions/2add2bcfa01fe80829b244a53f680ad1a58507d9/dags/pipelines/dag1.py
   AirflowTaskTimeout: DagBag import timeout for /tmp/airflow/dag_bundles/dh-pipeline-dags/versions/2add2bcfa01fe80829b244a53f680ad1a58507d9/dags/pipelines/dag1.py after 30.0s.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "Arunodoy18 (via GitHub)" <gi...@apache.org>.
Arunodoy18 commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3851877486

   Hi! I’d like to work on this issue.
   
   From what I understand, GitDagBundle + LocalExecutor is triggering a full git clone per task, causing inode changes and resulting in FileNotFoundError during execution.
   I’ll start by reproducing the issue locally and tracing the bundle clone lifecycle, then propose a fix that avoids unnecessary re-cloning while preserving correctness and isolation.
   I’ll include tests and ensure the fix passes all CI checks.
   Please let me know if there are any preferred implementation directions. Thanks!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on issue #61396:
URL: https://github.com/apache/airflow/issues/61396#issuecomment-3880435348

   > After investigating this, I think we should remove the prune_dotgit_folder arg completely and users should depend on stale bundle cleanup for cleaning of the bundles. The gain we got from pruning dotgit for the version directly is washed off by constantly cloning of the version directory. cc @ashb @jedcunningham 
   
   
   @ephraimbuddy this saved 100s of Mbs from some users. So i dont think a straight revert is the right long term option. Though my removing the .git dir we loose the ability to tell if the checkout is clean or not (the re-clone issue we could fix in other ways)
   
   Didn't we have some other ideas at the time? Worktrees?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError) [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy closed issue #61396: Constant Inode flips on LocalExecutor cause task failures (FileNotFoundError)
URL: https://github.com/apache/airflow/issues/61396


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org