You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dstandish (via GitHub)" <gi...@apache.org> on 2025/03/10 14:03:51 UTC
[I] Issue with bundle lock in dag bundle versioning [airflow]
dstandish opened a new issue, #47567:
URL: https://github.com/apache/airflow/issues/47567
### Body
Hey team :slightly_smiling_face:
Is the GitDagBundle working yet? And if so, what am I missing?
I set up a git_default connection with content read permissions for all my repos and this config:
[dag_processor]
dag_bundle_config_list=[{"name": "dags-folder", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {}}, {"name": "tjanif/dynamic-task-mapping-tutorial", "classpath": "airflow.dag_processing.bundles.git.GitDagBundle", "kwargs": {"subdir": "dags", "tracking_ref": "main", "refresh_interval": 30}}]
and my dag-processor crashes with:
dag-processor | FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/hm/tgdl9dqs2j18vhrsk7s6c1q80000gn/T/airflow/dag_bundles/_locks/tjanif/dynamic-task-mapping-tutorial.lock'
directly after starting it. (Full error in :thread: )
```
dag-processor | [2025-03-10T12:04:34.805+0100] {manager.py:246} INFO - Checking for new files in bundle dags-folder every 300 seconds
dag-processor | [2025-03-10T12:04:34.805+0100] {manager.py:246} INFO - Checking for new files in bundle tjanif/dynamic-task-mapping-tutorial every 30 seconds
dag-processor | [2025-03-10T12:04:34.806+0100] {manager.py:504} INFO - Refreshing bundle dags-folder
dag-processor | [2025-03-10T12:04:34.807+0100] {manager.py:554} INFO - Searching for files in dags-folder at /Users/tamara.fingerlin/airflow/dags_manning
dag-processor | [2025-03-10T12:04:34.815+0100] {manager.py:556} INFO - Found 6 files for bundle dags-folder
dag-processor | [2025-03-10T12:04:34.823+0100] {dag_processor_job_runner.py:63} ERROR - Exception when executing DagProcessorJob
dag-processor | Traceback (most recent call last):
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/jobs/dag_processor_job_runner.py", line 61, in _execute
dag-processor | self.processor.run()
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 252, in run
dag-processor | return self._run_parsing_loop()
dag-processor | ^^^^^^^^^^^^^^^^^^^^^^^^
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 323, in _run_parsing_loop
dag-processor | self._refresh_dag_bundles(known_files=known_files)
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 474, in _refresh_dag_bundles
dag-processor | bundle.initialize()
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/dag_processing/bundles/git.py", line 205, in initialize
dag-processor | self._initialize()
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/dag_processing/bundles/git.py", line 186, in _initialize
dag-processor | with self.lock():
dag-processor | ^^^^^^^^^^^
dag-processor | File "/Users/tamara.fingerlin/.pyenv/versions/3.12.8/lib/python3.12/contextlib.py", line 137, in __enter__
dag-processor | return next(self.gen)
dag-processor | ^^^^^^^^^^^^^^
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/dag_processing/bundles/base.py", line 338, in lock
dag-processor | with open(lock_file_path, "w") as lock_file:
dag-processor | ^^^^^^^^^^^^^^^^^^^^^^^^^
dag-processor | FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/hm/tgdl9dqs2j18vhrsk7s6c1q80000gn/T/airflow/dag_bundles/_locks/tjanif/dynamic-task-mapping-tutorial.lock'
dag-processor | Traceback (most recent call last):
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/bin/airflow", line 12, in <module>
dag-processor | sys.exit(main())
dag-processor | ^^^^^^
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/__main__.py", line 58, in main
dag-processor | args.func(args)
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/cli/cli_config.py", line 49, in command
dag-processor | return func(*args, **kwargs)
dag-processor | ^^^^^^^^^^^^^^^^^^^^^
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/utils/cli.py", line 112, in wrapper
dag-processor | return f(*args, **kwargs)
dag-processor | ^^^^^^^^^^^^^^^^^^
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
dag-processor | return func(*args, **kwargs)
dag-processor | ^^^^^^^^^^^^^^^^^^^^^
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/cli/commands/local_commands/dag_processor_command.py", line 54, in dag_processor
dag-processor | run_command_with_daemon_option(
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/cli/commands/local_commands/daemon_utils.py", line 86, in run_command_with_daemon_option
dag-processor | callback()
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/cli/commands/local_commands/dag_processor_command.py", line 57, in <lambda>
dag-processor | callback=lambda: run_job(job=job_runner.job, execute_callable=job_runner._execute),
dag-processor | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/utils/session.py", line 101, in wrapper
dag-processor | return func(*args, session=session, **kwargs)
dag-processor | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/jobs/job.py", line 342, in run_job
dag-processor | return execute_job(job, execute_callable=execute_callable)
dag-processor | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/jobs/job.py", line 371, in execute_job
dag-processor | ret = execute_callable()
dag-processor | ^^^^^^^^^^^^^^^^^^
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/jobs/dag_processor_job_runner.py", line 61, in _execute
dag-processor | self.processor.run()
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 252, in run
dag-processor | return self._run_parsing_loop()
dag-processor | ^^^^^^^^^^^^^^^^^^^^^^^^
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 323, in _run_parsing_loop
dag-processor | self._refresh_dag_bundles(known_files=known_files)
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 474, in _refresh_dag_bundles
dag-processor | bundle.initialize()
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/dag_processing/bundles/git.py", line 205, in initialize
dag-processor | self._initialize()
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/dag_processing/bundles/git.py", line 186, in _initialize
dag-processor | with self.lock():
dag-processor | ^^^^^^^^^^^
dag-processor | File "/Users/tamara.fingerlin/.pyenv/versions/3.12.8/lib/python3.12/contextlib.py", line 137, in __enter__
dag-processor | return next(self.gen)
dag-processor | ^^^^^^^^^^^^^^
dag-processor | File "/Users/tamara.fingerlin/0_PARA/Projects/Airflow3.0/a wheel/beta2/lib/python3.12/site-packages/airflow/dag_processing/bundles/base.py", line 338, in lock
dag-processor | with open(lock_file_path, "w") as lock_file:
dag-processor | ^^^^^^^^^^^^^^^^^^^^^^^^^
dag-processor | FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/hm/tgdl9dqs2j18vhrsk7s6c1q80000gn/T/airflow/dag_bundles/_locks/tjanif/dynamic-task-mapping-tutorial.lock'
```
@ephraimbuddy @jedcunningham
### Committer
- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issue with bundle lock in dag bundle versioning [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish closed issue #47567: Issue with bundle lock in dag bundle versioning
URL: https://github.com/apache/airflow/issues/47567
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issue with bundle lock in dag bundle versioning [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on issue #47567:
URL: https://github.com/apache/airflow/issues/47567#issuecomment-2716305545
resolved by https://github.com/apache/airflow/pull/47618
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issue with bundle lock in dag bundle versioning [airflow]
Posted by "TJaniF (via GitHub)" <gi...@apache.org>.
TJaniF commented on issue #47567:
URL: https://github.com/apache/airflow/issues/47567#issuecomment-2710730356
This is for beta2 standalone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issue with bundle lock in dag bundle versioning [airflow]
Posted by "TJaniF (via GitHub)" <gi...@apache.org>.
TJaniF commented on issue #47567:
URL: https://github.com/apache/airflow/issues/47567#issuecomment-2711476241
Some trial and error showed that the issue was the / in the bundle name. I had erroneously assumed the name was the repo. Apologies 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org