[GitHub] [airflow] gillbuchanan commented on issue #13680: "dag_id could not be found" when running airflow on KubernetesExecutor

[GitBox <gi...@apache.org>]

gillbuchanan commented on issue #13680:
URL: https://github.com/apache/airflow/issues/13680#issuecomment-763847427


   I'm having this issue too. After upgrading to 2.0.0, while running with `KubernetesExecutor` in an Azure k8s cluster, the worker pods have been failing with the same error message
   
   ```[2021-01-20 14:42:35,290] {dagbag.py:440} INFO - Filling up the DagBag from /opt/airflow/dags/test_dag.py
   Traceback (most recent call last):
     File "/home/airflow/.local/bin/airflow", line 8, in <module>
       sys.exit(main())
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__main__.py", line 40, in main
       args.func(args)
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 48, in command
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 89, in wrapper
       return f(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py", line 216, in task_run
       dag = get_dag(args.subdir, args.dag_id)
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 189, in get_dag
       'parse.'.format(dag_id)
   airflow.exceptions.AirflowException: dag_id could not be found: test. Either the dag did not exist or it failed to parse.
   ```
   
   I access my DAGs via an Azure file store mounted as a volume using a pvc, and have found that the worker pods are not being created with the same mounted volume. It seems that the previously available `airflow.cfg` values
   
   ```
   [kubernetes]
   dags_volume_claim = 
   dags_volume_subpath = 
   ```
   
   no longer exist in version 2.0.0, and the worker pods therefore cannot be created with the the proper volume mounts, leading to the above error.
   
   Any way around this?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org