[GitHub] [airflow] zhang699 commented on issue #13805: Could not get scheduler_job_id

[GitBox <gi...@apache.org>]

zhang699 commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-781793430


   I also encounter this problem, but curious about another questions, why load same DAG three times before the exception happen, and the last two of loaded DAG path seems incorrect.  
    
   ```
   [2021-02-19 03:35:50,204] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
   [2021-02-19 03:35:50,205] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..data/code-dag.py
   [2021-02-19 03:35:50,222] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
   [2021-02-19 03:35:50,223] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..2021_02_19_03_05_19.306169647/code-dag.py
   [2021-02-19 03:35:50,239] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
   ```
    
   ```
   airflow@webserver-6b975954d-pf6h8:/opt/airflow$ airflow dags backfill etl_dag1 -s 2020-01-20 -e 2020-01-21
   [2021-02-19 03:35:48,536] {settings.py:210} DEBUG - Setting up DB connection pool (PID 214)
   [2021-02-19 03:35:48,537] {settings.py:281} DEBUG - settings.prepare_engine_args(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=214
   [2021-02-19 03:35:48,636] {cli_action_loggers.py:40} DEBUG - Adding <function default_action_log at 0x7f7b77054320> to pre execution callback
   [2021-02-19 03:35:50,016] {cli_action_loggers.py:66} DEBUG - Calling callbacks: [<function default_action_log at 0x7f7b77054320>]
   /home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py:62 PendingDeprecationWarning: --ignore-first-depends-on-past is deprecated as the value is always set to True
   [2021-02-19 03:35:50,050] {dagbag.py:448} INFO - Filling up the DagBag from /home/airflow/dags
   [2021-02-19 03:35:50,051] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/code-dag.py
   /home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:26 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1Volume`.
   /home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:27 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1VolumeMount`.
   [2021-02-19 03:35:50,204] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
   [2021-02-19 03:35:50,205] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..data/code-dag.py
   [2021-02-19 03:35:50,222] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
   [2021-02-19 03:35:50,223] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..2021_02_19_03_05_19.306169647/code-dag.py
   [2021-02-19 03:35:50,239] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
   [2021-02-19 03:35:50,240] {executor_loader.py:82} DEBUG - Loading core executor: KubernetesExecutor
   [2021-02-19 03:35:50,398] {kubernetes_executor.py:473} INFO - Start Kubernetes executor
   [2021-02-19 03:35:50,415] {cli_action_loggers.py:84} DEBUG - Calling callbacks: []
   Traceback (most recent call last):
     File "/home/airflow/.local/bin/airflow", line 8, in <module>
       sys.exit(main())
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
       args.func(args)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 89, in wrapper
       return f(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py", line 116, in dag_backfill
       run_backwards=args.run_backwards,
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dag.py", line 1706, in run
       job.run()
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 237, in run
       self._execute()
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper
       return func(*args, session=session, **kwargs)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/backfill_job.py", line 788, in _execute
       executor.start()
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 475, in start
       raise AirflowException("Could not get scheduler_job_id")
   airflow.exceptions.AirflowException: Could not get scheduler_job_id
   [2021-02-19 03:35:50,429] {settings.py:292} DEBUG - Disposing DB connection pool (PID 214)
   ```
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org