[jira] [Created] (AIRFLOW-4248) FileExistsError when creating log dir in file_processor_handler.py

["Kevin Pullin (JIRA)" <ji...@apache.org>]

Kevin Pullin created AIRFLOW-4248:
-------------------------------------

             Summary: FileExistsError when creating log dir in file_processor_handler.py
                 Key: AIRFLOW-4248
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4248
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: Kevin Pullin
            Assignee: Kevin Pullin


Similar to AIRFLOW-2065, the `_init_file` method in `file_processor_handler` does not handle the race condition of multiple processes creating the same directory on a shared logging filesystem.

If the error occurs in the scheduler process, tasks do not execute (at least when using the Kubernetes executor). However the process remains alive and requires a restart to resume running tasks.

 

Example log message:

 
{noformat}
[2019-04-03 23:59:55,091] {{sqlalchemy.py:79}} WARNING - DB connection invalidated. Reconnecting...
Process DagFileProcessor41527-Process:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 374, in helper
    set_context(log, file_path)
  File "/usr/local/lib/python3.6/site-packages/airflow/utils/log/logging_mixin.py", line 170, in set_context
    handler.set_context(value)
  File "/usr/local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py", line 66, in set_context
    local_loc = self._init_file(filename)
  File "/usr/local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py", line 141, in _init_file
    os.makedirs(directory)
  File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/usr/local/airflow/logs/scheduler/2019-04-04'{noformat}
 

The path `/usr/local/airflow/logs` is on a shared NFS volume.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)