You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jack (Jira)" <ji...@apache.org> on 2019/09/25 10:22:00 UTC
[jira] [Commented] (AIRFLOW-4248) FileExistsError when creating log
dir in file_processor_handler.py
[ https://issues.apache.org/jira/browse/AIRFLOW-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937600#comment-16937600 ]
jack commented on AIRFLOW-4248:
-------------------------------
[~ash] this was merged a while ago.
Can this be closed? (Maybe cherry picked to 1.10.6) ?
> FileExistsError when creating log dir in file_processor_handler.py
> ------------------------------------------------------------------
>
> Key: AIRFLOW-4248
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4248
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Kevin Pullin
> Assignee: Kevin Pullin
> Priority: Major
>
> Similar to AIRFLOW-2065, the `_init_file` method in `file_processor_handler` does not handle the race condition of multiple processes creating the same directory on a shared logging filesystem.
> If the error occurs in the scheduler process, tasks do not execute (at least when using the Kubernetes executor). However the process remains alive and requires a restart to resume running tasks.
>
> Example log message:
>
> {noformat}
> [2019-04-03 23:59:55,091] {{sqlalchemy.py:79}} WARNING - DB connection invalidated. Reconnecting...
> Process DagFileProcessor41527-Process:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
> self.run()
> File "/usr/local/lib/python3.6/multiprocessing/process.py", line 93, in run
> self._target(*self._args, **self._kwargs)
> File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 374, in helper
> set_context(log, file_path)
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/log/logging_mixin.py", line 170, in set_context
> handler.set_context(value)
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py", line 66, in set_context
> local_loc = self._init_file(filename)
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py", line 141, in _init_file
> os.makedirs(directory)
> File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
> mkdir(name, mode)
> FileExistsError: [Errno 17] File exists: '/usr/local/airflow/logs/scheduler/2019-04-04'{noformat}
>
> The path `/usr/local/airflow/logs` is on a shared NFS volume.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)