[GitHub] [airflow] potiuk commented on issue #21375: Airflow launched two identical tasks in parallel with K8S executor

[GitBox <gi...@apache.org>]

potiuk commented on issue #21375:
URL: https://github.com/apache/airflow/issues/21375#issuecomment-1031259843


   Seems like in some circumstances, Kubernetes Executor might run the same task as two separate PODs running in parallel:
   
   I'm sure it's not the same job, this is the log where it's succeeded to delete the file in the ftp server:
   ```
   [2022-01-27, 03:09:34 UTC] {sftp_to_gcs.py:193} INFO - Executing delete of input_20220127.csv
   [2022-01-27, 03:09:34 UTC] {sftp.py:158} DEBUG - [chan 0] remove(b'input_20220127.csv')
   [2022-01-27, 03:09:34 UTC] {sftp.py:158} INFO - [chan 0] sftp session closed.
   [2022-01-27, 03:09:34 UTC] {channel.py:1212} DEBUG - [chan 0] EOF sent (0)
   ```
   
   and this is the log from the second job:
   ```
   [2022-01-27, 03:09:34 UTC] {sftp_to_gcs.py:193} INFO - Executing delete of input_20220127.csv
   [2022-01-27, 03:09:34 UTC] {sftp.py:158} DEBUG - [chan 0] remove(b'input_20220127.csv')
   [2022-01-27, 03:09:34 UTC] {taskinstance.py:1703} ERROR - Task failed with exception
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
       self._execute_task_with_callbacks(context)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
       result = self._execute_task(context, self.task)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
       result = execute_callable(context=context)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/transfers/sftp_to_gcs.py", line 144, in execute
       self._copy_single_object(
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/transfers/sftp_to_gcs.py", line 194, in _copy_single_object
       sftp_hook.delete_file(source_path)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/sftp/hooks/sftp.py", line 251, in delete_file
       conn.remove(path)
     File "/home/airflow/.local/lib/python3.9/site-packages/pysftp/__init__.py", line 728, in remove
       self._sftp.remove(remotefile)
     File "/home/airflow/.local/lib/python3.9/site-packages/paramiko/sftp_client.py", line 398, in remove
       self._request(CMD_REMOVE, path)
     File "/home/airflow/.local/lib/python3.9/site-packages/paramiko/sftp_client.py", line 822, in _request
       return self._read_response(num)
     File "/home/airflow/.local/lib/python3.9/site-packages/paramiko/sftp_client.py", line 874, in _read_response
       self._convert_status(msg)
     File "/home/airflow/.local/lib/python3.9/site-packages/paramiko/sftp_client.py", line 903, in _convert_status
       raise IOError(errno.ENOENT, text)
   FileNotFoundError: [Errno 2] No such file
   ```
   
   And those lines came from the same log.
   We run this task everyday at 3AM and it's failed time to time (I would say 3 to 4 times in one month)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org