You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Sajid Sajid (Jira)" <ji...@apache.org> on 2022/02/25 19:47:00 UTC
[jira] [Assigned] (AIRFLOW-6124) tmp folder gets deleted, shell-init: error retrieving current directory
[ https://issues.apache.org/jira/browse/AIRFLOW-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sajid Sajid reassigned AIRFLOW-6124:
------------------------------------
Assignee: Sajid Sajid
> tmp folder gets deleted, shell-init: error retrieving current directory
> -----------------------------------------------------------------------
>
> Key: AIRFLOW-6124
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6124
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
> Affects Versions: 1.10.6
> Reporter: Ravi Kaushik
> Assignee: Sajid Sajid
> Priority: Critical
> Attachments: Error.png
>
>
> My dag has three SparkSubmitOperators and throws this error occasionally after executing one task. If I trigger the job manually by ignoring all dependencies, it works well. Most likely /tmp folder is getting deleted and it cannot find it again
>
> {{```[2019-11-29 12:58:54,548] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,547] \{base_hook.py:84} INFO - Using connection to: id: spark_default. Host: yarn, Port: None, Schema: None, Login: None, Password: None, extra: \{'deploy-mode': 'cluster', 'spark-binary': '/usr/bin/spark2-submit/'}
> [2019-11-29 12:58:54,548] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,548] \{spark_submit_hook.py:302} INFO - Spark-Submit cmd: ['spark-submit', '--master', 'yarn', '--conf', 'spark.yarn.maxAppAttempts=2', '--files', '/home/centos/airflow/dags/conf/aws_keys.pickle', '--num-executors', '4', '--executor-cores', '4', '--executor-memory', '20G', '--name', 'universal_cross_matrix_joins_AUS_27-11-2019', '--deploy-mode', 'cluster', '/home/centos/airflow/scripts/near-airflow-repo/scripts/universal_cross_matrix/join_ifa_ncid.py', '--input_date', '27-11-2019', '--country', 'AUS', '--tenant_id', '63ce7457']
> [2019-11-29 12:58:54,553] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,553] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
> [2019-11-29 12:58:54,555] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,555] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
> [2019-11-29 12:58:54,557] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,557] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
> [2019-11-29 12:58:54,560] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,560] \{spark_submit_hook.py:433} INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
> [2019-11-29 12:58:54,563] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,563] \{spark_submit_hook.py:433} INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
> [2019-11-29 12:58:54,565] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,564] \{spark_submit_hook.py:433} INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
> [2019-11-29 12:58:54,567] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,567] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
> [2019-11-29 12:58:54,569] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,569] \{spark_submit_hook.py:433} INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
> [2019-11-29 12:58:54,571] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,571] \{spark_submit_hook.py:433} INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
> [2019-11-29 12:58:54,574] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,574] \{spark_submit_hook.py:433} INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
> [2019-11-29 12:58:54,576] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,576] \{spark_submit_hook.py:433} INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory```}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)