[jira] [Updated] (AIRFLOW-2215) celery task launches subprocess without environment vars

["John Arnold (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

John Arnold updated AIRFLOW-2215:
---------------------------------
    Description: 
The celery task that is ran by CeleryExecutor does not preserve the environment variables when launching task instances in a subprocess.  specifically, AIRFLOW_HOME is lost.  In my environment, this results in the command failing.  Typically the command is something like 'airflow run <dag> <operator> <timestamp> -sd < dag path>'



@app.task
def execute_command(command):
 log = LoggingMixin().log
 log.info("Executing command in Celery: %s", command)
 env = os.environ.copy()
 try:
 subprocess.check_call(command, shell=True, close_fds=True, env=env)
 except subprocess.CalledProcessError as e:
 log.error(e)
 raise AirflowException('Celery command failed')

 

I believe the environment vars should be preserved in the subtask.  Also, the logging kind of sucks.

> celery task launches subprocess without environment vars
> --------------------------------------------------------
>
>                 Key: AIRFLOW-2215
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2215
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler, worker
>    Affects Versions: 1.9.0
>            Reporter: John Arnold
>            Priority: Major
>
> The celery task that is ran by CeleryExecutor does not preserve the environment variables when launching task instances in a subprocess.  specifically, AIRFLOW_HOME is lost.  In my environment, this results in the command failing.  Typically the command is something like 'airflow run <dag> <operator> <timestamp> -sd < dag path>'
> @app.task
> def execute_command(command):
>  log = LoggingMixin().log
>  log.info("Executing command in Celery: %s", command)
>  env = os.environ.copy()
>  try:
>  subprocess.check_call(command, shell=True, close_fds=True, env=env)
>  except subprocess.CalledProcessError as e:
>  log.error(e)
>  raise AirflowException('Celery command failed')
>  
> I believe the environment vars should be preserved in the subtask.  Also, the logging kind of sucks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)