[jira] [Commented] (AIRFLOW-2215) celery task launches subprocess without environment vars

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400056#comment-16400056 ] 

ASF subversion and git services commented on AIRFLOW-2215:
----------------------------------------------------------

Commit a8174047b24ec4c3eb62e5651b39ca44dc8dd94b in incubator-airflow's branch refs/heads/master from [~johnarnold]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a817404 ]

[AIRFLOW-2215] Update celery task to preserve environment variables and improve logging on exception

Closes #3126 from johnarnold/celery_env


> celery task launches subprocess without environment vars
> --------------------------------------------------------
>
>                 Key: AIRFLOW-2215
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2215
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler, worker
>    Affects Versions: 1.9.0
>            Reporter: John Arnold
>            Priority: Major
>             Fix For: 2.0.0
>
>
> The celery task that is ran by CeleryExecutor does not preserve the environment variables when launching task instances in a subprocess.  specifically, AIRFLOW_HOME is lost.  In my environment, this results in the command failing.  Typically the command is something like 'airflow run <dag> <operator> <timestamp> -sd < dag path>'
> @app.task
> def execute_command(command):
>  log = LoggingMixin().log
>  log.info("Executing command in Celery: %s", command)
>  env = os.environ.copy()
>  try:
>  subprocess.check_call(command, shell=True, close_fds=True, env=env)
>  except subprocess.CalledProcessError as e:
>  log.error(e)
>  raise AirflowException('Celery command failed')
>  
> I believe the environment vars should be preserved in the subtask.  Also, the logging kind of sucks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)