You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Ganesh Sivalingam (JIRA)" <ji...@apache.org> on 2017/12/13 13:35:00 UTC
[jira] [Created] (AIRFLOW-1918) DockerOperator forcing Jinja
templating on "command" when not required
Ganesh Sivalingam created AIRFLOW-1918:
------------------------------------------
Summary: DockerOperator forcing Jinja templating on "command" when not required
Key: AIRFLOW-1918
URL: https://issues.apache.org/jira/browse/AIRFLOW-1918
Project: Apache Airflow
Issue Type: Bug
Components: docker
Affects Versions: Airflow 1.8
Reporter: Ganesh Sivalingam
When creating a docker operator, and providing the command parameter like so:
{code: python}
create_cluster = DockerOperator(
image="my/image",
command="bash /home/ec2-user/create-cluster.sh",
cpus=1
{code}
I get the following error:
{code:java}
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask: --------------------------------------------------------------------------------
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask: Starting attempt 1 of 1
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask: --------------------------------------------------------------------------------
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask:
[2017-12-13 13:21:22,639] {base_task_runner.py:95} INFO - Subtask: [2017-12-13 13:21:22,639] {models.py:1342} INFO - Executing <Task(DockerOperator): create_flintrock_cluster> on 2017-12-13 13:21:20.147853
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: [2017-12-13 13:21:22,652] {models.py:1417} ERROR - bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent call last):
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1359, in run
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: self.render_templates()
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1579, in render_templates
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: rendered_content = rt(attr, content, jinja_context)
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2245, in render_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: return jinja_env.get_template(content).render(**context)
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 812, in get_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: return self._load_template(name, self.make_globals(globals))
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 774, in _load_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: cache_key = self.loader.get_source(self, name)[1]
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in get_source
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask: raise TemplateNotFound(template)
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask: TemplateNotFound: bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask: [2017-12-13 13:21:22,653] {models.py:1441} INFO - Marking task as FAILED.
[2017-12-13 13:21:22,667] {base_task_runner.py:95} INFO - Subtask: [2017-12-13 13:21:22,667] {models.py:1462} ERROR - bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent call last):
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/bin/airflow", line 28, in <module>
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: args.func(args)
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 422, in run
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: pool=args.pool,
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in wrapper
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: result = func(*args, **kwargs)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1359, in run
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: self.render_templates()
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1579, in render_templates
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: rendered_content = rt(attr, content, jinja_context)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2245, in render_template
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: return jinja_env.get_template(content).render(**context)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 812, in get_template
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: return self._load_template(name, self.make_globals(globals))
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 774, in _load_template
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: cache_key = self.loader.get_source(self, name)[1]
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in get_source
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: raise TemplateNotFound(template)
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: jinja2.exceptions.TemplateNotFound: bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:26,928] {jobs.py:2083} INFO - Task exited with return code 1
{code}
Within the DockerOperator code, if I comment out line 79 (`template_fields = ('command',)`) then the task completes successfully.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)