[jira] [Assigned] (AIRFLOW-2875) Env variables should have percent signs escaped before writing to tmp config

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned AIRFLOW-2875:
-------------------------------------

    Assignee:     (was: Holden Karau's magical unicorn)

> Env variables should have percent signs escaped before writing to tmp config
> ----------------------------------------------------------------------------
>
>                 Key: AIRFLOW-2875
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2875
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: configuration
>         Environment: Ubuntu
> Airflow 1.10rc2
>            Reporter: William Horton
>            Priority: Major
>
> I encountered this when I was using an environment variable for `AIRFLOW__CELERY__BROKER_URL`. The airflow worker was able to run and communicate with the SQS queue, but when it received a task and began to run it, I encountered an error with this trace:
> {code:java}
> [2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask mirroring Traceback (most recent call last):
> [2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask mirroring File "/opt/airflow/venv/bin/airflow", line 32, in <module>
> [2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask mirroring args.func(args)
> [2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask mirroring File "/opt/airflow/venv/local/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
> [2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask mirroring return f(*args, **kwargs)
> [2018-08-08 15:19:24,402] {base_task_runner.py:108} INFO - Job 13898: Subtask mirroring File "/opt/airflow/venv/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 460, in run
> [2018-08-08 15:19:24,403] {base_task_runner.py:108} INFO - Job 13898: Subtask mirroring conf.set(section, option, value)
> [2018-08-08 15:19:24,403] {base_task_runner.py:108} INFO - Job 13898: Subtask mirroring File "/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/_init_.py", line 1239, in set
> [2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask mirroring super(ConfigParser, self).set(section, option, value)
> [2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask mirroring File "/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/_init_.py", line 914, in set
> [2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask mirroring value)
> [2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask mirroring File "/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/_init_.py", line 392, in before_set
> [2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask mirroring "position %d" % (value, tmp_value.find('%')))
> [2018-08-08 15:19:24,406] {base_task_runner.py:108} INFO - Job 13898: Subtask mirroring ValueError: invalid interpolation syntax in <redacted-broker-url>{code}
> The issue was that the broker url had a percent sign, and when the cli called `conf.set(section, option, value)`, it was throwing because it interpreted the percent as an interpolation.
> To avoid this issue, I would propose that the environment variables be escaped when being written in `utils.configuration.tmp_configuration_copy`, so that when `conf.set` is called in `bin/cli`, it doesn't throw on these unescaped values.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)