You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/16 17:37:30 UTC
[GitHub] [airflow] Jake-Gillberg opened a new issue #14838: local_executor failes when password contains `&` in SQL_ALCHEMY_CONN_CMD
Jake-Gillberg opened a new issue #14838:
URL: https://github.com/apache/airflow/issues/14838
**Apache Airflow version / Environment**:
apache/airflow:2.0.1-python3.8
**What happened**:
This is a dev testing password, so I don't mind that it is leaked in the error message here:
`ERROR - Failed to execute task Cannot execute bash -cl 'echo postgresql+psycopg2://airflow:hAY&Bv+w*,q24+JN@airflow_db/airflow'. Error code is: 127. Output: postgresql+psycopg2://airflow:hAY , Stderr: bash: Bv+w*,q24+JN@airflow_db/airflow: No such file or directory.
`
**What you expected to happen**:
Expected the tutorial task to complete successfully.
**What do you think went wrong?**:
Using docker .env to hold sql connection information. Have specified `AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD` as `bash -cl 'echo postgresql+psycopg2://$${POSTGRES_USER}:$${POSTGRES_PASSWORD}@airflow_db/$${POSTGRES_DB}'` in `docker-compose.yml` (double `$$` to escape docker's replacement). This seems to work fine on initialization (webserver, scheduler, and init all run successfully) up until the local executor tries to log the completion of the task.
```
airflow@4bdf5cf69c11:/opt/airflow$ airflow dags backfill tutorial --start-date 2015-06-01 --end-date 2015-06-07
/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/dag_command.py:60 PendingDeprecationWarning: --ignore-first-depends-on-past is deprecated as the value is always set to True
[2021-03-16 16:37:36,822] {dagbag.py:448} INFO - Filling up the DagBag from /opt/airflow/dags
[2021-03-16 16:37:36,945] {backfill_job.py:902} INFO - Reset the following 2 TaskInstances:
<TaskInstance: tutorial.sleep 2015-06-01 00:00:00+00:00 [None]>
<TaskInstance: tutorial.templated 2015-06-01 00:00:00+00:00 [None]>
[2021-03-16 16:37:36,968] {base_executor.py:82} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'tutorial', 'print_date', '2015-06-01T00:00:00+00:00', '--ignore-depends-on-past', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/example.py', '--cfg-path', '/tmp/tmp5ywf__ot']
[2021-03-16 16:37:41,847] {local_executor.py:81} INFO - QueuedLocalWorker running ['airflow', 'tasks', 'run', 'tutorial', 'print_date', '2015-06-01T00:00:00+00:00', '--ignore-depends-on-past', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/example.py', '--cfg-path', '/tmp/tmp5ywf__ot']
[2021-03-16 16:37:41,855] {backfill_job.py:377} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 2 | succeeded: 0 | running: 1 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
[2021-03-16 16:37:41,877] {dagbag.py:448} INFO - Filling up the DagBag from /opt/airflow/dags/example.py
Running <TaskInstance: tutorial.print_date 2015-06-01T00:00:00+00:00 [queued]> on host 4bdf5cf69c11
[2021-03-16 16:37:41,974] {local_executor.py:127} ERROR - Failed to execute task Cannot execute bash -cl 'echo postgresql+psycopg2://airflow:hAY&Bv+w*,q24+JN@airflow_db/airflow'. Error code is: 127. Output: postgresql+psycopg2://airflow:hAY
, Stderr: bash: Bv+w*,q24+JN@airflow_db/airflow: No such file or directory
.
[2021-03-16 16:37:46,873] {backfill_job.py:282} ERROR - Executor reports task instance <TaskInstance: tutorial.print_date 2015-06-01 00:00:00+00:00 [queued]> finished (failed) although the task says its queued. Was the task killed externally? Info: None
[2021-03-16 16:37:46,874] {taskinstance.py:1455} ERROR - Executor reports task instance <TaskInstance: tutorial.print_date 2015-06-01 00:00:00+00:00 [queued]> finished (failed) although the task says its queued. Was the task killed externally? Info: None
NoneType: None
[2021-03-16 16:37:46,874] {taskinstance.py:1496} INFO - Marking task as UP_FOR_RETRY. dag_id=tutorial, task_id=print_date, execution_date=20150601T000000, start_date=20210316T160557, end_date=20210316T163746
[2021-03-16 16:37:46,885] {backfill_job.py:225} WARNING - Task instance <TaskInstance: tutorial.print_date 2015-06-01 00:00:00+00:00 [up_for_retry]> is up for retry
[2021-03-16 16:37:46,890] {backfill_job.py:377} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 3 | succeeded: 0 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #14838: local_executor failes when password contains `&` in SQL_ALCHEMY_CONN_CMD
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #14838:
URL: https://github.com/apache/airflow/issues/14838#issuecomment-826420988
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #14838: local_executor failes when password contains `&` in SQL_ALCHEMY_CONN_CMD
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #14838:
URL: https://github.com/apache/airflow/issues/14838#issuecomment-827083664
@Jake-Gillberg -> '&` is delimiter of bash command. If you want to use it in a password you must escape or quote it:
With &:
```
[jarek:~/code/airflow] make-code-pylint-2-8-compatible+ ± echo postgresql+psycopg2://airflow:hAY&Bv+w*,q24+JN@airflow_db/airflow
[1] 280867
postgresql+psycopg2://airflow:hAY
bash: Bv+w*,q24+JN@airflow_db/airflow: No such file or directory
[1]+ Done echo postgresql+psycopg2://airflow:hAY
```
With \\&:
```
[jarek:~/code/airflow] make-code-pylint-2-8-compatible+ 6s 127 ± echo postgresql+psycopg2://airflow:hAY\&Bv+w*,q24+JN@airflow_db/airflow
postgresql+psycopg2://airflow:hAY&Bv+w*,q24+JN@airflow_db/airflow
[jarek:~/code/airflow] make-code-pylint-2-8-compatible+ 14s ±
```
With quoting:
```
[jarek:~/code/airflow] make-code-pylint-2-8-compatible+ 14s ± echo "postgresql+psycopg2://airflow:hAY&Bv+w*,q24+JN@airflow_db/airflow"
postgresql+psycopg2://airflow:hAY&Bv+w*,q24+JN@airflow_db/airflow
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on issue #14838: local_executor failes when password contains `&` in SQL_ALCHEMY_CONN_CMD
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on issue #14838:
URL: https://github.com/apache/airflow/issues/14838#issuecomment-827078318
@Jake-Gillberg Are you just running the example dag as is? Completely unchanged? Because I don't see anything in the example dag that would the connection DSN.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #14838: local_executor failes when password contains `&` in SQL_ALCHEMY_CONN_CMD
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #14838:
URL: https://github.com/apache/airflow/issues/14838#issuecomment-826420988
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #14838: local_executor failes when password contains `&` in SQL_ALCHEMY_CONN_CMD
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14838:
URL: https://github.com/apache/airflow/issues/14838#issuecomment-827083664
@Jake-Gillberg -> '&` is delimited of bash command. If you want to use it in a password you must escape it:
With &:
```
[jarek:~/code/airflow] make-code-pylint-2-8-compatible+ ± echo postgresql+psycopg2://airflow:hAY&Bv+w*,q24+JN@airflow_db/airflow
[1] 280867
postgresql+psycopg2://airflow:hAY
bash: Bv+w*,q24+JN@airflow_db/airflow: No such file or directory
[1]+ Done echo postgresql+psycopg2://airflow:hAY
```
With \&:
```
[jarek:~/code/airflow] make-code-pylint-2-8-compatible+ 6s 127 ± echo postgresql+psycopg2://airflow:hAY\&Bv+w*,q24+JN@airflow_db/airflow
postgresql+psycopg2://airflow:hAY&Bv+w*,q24+JN@airflow_db/airflow
[jarek:~/code/airflow] make-code-pylint-2-8-compatible+ 14s ±
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #14838: local_executor failes when password contains `&` in SQL_ALCHEMY_CONN_CMD
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #14838:
URL: https://github.com/apache/airflow/issues/14838#issuecomment-808344201
What is your task definition? I think the problem is the quoting of your bash_command attribute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Jake-Gillberg commented on issue #14838: local_executor failes when password contains `&` in SQL_ALCHEMY_CONN_CMD
Posted by GitBox <gi...@apache.org>.
Jake-Gillberg commented on issue #14838:
URL: https://github.com/apache/airflow/issues/14838#issuecomment-808364008
It is exactly the tutorial task / dag: https://airflow.apache.org/docs/apache-airflow/stable/_modules/airflow/example_dags/tutorial.html
The problem isn't with running the task, but with reporting that it has finished to the airflow database
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #14838: local_executor failes when password contains `&` in SQL_ALCHEMY_CONN_CMD
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #14838:
URL: https://github.com/apache/airflow/issues/14838#issuecomment-827083664
@Jake-Gillberg -> '&` is delimiter of bash command. If you want to use it in a password you must escape or quote it:
With &:
```
[jarek:~/code/airflow] make-code-pylint-2-8-compatible+ ± echo postgresql+psycopg2://airflow:hAY&Bv+w*,q24+JN@airflow_db/airflow
[1] 280867
postgresql+psycopg2://airflow:hAY
bash: Bv+w*,q24+JN@airflow_db/airflow: No such file or directory
[1]+ Done echo postgresql+psycopg2://airflow:hAY
```
With \\&:
```
[jarek:~/code/airflow] make-code-pylint-2-8-compatible+ 6s 127 ± echo postgresql+psycopg2://airflow:hAY\&Bv+w*,q24+JN@airflow_db/airflow
postgresql+psycopg2://airflow:hAY&Bv+w*,q24+JN@airflow_db/airflow
[jarek:~/code/airflow] make-code-pylint-2-8-compatible+ 14s ±
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #14838: local_executor failes when password contains `&` in SQL_ALCHEMY_CONN_CMD
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #14838:
URL: https://github.com/apache/airflow/issues/14838#issuecomment-827078318
@Jake-Gillberg Are you just running the example dag as is? Completely unchanged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Jake-Gillberg commented on issue #14838: local_executor failes when password contains `&` in SQL_ALCHEMY_CONN_CMD
Posted by GitBox <gi...@apache.org>.
Jake-Gillberg commented on issue #14838:
URL: https://github.com/apache/airflow/issues/14838#issuecomment-826852953
Bumping. Also noting that this probably shouldn't be marked as pending-response, but please let me know if you need any more information from me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #14838: local_executor failes when password contains `&` in SQL_ALCHEMY_CONN_CMD
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #14838:
URL: https://github.com/apache/airflow/issues/14838#issuecomment-827083664
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #14838: local_executor failes when password contains `&` in SQL_ALCHEMY_CONN_CMD
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #14838:
URL: https://github.com/apache/airflow/issues/14838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #14838: local_executor failes when password contains `&` in SQL_ALCHEMY_CONN_CMD
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14838:
URL: https://github.com/apache/airflow/issues/14838#issuecomment-800468667
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org