You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/07/10 18:30:59 UTC
[GitHub] [airflow] prakshalj0512 opened a new issue #9753: JSON notation in Airflow DAG comments causing KeyError
prakshalj0512 opened a new issue #9753:
URL: https://github.com/apache/airflow/issues/9753
-->
**Apache Airflow version**: 1.10.11 (code is working fine in 1.10.10)
**Environment**: Python 3.7
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release): Catalina 10.15.5
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
KeyError: '"maxDBEntryAgeInDays"'
**What you expected to happen**:
Expected the block comment on the top of the DAG to be copied as is into the DAG description box in the UI
**How to reproduce it**:
Add the following code to the top of your DAG
"""
This is a test dag. this is the description of it. This is how you can trigger it with CLI.
airflow trigger_dag --conf '{"randomKey":30}' airflow-test-dag
--conf options:
randomKey:<INT> - Optional
"""
**Anything else we need to know**: Works fine with Airflow 1.10.10 and Python 3.7.
** UI Logs: **
```
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 69, in inner
return self._run_view(f, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 368, in _run_view
return fn(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/flask_login/utils.py", line 258, in decorated_view
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line 380, in view_func
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line 286, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 1749, in graph
doc_md = wrapped_markdown(getattr(dag, 'doc_md', None), css_class='dag-doc')
File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 258, in wrapped_markdown
).format(css_class=css_class)
File "/usr/local/lib/python3.7/site-packages/markupsafe/__init__.py", line 213, in format
return self.__class__(formatter.vformat(self, args, kwargs))
File "/usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/lib/python3.7/string.py", line 190, in vformat
result, _ = self._vformat(format_string, args, kwargs, used_args, 2)
File "/usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/lib/python3.7/string.py", line 230, in _vformat
obj, arg_used = self.get_field(field_name, args, kwargs)
File "/usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/lib/python3.7/string.py", line 295, in get_field
obj = self.get_value(first, args, kwargs)
File "/usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/lib/python3.7/string.py", line 252, in get_value
return kwargs[key]
File "/usr/local/lib/python3.7/site-packages/markupsafe/__init__.py", line 249, in __getitem__
return self._kwargs[key]
KeyError: '"randomKey"'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on issue #9753: JSON notation in Airflow DAG comments causing KeyError
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #9753:
URL: https://github.com/apache/airflow/issues/9753#issuecomment-656887293
I was able to reproduce it but looks like it can be easily fixed if you remove the following from docstrings from the DAG (https://github.com/teamclairvoyant/airflow-maintenance-dags/blob/63ddbfb19d53fa7b44343dc0525cf1b60f16fed6/db-cleanup/airflow-db-cleanup.py#L6-L9):
```
airflow trigger_dag --conf '{"maxDBEntryAgeInDays":30}' airflow-db-cleanup
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #9753: JSON notation in Airflow DAG comments causing KeyError
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #9753:
URL: https://github.com/apache/airflow/issues/9753#issuecomment-657217783
> Isn't that the issue with DAG serialization of the comment field? It looks awfully like it.
No, I could reproduce it without DAG Serialization.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #9753: JSON notation in Airflow DAG comments causing KeyError
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #9753:
URL: https://github.com/apache/airflow/issues/9753#issuecomment-656875791
Can you show us your DAG please?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] prakshalj0512 edited a comment on issue #9753: JSON notation in Airflow DAG comments causing KeyError
Posted by GitBox <gi...@apache.org>.
prakshalj0512 edited a comment on issue #9753:
URL: https://github.com/apache/airflow/issues/9753#issuecomment-656882130
https://github.com/teamclairvoyant/airflow-maintenance-dags/blob/master/db-cleanup/airflow-db-cleanup.py
Reference Lines 176-177
```
if hasattr(dag, 'doc_md'):
dag.doc_md = __doc__
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] prakshalj0512 commented on issue #9753: JSON notation in Airflow DAG comments causing KeyError
Posted by GitBox <gi...@apache.org>.
prakshalj0512 commented on issue #9753:
URL: https://github.com/apache/airflow/issues/9753#issuecomment-656882130
https://github.com/teamclairvoyant/airflow-maintenance-dags/blob/master/db-cleanup/airflow-db-cleanup.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] prakshalj0512 commented on issue #9753: JSON notation in Airflow DAG comments causing KeyError
Posted by GitBox <gi...@apache.org>.
prakshalj0512 commented on issue #9753:
URL: https://github.com/apache/airflow/issues/9753#issuecomment-656828497
```
Flask 1.1.2
Flask-Admin 1.5.4
Flask-AppBuilder 2.3.0
Flask-Babel 1.0.0
Flask-Caching 1.3.3
Flask-JWT-Extended 3.24.1
Flask-Login 0.4.1
Flask-OpenID 1.2.5
Flask-RESTful 0.3.8
Flask-SQLAlchemy 2.4.1
Flask-WTF 0.14.3
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #9753: JSON notation in Airflow DAG comments causing KeyError
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #9753:
URL: https://github.com/apache/airflow/issues/9753#issuecomment-656828044
@prakshalj0512 what version of flask do you use?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #9753: JSON notation in Airflow DAG comments causing KeyError
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #9753:
URL: https://github.com/apache/airflow/issues/9753#issuecomment-656887293
I was able to reproduce it but looks like it can be easily fixed if you remove the following from docstrings from the DAG (https://github.com/teamclairvoyant/airflow-maintenance-dags/blob/63ddbfb19d53fa7b44343dc0525cf1b60f16fed6/db-cleanup/airflow-db-cleanup.py#L6-L9):
```
airflow trigger_dag --conf '{"maxDBEntryAgeInDays":30}' airflow-db-cleanup
--conf options:
maxDBEntryAgeInDays:<INT> - Optional
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #9753: JSON notation in Airflow DAG comments causing KeyError
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #9753:
URL: https://github.com/apache/airflow/issues/9753#issuecomment-657218855
Right I know what it is and the problem is also in master. Fix is coming.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #9753: JSON notation in Airflow DAG comments causing KeyError
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #9753:
URL: https://github.com/apache/airflow/issues/9753#issuecomment-657216283
Isn't that the issue with DAG serialization of the comment field? It looks awfully like it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] prakshalj0512 commented on issue #9753: JSON notation in Airflow DAG comments causing KeyError
Posted by GitBox <gi...@apache.org>.
prakshalj0512 commented on issue #9753:
URL: https://github.com/apache/airflow/issues/9753#issuecomment-656887936
Yes, I narrowed it down to the curly braces that were causing the issue. Not sure what's causing them to be interpreted as part of JSON as opposed to the comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #9753: JSON notation in Airflow DAG comments causing KeyError
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #9753:
URL: https://github.com/apache/airflow/issues/9753
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org