You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/31 20:51:54 UTC
[GitHub] [airflow] paulhtremblay opened a new issue #13414: DAG raises error when passing non serializable JSON object via trigger
paulhtremblay opened a new issue #13414:
URL: https://github.com/apache/airflow/issues/13414
When passing a non serializable JSON object in a trigger, I get the following error below. The logs become unavailable.
my code:
task_trigger_ad_attribution = TriggerDagRunOperator(
37 task_id='trigger_ad_attribution',
38 trigger_dag_id=AD_ATTRIBUTION_DAG_ID,
39 conf={"message": "Triggered from display trigger",
40 'trigger_info':
41 {'dag_id':DAG_ID,
42 'now':datetime.datetime.now(),
43 },
44 'trigger_date' : '{{execution_date}}'
45 },
46 )
Ooops!
Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.
Python version: 3.6.9
Airflow version: 2.0.0
Node: henry-Inspiron-5566
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/home/henry/Envs2/airflow3/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/henry/Envs2/airflow3/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/henry/Envs2/airflow3/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/henry/Envs2/airflow3/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/henry/Envs2/airflow3/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/henry/Envs2/airflow3/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/henry/Envs2/airflow3/lib/python3.6/site-packages/airflow/www/auth.py", line 34, in decorated
return func(*args, **kwargs)
File "/home/henry/Envs2/airflow3/lib/python3.6/site-packages/airflow/www/decorators.py", line 97, in view_func
return f(*args, **kwargs)
File "/home/henry/Envs2/airflow3/lib/python3.6/site-packages/airflow/www/decorators.py", line 60, in wrapper
return f(*args, **kwargs)
File "/home/henry/Envs2/airflow3/lib/python3.6/site-packages/airflow/www/views.py", line 1997, in tree
data = htmlsafe_json_dumps(data, separators=(',', ':'))
File "/home/henry/Envs2/airflow3/lib/python3.6/site-packages/jinja2/utils.py", line 614, in htmlsafe_json_dumps
dumper(obj, **kwargs)
File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python3.6/json/encoder.py", line 180, in default
o.__class__.__name__)
TypeError: Object of type 'datetime' is not JSON serializable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #13414: DAG raises error when passing non serializable JSON object via trigger
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #13414:
URL: https://github.com/apache/airflow/issues/13414
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13414: DAG raises error when passing non serializable JSON object via trigger
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13414:
URL: https://github.com/apache/airflow/issues/13414#issuecomment-753136754
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13414: DAG raises error when passing non serializable JSON object via trigger
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13414:
URL: https://github.com/apache/airflow/issues/13414#issuecomment-776719383
Currently, we don't support it as we need to de-serialize it for Webserver and show it in the UI in List DagRun page. Needs a bigger change to support other objects that are not JSON serializable atm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wuchichung commented on issue #13414: DAG raises error when passing non serializable JSON object via trigger
Posted by GitBox <gi...@apache.org>.
wuchichung commented on issue #13414:
URL: https://github.com/apache/airflow/issues/13414#issuecomment-919804598
We need this feature because previous we can get over it by passing a python_callable and now it is not possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #13414: DAG raises error when passing non serializable JSON object via trigger
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13414:
URL: https://github.com/apache/airflow/issues/13414#issuecomment-776655239
@hedrickw That can _probably_ be done, yes, just a little bit more invovled a fix.
It would be nice though, I agree.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] paulhtremblay commented on issue #13414: DAG raises error when passing non serializable JSON object via trigger
Posted by GitBox <gi...@apache.org>.
paulhtremblay commented on issue #13414:
URL: https://github.com/apache/airflow/issues/13414#issuecomment-769446594
Yes, in some contexts I change datetime to string. This fix should prevent unexplained crashes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #13414: DAG raises error when passing non serializable JSON object via trigger
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #13414:
URL: https://github.com/apache/airflow/issues/13414#issuecomment-753458492
Interesting, I think that in general we always assumed that `conf` will be JSON serialisable as it's usually passed via UI/API but the `TriggerDagRunOperator` is something different. Additionally the `conf` column of `DagRun` is `PickleType` and I thought that we abandoned pickling? CC @kaxil
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hedrickw commented on issue #13414: DAG raises error when passing non serializable JSON object via trigger
Posted by GitBox <gi...@apache.org>.
hedrickw commented on issue #13414:
URL: https://github.com/apache/airflow/issues/13414#issuecomment-776173348
@kaxil any reason we wouldn't allow conf to be an XComArg?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hedrickw commented on issue #13414: DAG raises error when passing non serializable JSON object via trigger
Posted by GitBox <gi...@apache.org>.
hedrickw commented on issue #13414:
URL: https://github.com/apache/airflow/issues/13414#issuecomment-776717798
@ashb Ya it currently works in 2.0.0 if your XComArg output is a Dict, I have a pipeline setup that uses this functionality so I might just write my own version of the TriggerDagRun Operator for now since i needed some of the bug fixes from 2.0.1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on issue #13414: DAG raises error when passing non serializable JSON object via trigger
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #13414:
URL: https://github.com/apache/airflow/issues/13414#issuecomment-769248659
We need to add the diff at the end in `TriggerDagRunOperator`, we don't want to allow arbitrary objects. To pass datetime you should convert it to str, you could do `datetime.datetime.now().isoformat()`:
```python
In [2]: import datetime
In [3]: datetime.datetime.now().isoformat()
Out[3]: '2021-01-28T17:28:14.082547'
```
```diff
diff --git a/airflow/operators/trigger_dagrun.py b/airflow/operators/trigger_dagrun.py
index 63d336115..86078bbec 100644
--- a/airflow/operators/trigger_dagrun.py
+++ b/airflow/operators/trigger_dagrun.py
@@ -23,6 +23,7 @@ from typing import Dict, List, Optional, Union
from airflow.api.common.experimental.trigger_dag import trigger_dag
from airflow.exceptions import AirflowException, DagNotFound, DagRunAlreadyExists
from airflow.models import BaseOperator, BaseOperatorLink, DagBag, DagModel, DagRun
+from airflow.settings import json
from airflow.utils import timezone
from airflow.utils.decorators import apply_defaults
from airflow.utils.helpers import build_airflow_url_with_query
@@ -108,6 +109,11 @@ class TriggerDagRunOperator(BaseOperator):
self.execution_date: Optional[datetime.datetime] = execution_date # type: ignore
+ try:
+ json.dumps(self.conf)
+ except TypeError:
+ raise AirflowException("conf parameter should be JSON Serializable")
+
def execute(self, context: Dict):
if isinstance(self.execution_date, datetime.datetime):
execution_date = self.execution_date
```
This diff will enforce that value passed to `conf` is JSON Serializable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13414: DAG raises error when passing non serializable JSON object via trigger
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13414:
URL: https://github.com/apache/airflow/issues/13414#issuecomment-769248659
We need to add the diff at the end in `TriggerDagRunOperator`, we don't want to allow arbitrary objects. To pass datetime you should convert it to str, you could do `datetime.datetime.now().isoformat()`:
```python
In [2]: import datetime
In [3]: datetime.datetime.now().isoformat()
Out[3]: '2021-01-28T17:28:14.082547'
```
```diff
diff --git a/airflow/operators/trigger_dagrun.py b/airflow/operators/trigger_dagrun.py
index 63d336115..86078bbec 100644
--- a/airflow/operators/trigger_dagrun.py
+++ b/airflow/operators/trigger_dagrun.py
@@ -23,6 +23,7 @@ from typing import Dict, List, Optional, Union
from airflow.api.common.experimental.trigger_dag import trigger_dag
from airflow.exceptions import AirflowException, DagNotFound, DagRunAlreadyExists
from airflow.models import BaseOperator, BaseOperatorLink, DagBag, DagModel, DagRun
+from airflow.settings import json
from airflow.utils import timezone
from airflow.utils.decorators import apply_defaults
from airflow.utils.helpers import build_airflow_url_with_query
@@ -108,6 +109,11 @@ class TriggerDagRunOperator(BaseOperator):
self.execution_date: Optional[datetime.datetime] = execution_date # type: ignore
+ try:
+ json.dumps(self.conf)
+ except TypeError:
+ raise AirflowException("conf parameter should be JSON Serializable")
+
def execute(self, context: Dict):
if isinstance(self.execution_date, datetime.datetime):
execution_date = self.execution_date
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org