You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/31 03:25:06 UTC
[GitHub] [airflow] 11010cy opened a new issue #22638: 'dict' object has no attribute 'dump'
11010cy opened a new issue #22638:
URL: https://github.com/apache/airflow/issues/22638
### Apache Airflow version
2.2.2
### What happened
get 500 while request 'api/v1/dags/{dag_id}/tasks'. It happens randomly.
```
AttributeError: 'dict' object has no attribute 'dump'
File "flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "flask/_compat.py", line 39, in reraise
raise value
File "flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "airflow/_vendor/connexion/decorators/decorator.py", line 48, in wrapper
response = function(request)
File "airflow/_vendor/connexion/decorators/uri_parsing.py", line 144, in wrapper
response = function(request)
File "airflow/_vendor/connexion/decorators/validation.py", line 384, in wrapper
return function(request)
File "airflow/_vendor/connexion/decorators/response.py", line 103, in wrapper
response = function(request)
File "airflow/_vendor/connexion/decorators/parameter.py", line 121, in wrapper
return function(**kwargs)
File "airflow/api_connexion/security.py", line 47, in decorated
return func(*args, **kwargs)
File "airflow/api_connexion/endpoints/task_endpoint.py", line 71, in get_tasks
return task_collection_schema.dump(task_collection)
File "marshmallow/schema.py", line 552, in dump
result = self._serialize(processed_obj, many=many)
File "marshmallow/schema.py", line 520, in _serialize
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
File "marshmallow/fields.py", line 338, in serialize
return self._serialize(value, attr, obj, **kwargs)
File "marshmallow/fields.py", line 765, in _serialize
return [self.inner._serialize(each, attr, obj, **kwargs) for each in value]
File "marshmallow/fields.py", line 765, in <listcomp>
return [self.inner._serialize(each, attr, obj, **kwargs) for each in value]
File "marshmallow/fields.py", line 634, in _serialize
return schema.dump(nested_obj, many=many)
File "marshmallow/schema.py", line 552, in dump
result = self._serialize(processed_obj, many=many)
File "marshmallow/schema.py", line 520, in _serialize
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
File "marshmallow/fields.py", line 338, in serialize
return self._serialize(value, attr, obj, **kwargs)
File "marshmallow/fields.py", line 1870, in _serialize
return self._serialize_method(obj)
File "airflow/api_connexion/schemas/task_schema.py", line 70, in get_params
return {k: v.dump() for k, v in params.items()}
File "airflow/api_connexion/schemas/task_schema.py", line 70, in <dictcomp>
return {k: v.dump() for k, v in params.items()}
```
After some inspection, I have located the cause of `task.params`.
In some cases, `current_app.dag_bag.get_dag(dag_id)` will return a dag with one of task.params is type `dict` instead of type `ParamsDict`. It happens in the same gunicorn worker process `15994`, while the `current_app.dag_bag.dags` supposed to be the same instance. Before the error happened, there was nothing wrong. But 2 minutes after, the error happend.
I have print the pid and the id of task.params before and when the above error happend.
BEFORE
```[params] dagbag min_serialized_dag_fetch_secs=0:00:10
[params] check_updated sd_last_updated_datetime=2022-03-30 04:18:07.180249+00:00
[params] pid=159994 thread=MainThread id=139950227846976
[params] get_tasks 139944558692128 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f476175d040>, 'tianji': <airflow.models.param.Param objec
t at 0x7f476175d130>, 'business': <airflow.models.param.Param object at 0x7f476175dcd0>}
[params] get_tasks 139944558781120 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f47617bbbb0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617bbf70>, 'business': <airflow.models.param.Param object at 0x7f476174c9a0>}
[params] get_tasks 139944558691008 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761704520>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761704550>, 'business': <airflow.models.param.Param object at 0x7f4761704250>}
[params] get_tasks 139944558691488 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761751d00>, 'tianji': <airflow.models.param.Param objec
t at 0x7f476184bc10>, 'business': <airflow.models.param.Param object at 0x7f47617bbe80>}
[params] get_tasks 139944558782080 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761742550>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761742c70>, 'business': <airflow.models.param.Param object at 0x7f4761742fa0>}
[params] get_tasks 139944558781360 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f47617041f0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617045e0>, 'business': <airflow.models.param.Param object at 0x7f47617044c0>}
[params] get_tasks 139944558782720 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761723460>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617234c0>, 'business': <airflow.models.param.Param object at 0x7f4761723550>}
[params] get_tasks 139949715461600 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761723fa0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761723fd0>, 'business': <airflow.models.param.Param object at 0x7f4761723eb0>}
[params] get_tasks 139949730158048 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f47617238e0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617239d0>, 'business': <airflow.models.param.Param object at 0x7f4761723790>}
[params] get_tasks 139944558961024 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761704e20>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761704430>, 'business': <airflow.models.param.Param object at 0x7f4761704850>}
[params] get_tasks 139949716343280 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f476175d970>, 'tianji': <airflow.models.param.Param objec
t at 0x7f476175d3a0>, 'business': <airflow.models.param.Param object at 0x7f476175dac0>}
[params] get_tasks 139944559033456 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f476174c5b0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f476174cb80>, 'business': <airflow.models.param.Param object at 0x7f476174cd60>}
127.0.0.1 - airflow [30/Mar/2022:14:04:38 +0800] "GET /api/v1/dags/xxx/tasks HTTP/1.0" 200 64063 "-" "OpenAPI-Generator/2.2.0/python"
```
WHEN
```params] dagbag min_serialized_dag_fetch_secs=0:00:10
[params] check_updated sd_last_updated_datetime=2022-03-30 04:18:07.180249+00:00
[params] pid=159994 thread=MainThread id=139950227846976
[params] get_tasks 139944558692128 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f476175d040>, 'tianji': <airflow.models.param.Param objec
t at 0x7f476175d130>, 'business': <airflow.models.param.Param object at 0x7f476175dcd0>}
[params] get_tasks 139944558781120 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f47617bbbb0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617bbf70>, 'business': <airflow.models.param.Param object at 0x7f476174c9a0>}
[params] get_tasks 139944558691008 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761704520>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761704550>, 'business': <airflow.models.param.Param object at 0x7f4761704250>}
[params] get_tasks 139944558691488 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761751d00>, 'tianji': <airflow.models.param.Param objec
t at 0x7f476184bc10>, 'business': <airflow.models.param.Param object at 0x7f47617bbe80>}
[params] get_tasks 139944558782080 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761742550>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761742c70>, 'business': <airflow.models.param.Param object at 0x7f4761742fa0>}
[params] get_tasks 139944558781360 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f47617041f0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617045e0>, 'business': <airflow.models.param.Param object at 0x7f47617044c0>}
[params] get_tasks 139944558782720 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761723460>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617234c0>, 'business': <airflow.models.param.Param object at 0x7f4761723550>}
[params] get_tasks 139949715461600 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761723fa0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761723fd0>, 'business': <airflow.models.param.Param object at 0x7f4761723eb0>}
[params] get_tasks 139949730158048 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f47617238e0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617239d0>, 'business': <airflow.models.param.Param object at 0x7f4761723790>}
[params] get_tasks 139944558961024 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761704e20>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761704430>, 'business': <airflow.models.param.Param object at 0x7f4761704850>}
[params] get_tasks 139944557266624 <class 'dict'> {xxx}
[params] get_tasks 139944559033456 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f476174c5b0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f476174cb80>, 'business': <airflow.models.param.Param object at 0x7f476174cd60>}
[^[[34m2022-03-30 14:06:30,706^[[0m] {^[[34mapp.py:^[[0m1891} ERROR^[[0m - Exception on /api/v1/dags/troubleshooter_15/tasks [GET]^[[0m
```
We could see that task `139949716343280` with type `ParamsDict` changed to `139944557266624` with type `dict`, while the other task object hasn't changed.
Does anyone knows: when and where could the `task.params` in a dag of `current_app.dag_bag.dags` be changed from `ParamsDict` to `dict`
### What you think should happen instead
There's maybe a bug related to `current_app.dag_bag.dags`.
### How to reproduce
I have no idea, it happened occasionally.
### Operating System
Linux Server 7.2 (Paladin)
### Versions of Apache Airflow Providers
apache-airflow-providers-alibaba==1.0.0
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-mysql==2.1.1
apache-airflow-providers-sqlite==2.0.1
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #22638: 'dict' object has no attribute 'dump'
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #22638:
URL: https://github.com/apache/airflow/issues/22638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] 11010cy edited a comment on issue #22638: 'dict' object has no attribute 'dump'
Posted by GitBox <gi...@apache.org>.
11010cy edited a comment on issue #22638:
URL: https://github.com/apache/airflow/issues/22638#issuecomment-1084216788
I found that it caused by requesting `/dags/{}/dagRuns/{dag_run_id}/taskInstances/{task_instance_id}/logs/1`.
```
// utils/log/file_task_handler.py _render_filename
context = ti.get_template_context()
```
but get_template_context has the side effect that it changed task.params to dict. (params.validate() return a dict)
```
// models/taskintance.py get_template_context
validated_params = task.params = params.validate()
```
The related commits are:
https://github.com/apache/airflow/commit/2258e13cc78faec80054c223eca9378fd33e18fd
https://github.com/apache/airflow/commit/23a68fa4e61ba795340f095ab64fbcaeebd57db4
The problem was introduced by https://github.com/apache/airflow/commit/23a68fa4e61ba795340f095ab64fbcaeebd57db4, fixed at https://github.com/apache/airflow/commit/2258e13cc78faec80054c223eca9378fd33e18fd (not released). And exists in apache-airflow >= 2.1.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #22638: 'dict' object has no attribute 'dump'
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #22638:
URL: https://github.com/apache/airflow/issues/22638#issuecomment-1084032819
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22638: 'dict' object has no attribute 'dump'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22638:
URL: https://github.com/apache/airflow/issues/22638#issuecomment-1084273043
So seems that one is fixed in #21495. Closing it then. Will be released as part of 2.3.0 I believe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] 11010cy edited a comment on issue #22638: 'dict' object has no attribute 'dump'
Posted by GitBox <gi...@apache.org>.
11010cy edited a comment on issue #22638:
URL: https://github.com/apache/airflow/issues/22638#issuecomment-1084216788
I found that it caused by requesting `/dags/{}/dagRuns/{dag_run_id}/taskInstances/{task_instance_id}/logs/1`.
```
// utils/log/file_task_handler.py _render_filename
context = ti.get_template_context()
```
but get_template_context has the side effect that it changed task.params to dict. (params.validate() return a dict)
```
// models/taskintance.py get_template_context
validated_params = task.params = params.validate()
```
The related commits are:
https://github.com/apache/airflow/commit/2258e13cc78faec80054c223eca9378fd33e18fd
https://github.com/apache/airflow/commit/23a68fa4e61ba795340f095ab64fbcaeebd57db4
The problem was introduced by https://github.com/apache/airflow/commit/23a68fa4e61ba795340f095ab64fbcaeebd57db4, fixed at https://github.com/apache/airflow/commit/2258e13cc78faec80054c223eca9378fd33e18fd(not released). And exists in apache-airflow >= 2.1.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] 11010cy commented on issue #22638: 'dict' object has no attribute 'dump'
Posted by GitBox <gi...@apache.org>.
11010cy commented on issue #22638:
URL: https://github.com/apache/airflow/issues/22638#issuecomment-1084216788
I found that it caused by requesting `/dags/{}/dagRuns/{dag_run_id}/taskInstances/{task_instance_id}/logs/1`.
```
// utils/log/file_task_handler.py _render_filename
context = ti.get_template_context()
```
but get_template_context has the side effect that it changed task.params to dict. (params.validate() return a dict)
```
// models/taskintance.py get_template_context
validated_params = task.params = params.validate()
```
The related commits are:
https://github.com/apache/airflow/commit/2258e13cc78faec80054c223eca9378fd33e18fd
https://github.com/apache/airflow/commit/23a68fa4e61ba795340f095ab64fbcaeebd57db4
The problem was introduced by https://github.com/apache/airflow/commit/23a68fa4e61ba795340f095ab64fbcaeebd57db4, fixed at https://github.com/apache/airflow/commit/2258e13cc78faec80054c223eca9378fd33e18fd. And exists in apache-airflow >= 2.1.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org