You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/10/17 06:23:11 UTC
[GitHub] [airflow] tirkarthi opened a new issue, #27084: Airflow TaskInstance endpoing API error for old task instances : 'V1Container' object has no attribute '_startup_probe'
tirkarthi opened a new issue, #27084:
URL: https://github.com/apache/airflow/issues/27084
### Apache Airflow version
main (development)
### What happened
Opening issue as per comment : https://github.com/apache/airflow/issues/23727#issuecomment-1207530417 . We have also noticed this issue where we have a 2.1.x setup using Kubernetes executor and on upgrade the new task instances created post upgrade are working fine in task instance endpoint. The old objects fail with similar traceback as below. We also tried main branch (2.4.1 as of writing) and it also has same issue. It seems a fix similar to https://github.com/apache/airflow/pull/24117 has to be made with a custom String field for `executor_config` in task instance schema that calls the `_serialize` method and on error returns empty dict as string. We have a fix internally though a test case might not be possible since it needs an older value of executor_config that we can't export due to private data. I will be happy to make a PR with the fix though and opened this issue for discussion,
```
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/api_connexion/endpoints/task_instance_endpoint.py", line 412, in get_task_instances_batch
return task_instance_collection_schema.dump(
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 557, in dump
result = self._serialize(processed_obj, many=many)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 525, in _serialize
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 342, in serialize
return self._serialize(value, attr, obj, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 774, in _serialize
return [self.inner._serialize(each, attr, obj, **kwargs) for each in value]
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 774, in <listcomp>
return [self.inner._serialize(each, attr, obj, **kwargs) for each in value]
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 643, in _serialize
return schema.dump(nested_obj, many=many)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 557, in dump
result = self._serialize(processed_obj, many=many)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 525, in _serialize
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 342, in serialize
return self._serialize(value, attr, obj, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 893, in _serialize
return utils.ensure_text_type(value)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/utils.py", line 212, in ensure_text_type
return str(val)
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 214, in __repr__
return self.to_str()
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 210, in to_str
return pprint.pformat(self.to_dict())
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 196, in to_dict
result[attr] = value.to_dict()
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod_spec.py", line 1058, in to_dict
result[attr] = list(map(
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod_spec.py", line 1059, in <lambda>
lambda x: x.to_dict() if hasattr(x, "to_dict") else x,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_container.py", line 660, in to_dict
value = getattr(self, attr)
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_container.py", line 458, in startup_probe
return self._startup_probe
AttributeError: 'V1Container' object has no attribute '_startup_probe'
```
cc: @dstandish @joshzana
### What you think should happen instead
Older task instances that cannot be serialized should possibly return empty dict instead of failing completely.
### How to reproduce
1. Upgrade to main branch from an older version like 2.1.x with task instances and executor_config using Kubernetes as executor.
2. Hit task instance endpoint of old dagrun to fetch task instances with executor_config serialized. The traceback is thrown.
### Operating System
Redhat
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #27084: Airflow TaskInstance endpoing API error for old task instances : 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27084:
URL: https://github.com/apache/airflow/issues/27084#issuecomment-1281097228
Do mention this issue in the PR and tag me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27084: Airflow TaskInstance endpoing API error for old task instances : 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27084:
URL: https://github.com/apache/airflow/issues/27084#issuecomment-1374415082
Closing it provisionally then as fixed (we can always re-open if it is not).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #27084: Airflow TaskInstance endpoing API error for old task instances : 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #27084: Airflow TaskInstance endpoing API error for old task instances : 'V1Container' object has no attribute '_startup_probe'
URL: https://github.com/apache/airflow/issues/27084
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #27084: Airflow TaskInstance endpoing API error for old task instances : 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27084:
URL: https://github.com/apache/airflow/issues/27084#issuecomment-1281096148
Oh I see so your concern is specifically with the API. I didn't register that initially. Yeah I'm sure there could be a fix for the scenario you mention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #27084: Airflow TaskInstance endpoing API error for old task instances : 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27084:
URL: https://github.com/apache/airflow/issues/27084#issuecomment-1280375941
In 2.4.1 you probably still get that error because your raw value has already been pickled with the old k8s library and with the new version of the libary you can't unpickle it so there's nothnig you can do at that point.
What https://github.com/apache/airflow/pull/24117 does though is fix it on a go forward basis (by serializing to json) so we shouldn't have issues.
Note that we followed up that one with https://github.com/apache/airflow/pull/26191, which fixed an issue we didn't catch in testing where the webserver would bork the executor config by repeatedly applying the serialization logic.
I don't think there's anything to be done about configs that are already pickled with the old library in this way, but if you have something to contribute feel free to open a PR. There's always a way to test.
In any case this appears to be a duplicate of https://github.com/apache/airflow/issues/23727.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27084: Airflow TaskInstance endpoing API error for old task instances : 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27084:
URL: https://github.com/apache/airflow/issues/27084#issuecomment-1289145544
@tirkarthi - I assigned you to it :). If you think you can make PR - cool, if you think it's not worth - let us know and we will close this one (you can also close it yourself).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #27084: Airflow TaskInstance endpoing API error for old task instances : 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27084:
URL: https://github.com/apache/airflow/issues/27084#issuecomment-1374383232
I believe this is resolved by https://github.com/apache/airflow/pull/28454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tirkarthi commented on issue #27084: Airflow TaskInstance endpoing API error for old task instances : 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
tirkarthi commented on issue #27084:
URL: https://github.com/apache/airflow/issues/27084#issuecomment-1280507909
Thanks @dstandish, agree with your point that the pickled value stored in database is incompatible. Some of our users might use ~ to fetch all task instances of all dagruns where they hit this with one faulty task instance causing server error for all other compatible task instance objects in the API. They don't really use `executor_config` and hence we did a patch similar to linked PR where we added an except block to log the exception and return an empty dict. I will open a PR for discussion but will be okay if the PR is declined too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org