You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "foldvari (via GitHub)" <gi...@apache.org> on 2023/09/06 10:39:51 UTC
[GitHub] [airflow] foldvari opened a new issue, #34133: NamedTuple causes TypeError on XCOM push
foldvari opened a new issue, #34133:
URL: https://github.com/apache/airflow/issues/34133
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
Before upgrading to v2.6.3 from v2.5.3 we relied on that a NamedTuple can be a return value from a @task annotated task.
After upgrading to v.2.6.3 these tasks are failing with TypeError: Object of type MyClass is not JSON serializable
### What you think should happen instead
New version should be backwards compatible and should continue supporting NamedTuples on XCOM.
### How to reproduce
```python
@task()
def a():
class MyClass(NamedTuple):
a: str
b: int
value = MyClass('A', 42)
return value
```
### Operating System
Debian GNU/Linux 11 (bullseye)
### Versions of Apache Airflow Providers
```
apache-airflow-providers-amazon==7.3.0
apache-airflow-providers-celery==3.1.0
apache-airflow-providers-cncf-kubernetes==5.2.2
apache-airflow-providers-common-sql==1.3.4
apache-airflow-providers-docker==3.5.1
apache-airflow-providers-elasticsearch==4.4.0
apache-airflow-providers-ftp==3.3.1
apache-airflow-providers-google==8.11.0
apache-airflow-providers-grpc==3.1.0
apache-airflow-providers-hashicorp==3.3.0
apache-airflow-providers-http==2.0.3
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-microsoft-azure==5.2.1
apache-airflow-providers-microsoft-winrm==2.0.2
apache-airflow-providers-mysql==4.0.2
apache-airflow-providers-odbc==3.2.1
apache-airflow-providers-postgres==5.4.0
apache-airflow-providers-redis==3.1.0
apache-airflow-providers-samba==2.0.0
apache-airflow-providers-sendgrid==3.1.0
apache-airflow-providers-sftp==2.4.1
apache-airflow-providers-slack==7.2.0
apache-airflow-providers-snowflake==4.0.4
apache-airflow-providers-sqlite==3.3.1
apache-airflow-providers-ssh==3.5.0
```
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #34133: NamedTuple causes TypeError on XCOM push
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #34133:
URL: https://github.com/apache/airflow/issues/34133#issuecomment-1708098608
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #34133: NamedTuple causes TypeError on XCOM push
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #34133:
URL: https://github.com/apache/airflow/issues/34133#issuecomment-1708950237
> An XCom is identified by a key (essentially its name), as well as the task_id and dag_id it came from. They can have any (serializable) value, but they are only designed for small amounts of data; do not use them to pass around large values, like dataframes.
`NamedTuple` is not json serializable, maybe we removed some of our custom serialization in the new serializer introduced in 2.6.0
I recommend replacing it by `pydantic.BaseModel`:
```python
@task()
def a():
from pydantic import BaseModel
class MyClass(BaseModel):
a: str
b: int
value = MyClass(a='A', b=42)
return value
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #34133: NamedTuple causes TypeError on XCOM push
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #34133:
URL: https://github.com/apache/airflow/issues/34133#issuecomment-1709378614
Also you probably missed @foldvari that we have serializers https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/serializers.html#serialization and you can add your own serializers to serialize NamedTuples of yours if you want.
Also you need to make sure your classes serialization/deserialization are added to https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#allowed-deserialization-classes
And you could even contribute it back if you write it in a generic-enough way.
The fact that it (accidentally) worked before was an implementation detail not a promise.
converting it to a discussion in case more discussion is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #34133: NamedTuple causes TypeError on XCOM push
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #34133: NamedTuple causes TypeError on XCOM push
URL: https://github.com/apache/airflow/issues/34133
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org