You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/06 09:35:31 UTC
[GitHub] [airflow] herunyu opened a new issue, #26174: API Endpoints - /xcomEntries/{xcom_key} cannot deserialize customized xcom backend
herunyu opened a new issue, #26174:
URL: https://github.com/apache/airflow/issues/26174
### Description
We use S3 as our xcom backend database and write serialize/deserialize method for xcoms.
However, when we want to access xcom through REST API, it returns the s3 file url instead of the deserialized value. Could you please add the feature to support customized xcom backend for REST API access?
### Use case/motivation
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #26174: API Endpoints - /xcomEntries/{xcom_key} cannot deserialize customized xcom backend
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #26174:
URL: https://github.com/apache/airflow/issues/26174#issuecomment-1238967277
Ah I see, thanks for the additional context. I think calling `deseralize_value` would be problematic because not everything deserialised can be sent through the wire, which is probably why the current implementation doesn’t do that.
One way I can think of to make this work would be to add an additional function in the XCom backend protocol for “serialise for API”. The default (JSON) backend would simply use the same serialisation logic used to save the value to the database, but custom backends can implement it differently to dump data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #26174: API Endpoints - /xcomEntries/{xcom_key} cannot deserialize customized xcom backend
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #26174:
URL: https://github.com/apache/airflow/issues/26174#issuecomment-1238971885
Note that we’re quite close to the feature freeze now (it should happen sometime this month), so I can’t guarantee this will make it into 2.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #26174: API Endpoints - /xcomEntries/{xcom_key} cannot deserialize customized xcom backend
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #26174: API Endpoints - /xcomEntries/{xcom_key} cannot deserialize customized xcom backend
URL: https://github.com/apache/airflow/issues/26174
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] herunyu commented on issue #26174: API Endpoints - /xcomEntries/{xcom_key} cannot deserialize customized xcom backend
Posted by GitBox <gi...@apache.org>.
herunyu commented on issue #26174:
URL: https://github.com/apache/airflow/issues/26174#issuecomment-1237920416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #26174: API Endpoints - /xcomEntries/{xcom_key} cannot deserialize customized xcom backend
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #26174:
URL: https://github.com/apache/airflow/issues/26174#issuecomment-1237907880
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] herunyu commented on issue #26174: API Endpoints - /xcomEntries/{xcom_key} cannot deserialize customized xcom backend
Posted by GitBox <gi...@apache.org>.
herunyu commented on issue #26174:
URL: https://github.com/apache/airflow/issues/26174#issuecomment-1238969846
Cool. That's what we want with the API. We look forward to having this feature in 2.4. Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #26174: API Endpoints - /xcomEntries/{xcom_key} cannot deserialize customized xcom backend
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #26174:
URL: https://github.com/apache/airflow/issues/26174#issuecomment-1237918584
Can you provide the traceback emitted in the logs when the error occurs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] herunyu commented on issue #26174: API Endpoints - /xcomEntries/{xcom_key} cannot deserialize customized xcom backend
Posted by GitBox <gi...@apache.org>.
herunyu commented on issue #26174:
URL: https://github.com/apache/airflow/issues/26174#issuecomment-1237928282
There is no error. So what we did is save the json into a pickle file by overwriting the serialize_value method. And we deserialized it when we do ti.xcom_pull. So we will get the actual value we saved.
But now when we get xcom through REST API, we will get the pickle file url like in the airflow UI
And we check the API code and it seems it did not call deserialize_value to get the actual value we want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] herunyu commented on issue #26174: API Endpoints - /xcomEntries/{xcom_key} cannot deserialize customized xcom backend
Posted by GitBox <gi...@apache.org>.
herunyu commented on issue #26174:
URL: https://github.com/apache/airflow/issues/26174#issuecomment-1238974449
I see. We will do some workaround at this moment. No worries. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org