You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/09 08:39:38 UTC
[GitHub] [airflow] lukas-at-harren opened a new issue #17497: XCom values are not deserialised correctly in GCSToBigQueryOperator
lukas-at-harren opened a new issue #17497:
URL: https://github.com/apache/airflow/issues/17497
**Apache Airflow version**: 2.1.2
**Apache Airflow Provider versions** (please include all providers that are relevant to your bug):
* apache-airflow-providers-google==5.0.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): irrelevant
**Environment**: Kubernetes, Postgresql
**What happened**:
* I have a DAG, one task outputs a list of strings as return value.
* Checking the XCom table, I see the correct value:
* Now passing that value on to `GCSToBigQueryOperator` as `source_objects` does not deserialize it correctly. The value remains a Python-list. The Operator does the following in its `__init__`:
```
super().__init__(**kwargs)
# GCS config
if src_fmt_configs is None:
src_fmt_configs = {}
if time_partitioning is None:
time_partitioning = {}
self.bucket = bucket
self.source_objects = source_objects if isinstance(source_objects, list) else [source_objects]
```
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_modules/airflow/providers/google/cloud/transfers/gcs_to_bigquery.html#GCSToBigQueryOperator
```
self.source_objects = source_objects if isinstance(source_objects, list) else [source_objects]
```
* Now for some reason `isinstance(source_objects, list)` is False, and the value gets wrapped in another list.
* Leading to errors within the Operator
```
google.api_core.exceptions.NotFound: 404 Not found: URI gs://<redacted>/['live/ingest_events_privacy.all/data/20210802/13/tmp0nova2ki.jsonl']
```
* Looking at the Rendered Template of the task, it also shows that something has gone wrong:
**What you expected to happen**:
* I expect to have the deserialised XCom value available as a list.
**How to reproduce it**:
* DAG with two stages
* stage one produces a list of strings as return value
* stage two consumes that list as a XComArg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #17497: XCom values are not deserialised correctly in GCSToBigQueryOperator
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17497:
URL: https://github.com/apache/airflow/issues/17497#issuecomment-895047957
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lukas-at-harren commented on issue #17497: XCom values are not deserialised correctly in GCSToBigQueryOperator
Posted by GitBox <gi...@apache.org>.
lukas-at-harren commented on issue #17497:
URL: https://github.com/apache/airflow/issues/17497#issuecomment-923872809
@eladkal my bad, I was thinking `XComArg` would take care of that.
For everybody else landing here, the Operator must be initialized with the option `render_template_as_native_obj=True` in order for `XComArg` to spit out Python objects instead of just a string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #17497: XCom values are not deserialised correctly in GCSToBigQueryOperator
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #17497:
URL: https://github.com/apache/airflow/issues/17497
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #17497: XCom values are not deserialised correctly in GCSToBigQueryOperator
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #17497:
URL: https://github.com/apache/airflow/issues/17497#issuecomment-923157051
This is not a bug. Xcoms are strings.
You are assuming that the deserialised process also converts the string into a Python list which is not what is happening.
If you want to render the xcom as Python native type (like list) you should follow this [doc](https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html#rendering-fields-as-native-python-objects).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org