You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/21 15:13:25 UTC
[GitHub] [airflow] rochamc opened a new issue #13812: After migrating from version 1.10.12 to 2.0.0, SimpleHttpOperator produces the error "Data too long for column 'value' at row 1" when pushing to XCOM
rochamc opened a new issue #13812:
URL: https://github.com/apache/airflow/issues/13812
**Apache Airflow version**: 2.0.0
**Environment**:
- **Cloud provider or hardware configuration**: On-premise private cloud, appliance with 2vCPU and 8GB RAM (not production server)
- **OS** (e.g. from /etc/os-release): Debian GNU/Linux 9 (stretch)
- **Kernel** (e.g. `uname -a`): Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) x86_64 GNU/Linux
- **Install tools**: manual installation on host OS (no docker/kubernetes)
**What happened**:
sqlalchemy.exc.DataError: (_mysql_exceptions.DataError) (1406, "Data too long for column 'value' at row 1")
[SQL: INSERT INTO xcom (`key`, value, timestamp, execution_date, task_id, dag_id) VALUES (%s, %s, %s, %s, %s, %s)]
**What you expected to happen**:
We are migrating all of our DAGs to the new Apache Airflow 2.0.0 version (from the 1.10.12 version). One of the DAGs uses the SimpleHTTPOperator to fetch JSON data from an API and saves the JSON into the XCOM so that the next task can fetch, process and save it. For the migration, we had to change the SimpleHTTPOperator parameter from "xcom_push=true" to "do_xcom_push=true". We expected that after doing the parameter change, the DAG would run like it does in the production server (in the older 1.10.12 version). We made the same change to other DAGs and it worked properly, but when we do that in the 2.0.0 we get the error "Data too long for column 'value' at row 1". Apparently in this new version (2.0.0), the XCOM max column size is lower than the previous version we are using.
**How to reproduce it**:
Fetch an API using the SimpleHTTPOperator with the parameter do_xcom_push to true with a response longer than the xcom column size.
`get_resource = SimpleHttpOperator(
task_id='get_example',
method='GET',
endpoint='/endpoint/resource/',
http_conn_id = 'connection_example',
trigger_rule="all_done",
do_xcom_push=True,
dag=dag
)`
Also, Before migrating those SimpleHttpOperator tasks to python tasks, we would like to know if there is any other way of passing the SimpleHTTPOperator response without the XCOM usage?
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on issue #13812: After migrating from version 1.10.12 to 2.0.0, SimpleHttpOperator produces the error "Data too long for column 'value' at row 1" when pushing to XCOM
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on issue #13812:
URL: https://github.com/apache/airflow/issues/13812#issuecomment-807355015
@rochamc What database are you using?
Since you are using MySQL, I suspect you are hitting the database column size in MySQL. Airflow uses SQLAlchecmy BLOB on MySQL which limits the xcom.value to 65,535 bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #13812: After migrating from version 1.10.12 to 2.0.0, SimpleHttpOperator produces the error "Data too long for column 'value' at row 1" when pushing to XCOM
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #13812:
URL: https://github.com/apache/airflow/issues/13812
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka edited a comment on issue #13812: After migrating from version 1.10.12 to 2.0.0, SimpleHttpOperator produces the error "Data too long for column 'value' at row 1" when pushing to XCOM
Posted by GitBox <gi...@apache.org>.
vikramkoka edited a comment on issue #13812:
URL: https://github.com/apache/airflow/issues/13812#issuecomment-807355015
@rochamc What database are you using?
Since you are using MySQL, I suspect you are hitting the database column size in MySQL. Airflow uses SQLAlchecmy BLOB on MySQL which limits the xcom.value to 65,535 bytes
A similar issue was reported earlier https://github.com/apache/airflow/issues/9299
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13812: After migrating from version 1.10.12 to 2.0.0, SimpleHttpOperator produces the error "Data too long for column 'value' at row 1" when pushing to XCOM
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13812:
URL: https://github.com/apache/airflow/issues/13812#issuecomment-764711568
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13812: After migrating from version 1.10.12 to 2.0.0, SimpleHttpOperator produces the error "Data too long for column 'value' at row 1" when pushing to XCOM
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13812:
URL: https://github.com/apache/airflow/issues/13812#issuecomment-764711568
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #13812: After migrating from version 1.10.12 to 2.0.0, SimpleHttpOperator produces the error "Data too long for column 'value' at row 1" when pushing to XCOM
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #13812:
URL: https://github.com/apache/airflow/issues/13812#issuecomment-952424314
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #13812: After migrating from version 1.10.12 to 2.0.0, SimpleHttpOperator produces the error "Data too long for column 'value' at row 1" when pushing to XCOM
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #13812:
URL: https://github.com/apache/airflow/issues/13812#issuecomment-960307994
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13812: After migrating from version 1.10.12 to 2.0.0, SimpleHttpOperator produces the error "Data too long for column 'value' at row 1" when pushing to XCOM
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13812:
URL: https://github.com/apache/airflow/issues/13812#issuecomment-764960063
https://github.com/apache/airflow/blob/master/UPDATING.md#the-default-value-for-core-enable_xcom_pickling-has-been-changed-to-false
You can turn this back to `True`:
```ini
[core]
enable_xcom_pickling = True
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #13812: After migrating from version 1.10.12 to 2.0.0, SimpleHttpOperator produces the error "Data too long for column 'value' at row 1" when pushing to XCOM
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #13812:
URL: https://github.com/apache/airflow/issues/13812
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rochamc commented on issue #13812: After migrating from version 1.10.12 to 2.0.0, SimpleHttpOperator produces the error "Data too long for column 'value' at row 1" when pushing to XCOM
Posted by GitBox <gi...@apache.org>.
rochamc commented on issue #13812:
URL: https://github.com/apache/airflow/issues/13812#issuecomment-767722688
> https://github.com/apache/airflow/blob/master/UPDATING.md#the-default-value-for-core-enable_xcom_pickling-has-been-changed-to-false
> You can turn this back to `True`:
>
> ```ini
> [core]
> enable_xcom_pickling = True
> ```
Thank you for your reply. We've made this change but the error continues to happen.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13812: After migrating from version 1.10.12 to 2.0.0, SimpleHttpOperator produces the error "Data too long for column 'value' at row 1" when pushing to XCOM
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13812:
URL: https://github.com/apache/airflow/issues/13812#issuecomment-764960063
https://github.com/apache/airflow/blob/master/UPDATING.md#the-default-value-for-core-enable_xcom_pickling-has-been-changed-to-false
You can turn this back to `True`:
```ini
[core]
enable_xcom_pickling = True
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #13812: After migrating from version 1.10.12 to 2.0.0, SimpleHttpOperator produces the error "Data too long for column 'value' at row 1" when pushing to XCOM
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #13812:
URL: https://github.com/apache/airflow/issues/13812#issuecomment-960307994
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org