[jira] [Commented] (AIRFLOW-1583) PostgreSQL error when pushing a value bigger than 1GB

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16601379#comment-16601379 ] 

Apache Spark commented on AIRFLOW-1583:
---------------------------------------

User 'rfrenoy' has created a pull request for this issue:
https://github.com/apache/incubator-airflow/pull/2593

> PostgreSQL error when pushing a value bigger than 1GB
> -----------------------------------------------------
>
>                 Key: AIRFLOW-1583
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1583
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: xcom
>    Affects Versions: Airflow 1.8, Airflow 1.7.0
>            Reporter: Rémy Frenoy
>            Assignee: Rémy Frenoy
>            Priority: Minor
>   Original Estimate: 0.25h
>  Remaining Estimate: 0.25h
>
> When a task Operator or a PythonOperator returns a value, this value is automatically pushed to XCom. This results in a postgresql error when the value is bigger than 1GB :
> ERROR:  invalid memory alloc request size 1073741824
> A size limit could be added on the value size in the xcom-push() function in model.py. When value is higher than the limit size, a default value (maybe a string explaining that the initial value was too high to be saved) could be pushed to XCom instead of the real one, avoiding the postgresql error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)