You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Len Frodgers (JIRA)" <ji...@apache.org> on 2017/12/08 15:40:00 UTC

[jira] [Created] (AIRFLOW-1898) Large XComs are not supported and fail silently

Len Frodgers created AIRFLOW-1898:
-------------------------------------

             Summary: Large XComs are not supported and fail silently
                 Key: AIRFLOW-1898
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1898
             Project: Apache Airflow
          Issue Type: Bug
          Components: xcom
    Affects Versions: Airflow 1.8
         Environment: MySQL
            Reporter: Len Frodgers


I am using Airlfow backed by MySQL and having problems with large XComs (> 64 KB). Xcom uses PickleType which is backed by BLOB on MySQL.

Unfortunately, MySQL by default truncates anything longer than BLOB (64 KB) when saving, so when unpickling such XComs, they are corrupt and an EOFError is raised.

Two things we need:

1) Validation when saving the XCom that it is not too big
2) Use MEDIUMBLOB as the underlying data type for the XCOM column on MySQL so large XComs can be stored (supports  up to 12 MB i think)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)