[jira] [Assigned] (AIRFLOW-62) XCom push not working reliably

["Jeremiah Lowin (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-62?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeremiah Lowin reassigned AIRFLOW-62:
-------------------------------------

    Assignee: Jeremiah Lowin

> XCom push not working reliably
> ------------------------------
>
>                 Key: AIRFLOW-62
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-62
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: db, operators
>    Affects Versions: Airflow 1.7.0
>         Environment: Postgres backed Airflow running with Celery inside of the puckel Docker setup.
>            Reporter: Alex Papanicolaou
>            Assignee: Jeremiah Lowin
>
> I have a DAG that polls for activity in various data streams from a database and then uploads the activity statuses to a table.  Each of the polling tasks are python operators that once they get the polling result, return a dict as an XCom push.  The dict contains two entries which are strings, one which is a bool, and one which is a datetime object.  There is a final task that pulls all the results and uploads the collective statuses to a table.  I chose this pattern since I figured it might be better to do one collective write operation on all the results.
> Before I moved ahead to the github master branch I was using 1.7.0 from PyPI and this worked fine.  Now that I am on the github master branch, I find that the XCom pushing is unreliable.  The returned values in the logs show up correctly but when doing the XCom pull, I get None for some of the returned values.  Investigating the XCom result in the Webserver also shows nothing there.  But if I rerun a task where the XCom failed, the push works and the XCom result is as it should be.
> Nothing appears to have changed in the codebase so I am at a loss.  Perhaps it really wasn't working before?  How would the backing postgres handle these simultaneous writes?  I can't imagine that would be a problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)