You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Martijn Vermaat (JIRA)" <ji...@apache.org> on 2016/07/11 22:17:11 UTC
[jira] [Commented] (AIRFLOW-62) XCom push not working reliably
[ https://issues.apache.org/jira/browse/AIRFLOW-62?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371759#comment-15371759 ]
Martijn Vermaat commented on AIRFLOW-62:
----------------------------------------
I am experiencing the same issue. First task is returning a tuple of strings (I'm not pushing explicitely), second task uses it with xcom_pull. It works most of the time (and always in my dev setup), but every now and then the xcom_pull result is missing. Even though from the logs I can see the first task ran succesfully and returned the two strings.
Using Celery with RabbitMQ broker and PostgreSQL backend.
airflow==1.7.1.3
psycopg2==2.6.1
celery==3.1.23
> XCom push not working reliably
> ------------------------------
>
> Key: AIRFLOW-62
> URL: https://issues.apache.org/jira/browse/AIRFLOW-62
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
> Affects Versions: Airflow 1.7.0
> Environment: Postgres backed Airflow running with Celery inside of the puckel Docker setup.
> Reporter: Alex Papanicolaou
> Assignee: Jeremiah Lowin
> Fix For: Airflow 1.7.0
>
>
> I have a DAG that polls for activity in various data streams from a database and then uploads the activity statuses to a table. Each of the polling tasks are python operators that once they get the polling result, return a dict as an XCom push. The dict contains two entries which are strings, one which is a bool, and one which is a datetime object. There is a final task that pulls all the results and uploads the collective statuses to a table. I chose this pattern since I figured it might be better to do one collective write operation on all the results.
> Before I moved ahead to the github master branch I was using 1.7.0 from PyPI and this worked fine. Now that I am on the github master branch, I find that the XCom pushing is unreliable. The returned values in the logs show up correctly but when doing the XCom pull, I get None for some of the returned values. Investigating the XCom result in the Webserver also shows nothing there. But if I rerun a task where the XCom failed, the push works and the XCom result is as it should be.
> Nothing appears to have changed in the codebase so I am at a loss. Perhaps it really wasn't working before? How would the backing postgres handle these simultaneous writes? I can't imagine that would be a problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)