You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Jeremiah Lowin (JIRA)" <ji...@apache.org> on 2016/05/06 16:13:12 UTC
[jira] [Closed] (AIRFLOW-61) Corner case with deadlocked
multiprocessing queues
[ https://issues.apache.org/jira/browse/AIRFLOW-61?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremiah Lowin closed AIRFLOW-61.
---------------------------------
Resolution: Fixed
Assignee: Jeremiah Lowin
Merged in https://github.com/apache/incubator-airflow/pull/1473
[~criccomini]: Leaving the "fix version" at 1.7.1 for now
> Corner case with deadlocked multiprocessing queues
> --------------------------------------------------
>
> Key: AIRFLOW-61
> URL: https://issues.apache.org/jira/browse/AIRFLOW-61
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: Airflow 1.7.0
> Reporter: Jeremiah Lowin
> Assignee: Jeremiah Lowin
> Fix For: Airflow 1.7.1
>
>
> I've never experienced this before, but it came up while testing this morning. Joining a process that puts items in a queue can apparently lead to deadlocks. This happens in the multiprocessing-enabled scheduler.
> See for example: https://docs.python.org/3/library/multiprocessing.html#all-start-methods ("Joining processes that use queues") and related issues:
> http://stackoverflow.com/questions/31665328/python-3-multiprocessing-queue-deadlock-when-calling-join-before-the-queue-is-em
> http://stackoverflow.com/questions/31708646/process-join-and-queue-dont-work-with-large-numbers
> http://stackoverflow.com/questions/19071529/python-multiprocessing-125-list-never-finishes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)