[jira] [Commented] (AIRFLOW-4440) Avoid terminating processes using shared resources

["Jarek Potiuk (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16830207#comment-16830207 ] 

Jarek Potiuk commented on AIRFLOW-4440:
---------------------------------------

CC: [~ash]

> Avoid terminating processes using shared resources
> --------------------------------------------------
>
>                 Key: AIRFLOW-4440
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4440
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Jarek Potiuk
>            Priority: Major
>
> According to [https://docs.python.org/3.5/library/multiprocessing.html#programming-guidelines] we should avoid terminating processes when shared resources (Locks, Queues etc. are used)
> We are using terminate() in a few places in the code and we should replace it with more graceful way of stopping the processes (sending poison pills) - see the last example in [https://docs.python.org/3.5/library/multiprocessing.html|https://docs.python.org/3.5/library/multiprocessing.html#programming-guidelines] where 'STOP' is sent to shutdown the process. 
> This also prevents graceful joins in a number of cases - for example in 
> DagFileProcessorAgent.end() where we had to comment it out until this problem is solved.
>  
> We should review and remove all terminates() unless they are absolutely necessary and replae them with more graceful mechanism of shutting down.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)