[GitHub] [airflow] potiuk commented on issue #14896: Error: 'daemonic processes are not allowed to have children', after upgradeing to airflow:2.0.1

[GitBox <gi...@apache.org>]

potiuk commented on issue #14896:
URL: https://github.com/apache/airflow/issues/14896#issuecomment-908516288


   There is a known solution to this problem. Please use `billiard` https://pypi.org/project/billiard/  instead of `multiprocessing`. It is a fork of multiprocessing internally used in Celery and it was specifically forked by Celery developers to handle similar problems and limitations of the original multiprocessing library. Since it is used by Celery - you should have it installed already,  and it is 1-1 replacement to multiprocessing (literally you can do `import billiard as multiprocessing`).
   
   I saw already quite a number of confirmations that it works for people with similar problems, and I am just considering making an entry in Best Practices of Airflow to cover that piece (@ashb @kaxil - you might also be interested in that  :). This is an interesting finding I had by following up a number of issues and trying to help people. 
   
   @pmlewis, @damon09273 @ahazeemi @ivanrezic-maistra  - if you can also confirm that this solution works, I'd be more confident in adding best practice for that one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org