[GitHub] [airflow] potiuk edited a comment on issue #14896: Error: 'daemonic processes are not allowed to have children', after upgradeing to airflow:2.0.1

[GitBox <gi...@apache.org>]

potiuk edited a comment on issue #14896:
URL: https://github.com/apache/airflow/issues/14896#issuecomment-912136260


   Well, I think we are talking about two different issues with "deamonic processes" and this is the source of confusion here.
   
   1) The original stack trace here (by @ivanrezic-maistra ) was about using PythonOperator running "multiprocessing"  using LocalExecutor - and this is the one which I wrote about in the `billiard` context.  The issue was about using multiprocessing inside your own customised Python operators - which I saw several people solved by simply using `import billiard as multiprocessing` in their own code.
   
   2) The second issue added in the same thread by @damon09273 and @ahazeemi  also mentioned by @kaxil  have been merged and solved and released in Airflow 2.1.2 (see milestone at https://github.com/apache/airflow/pull/16700) - but this one was about CeleryKubernetes Executor and that was not a "custom" code - it was Airflow itself that failed in this case.,
   
   Even if the error message was the same, those two issues have very different root cause, and while they were somewhat hijacked here - the error 1) still requires the `billiard` importing (in the custom code) to be solved.
   
   So answering the question @damon09273 - it's fine for you now. but if someone tries to use local executor and do multi-processing in their custom code within Python Operator (or writes a custom operator) then `biliard` instead of `multiprocessing` used in the custom code should solve the problem. No need to change anything in celery executor.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org