[GitHub] [airflow] toughrogrammer commented on issue #11451: AttributeError: Can't pickle local object 'DaskExecutor.execute_async..airflow_run'

[GitBox <gi...@apache.org>]

toughrogrammer commented on issue #11451:
URL: https://github.com/apache/airflow/issues/11451#issuecomment-753307462


   I got similar error when I upgraded to 2.0.0
   
   ```
   ERROR - Exception when executing SchedulerJob._run_scheduler_loop
   Traceback (most recent call last):
     File "/Users/loki/Library/Caches/pypoetry/virtualenvs/airflow-RLSyOqyF-py3.8/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1275, in _execute
       self._run_scheduler_loop()
     File "/Users/loki/Library/Caches/pypoetry/virtualenvs/airflow-RLSyOqyF-py3.8/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1379, in _run_scheduler_loop
       self.executor.heartbeat()
     File "/Users/loki/Library/Caches/pypoetry/virtualenvs/airflow-RLSyOqyF-py3.8/lib/python3.8/site-packages/airflow/executors/base_executor.py", line 155, in heartbeat
       self.trigger_tasks(open_slots)
     File "/Users/loki/Library/Caches/pypoetry/virtualenvs/airflow-RLSyOqyF-py3.8/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 260, in trigger_tasks
       self._process_tasks(task_tuples_to_send)
     File "/Users/loki/Library/Caches/pypoetry/virtualenvs/airflow-RLSyOqyF-py3.8/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 269, in _process_tasks
       key_and_async_results = self._send_tasks_to_celery(task_tuples_to_send)
     File "/Users/loki/Library/Caches/pypoetry/virtualenvs/airflow-RLSyOqyF-py3.8/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 327, in _send_tasks_to_celery
       with Pool(processes=num_processes, initializer=reset_signals) as send_pool:
     File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/context.py", line 119, in Pool
       return Pool(processes, initializer, initargs, maxtasksperchild,
     File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/pool.py", line 212, in __init__
       self._repopulate_pool()
     File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
       return self._repopulate_pool_static(self._ctx, self.Process,
     File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static
       w.start()
     File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/process.py", line 121, in start
       self._popen = self._Popen(self)
     File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
       return Popen(process_obj)
     File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
       super().__init__(process_obj)
     File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
       self._launch(process_obj)
     File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
       reduction.dump(process_obj, fp)
     File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
       ForkingPickler(file, protocol).dump(obj)
   AttributeError: Can't pickle local object 'CeleryExecutor._send_tasks_to_celery.<locals>.reset_signals'
   
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org