You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/08 13:24:28 UTC
[GitHub] [airflow] JPFrancoia commented on issue #14063: macos python3.8 AttributeError: Can't pickle local object 'CeleryExecutor.trigger_tasks..reset_signals'
JPFrancoia commented on issue #14063:
URL: https://github.com/apache/airflow/issues/14063#issuecomment-775147658
I hit the same bug. Some info:
Airflow: 2.0.0
Platform: Mac OS (big sure, 11.2)
Python: 3.8.6 (managed with pyenv, fresh install)
Before testing, I did a reset of my DB and my venv is brand new too.
Stacktrace:
```
[2021-02-08 14:16:03,389] {scheduler_job.py:1293} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File "/Users/jpfrancoia/projects/commercial-pipelines/.venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1275, in _execute
self._run_scheduler_loop()
File "/Users/jpfrancoia/projects/commercial-pipelines/.venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1379, in _run_scheduler_loop
self.executor.heartbeat()
File "/Users/jpfrancoia/projects/commercial-pipelines/.venv/lib/python3.8/site-packages/airflow/executors/base_executor.py", line 155, in heartbeat
self.trigger_tasks(open_slots)
File "/Users/jpfrancoia/projects/commercial-pipelines/.venv/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 260, in trigger_tasks
self._process_tasks(task_tuples_to_send)
File "/Users/jpfrancoia/projects/commercial-pipelines/.venv/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 269, in _process_tasks
key_and_async_results = self._send_tasks_to_celery(task_tuples_to_send)
File "/Users/jpfrancoia/projects/commercial-pipelines/.venv/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 327, in _send_tasks_to_celery
with Pool(processes=num_processes, initializer=reset_signals) as send_pool:
File "/Users/jpfrancoia/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "/Users/jpfrancoia/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 212, in __init__
self._repopulate_pool()
File "/Users/jpfrancoia/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "/Users/jpfrancoia/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static
w.start()
File "/Users/jpfrancoia/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Users/jpfrancoia/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/Users/jpfrancoia/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/Users/jpfrancoia/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/Users/jpfrancoia/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Users/jpfrancoia/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'CeleryExecutor._send_tasks_to_celery.<locals>.reset_signals'
```
But running my dag with python 3.7.9 works totally fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org