You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/12 03:25:39 UTC
[GitHub] [airflow] lafrinte opened a new issue #11451: AttributeError: Can't pickle local object 'DaskExecutor.execute_async..airflow_run'
lafrinte opened a new issue #11451:
URL: https://github.com/apache/airflow/issues/11451
1. version info:
* apache-airflow: 1.10.12
* os: centos7.4
* python: 3.7.2
* apache-airflow executor: dask
2. error msg:
```
-bash-4.2$ airflow scheduler
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2020-07-16 18:48:30,885] {__init__.py:51} INFO - Using executor DaskExecutor
[2020-07-16 18:48:30,890] {scheduler_job.py:1346} INFO - Starting the scheduler
[2020-07-16 18:48:30,890] {scheduler_job.py:1354} INFO - Running execute loop for -1 seconds
[2020-07-16 18:48:30,890] {scheduler_job.py:1355} INFO - Processing each file at most -1 times
[2020-07-16 18:48:30,890] {scheduler_job.py:1358} INFO - Searching for files in /ansdep/opsapi/dags
[2020-07-16 18:48:30,890] {scheduler_job.py:1360} INFO - There are 0 files in /ansdep/opsapi/dags
[2020-07-16 18:48:30,898] {scheduler_job.py:1411} INFO - Resetting orphaned tasks for active dag runs
[2020-07-16 18:48:30,912] {dag_processing.py:556} INFO - Launched DagFileProcessorManager with pid: 70385
[2020-07-16 18:48:30,916] {settings.py:54} INFO - Configured default timezone <Timezone [Asia/Shanghai]>
[2020-07-16 18:49:22,984] {scheduler_job.py:951} INFO - 1 tasks up for execution:
<TaskInstance: bc081861-e948-4e59-92fa-fc14f4330eb2.first_for_flush_state 2020-07-16 10:49:00+00:00 [scheduled]>
[2020-07-16 18:49:22,990] {scheduler_job.py:982} INFO - Figuring out tasks to run in Pool(name=default_pool) with 128 open slots and 1 task instances ready to be queued
[2020-07-16 18:49:22,990] {scheduler_job.py:1010} INFO - DAG bc081861-e948-4e59-92fa-fc14f4330eb2 has 0/512 running and queued tasks
[2020-07-16 18:49:22,994] {scheduler_job.py:1060} INFO - Setting the following tasks to queued state:
<TaskInstance: bc081861-e948-4e59-92fa-fc14f4330eb2.first_for_flush_state 2020-07-16 10:49:00+00:00 [scheduled]>
[2020-07-16 18:49:23,000] {scheduler_job.py:1134} INFO - Setting the following 1 tasks to queued state:
<TaskInstance: bc081861-e948-4e59-92fa-fc14f4330eb2.first_for_flush_state 2020-07-16 10:49:00+00:00 [queued]>
[2020-07-16 18:49:23,000] {scheduler_job.py:1170} INFO - Sending ('bc081861-e948-4e59-92fa-fc14f4330eb2', 'first_for_flush_state', datetime.datetime(2020, 7, 16, 10, 49, tzinfo=<TimezoneInfo [UTC, GMT, +00:00:00, STD]>), 1) to executor with priority 4 and queue default
[2020-07-16 18:49:23,000] {base_executor.py:58} INFO - Adding to queue: ['airflow', 'run', 'bc081861-e948-4e59-92fa-fc14f4330eb2', 'first_for_flush_state', '2020-07-16T10:49:00+00:00', '--local', '--pool', 'default_pool', '-sd', '/ansdep/opsapi/dags/bc081861-e948-4e59-92fa-fc14f4330eb2.py']
/ansdep/python3/lib/python3.7/site-packages/airflow/executors/dask_executor.py:63: UserWarning: DaskExecutor does not support queues. All tasks will be run in the same cluster
'DaskExecutor does not support queues. '
distributed.protocol.pickle - INFO - Failed to serialize <function DaskExecutor.execute_async.<locals>.airflow_run at 0x7f748ce808c8>. Exception: Cell is empty
[2020-07-16 18:49:23,003] {scheduler_job.py:1384} ERROR - Exception when executing execute_helper
Traceback (most recent call last):
File "/ansdep/python3/lib/python3.7/site-packages/distributed/worker.py", line 3323, in dumps_function
result = cache_dumps[func]
File "/ansdep/python3/lib/python3.7/site-packages/distributed/utils.py", line 1549, in __getitem__
value = super().__getitem__(key)
File "/ansdep/python3/lib/python3.7/collections/__init__.py", line 1025, in __getitem__
raise KeyError(key)
KeyError: <function DaskExecutor.execute_async.<locals>.airflow_run at 0x7f748ce808c8>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/ansdep/python3/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 41, in dumps
result = pickle.dumps(x, **dump_kwargs)
AttributeError: Can't pickle local object 'DaskExecutor.execute_async.<locals>.airflow_run'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/ansdep/python3/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1382, in _execute
self._execute_helper()
File "/ansdep/python3/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1443, in _execute_helper
if not self._validate_and_run_task_instances(simple_dag_bag=simple_dag_bag):
File "/ansdep/python3/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1505, in _validate_and_run_task_instances
self.executor.heartbeat()
File "/ansdep/python3/lib/python3.7/site-packages/airflow/executors/base_executor.py", line 130, in heartbeat
self.trigger_tasks(open_slots)
File "/ansdep/python3/lib/python3.7/site-packages/airflow/executors/base_executor.py", line 154, in trigger_tasks
executor_config=simple_ti.executor_config)
File "/ansdep/python3/lib/python3.7/site-packages/airflow/executors/dask_executor.py", line 70, in execute_async
future = self.client.submit(airflow_run, pure=False)
File "/ansdep/python3/lib/python3.7/site-packages/distributed/client.py", line 1579, in submit
actors=actor,
File "/ansdep/python3/lib/python3.7/site-packages/distributed/client.py", line 2598, in _graph_to_futures
"tasks": valmap(dumps_task, dsk3),
File "/ansdep/python3/lib/python3.7/site-packages/toolz/dicttoolz.py", line 83, in valmap
rv.update(zip(iterkeys(d), map(func, itervalues(d))))
File "/ansdep/python3/lib/python3.7/site-packages/distributed/worker.py", line 3361, in dumps_task
return {"function": dumps_function(task[0]), "args": warn_dumps(task[1:])}
File "/ansdep/python3/lib/python3.7/site-packages/distributed/worker.py", line 3325, in dumps_function
result = pickle.dumps(func)
File "/ansdep/python3/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 52, in dumps
result = cloudpickle.dumps(x, **dump_kwargs)
File "/ansdep/python3/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 101, in dumps
cp.dump(obj)
File "/ansdep/python3/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 540, in dump
return Pickler.dump(self, obj)
File "/ansdep/python3/lib/python3.7/pickle.py", line 437, in dump
self.save(obj)
File "/ansdep/python3/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/ansdep/python3/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 722, in save_function
*self._dynamic_function_reduce(obj), obj=obj
File "/ansdep/python3/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 659, in _save_reduce_pickle5
dictitems=dictitems, obj=obj
File "/ansdep/python3/lib/python3.7/pickle.py", line 638, in save_reduce
save(args)
File "/ansdep/python3/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/ansdep/python3/lib/python3.7/pickle.py", line 786, in save_tuple
save(element)
File "/ansdep/python3/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/ansdep/python3/lib/python3.7/pickle.py", line 771, in save_tuple
save(element)
File "/ansdep/python3/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/ansdep/python3/lib/python3.7/site-packages/dill/_dill.py", line 1169, in save_cell
f = obj.cell_contents
ValueError: Cell is empty
[2020-07-16 18:49:23,014] {helpers.py:325} INFO - Sending Signals.SIGTERM to GPID 70385
[2020-07-16 18:49:23,041] {helpers.py:291} INFO - Process psutil.Process(pid=70385, status='terminated') (70385) terminated with exit code 0
[2020-07-16 18:49:23,041] {helpers.py:291} INFO - Process psutil.Process(pid=70575, status='terminated') (70575) terminated with exit code None
[2020-07-16 18:49:23,042] {scheduler_job.py:1387} INFO - Exited execute loop
```
the pickle error occured in distributed(dask) worker.py
* distributed.worker.dumps_function
```
def dumps_function(func):
""" Dump a function to bytes, cache functions """
try:
with _cache_lock:
result = cache_dumps[func]
except KeyError:
result = pickle.dumps(func) < -- occure err here
if len(result) < 100000:
with _cache_lock:
cache_dumps[func] = result
except TypeError: # Unhashable function
result = pickle.dumps(func)
return result
```
the argument `func` pass to dumps_function is getting from airflow.executor.dask_executor.DaskExecutor.execute_async
```
def execute_async(self, key, command, queue=None, executor_config=None):
if queue is not None:
warnings.warn(
'DaskExecutor does not support queues. '
'All tasks will be run in the same cluster'
)
if command[0:2] != ["airflow", "run"]:
raise ValueError('The command must start with ["airflow", "run"].')
def airflow_run():
return subprocess.check_call(command, close_fds=True)
future = self.client.submit(airflow_run, pure=False) < -- here to pass the airflow_run func obj to dumps_function
self.futures[future] = key
```
from some reason, i try to mock for pickle a func in func in ipython
the class A is like the dumps_function which is waiting a func object to pickle with.
the class B is like the executor
```
In [1]: import pickle, subprocess
In [2]: class A:
...: def __init__(self, func):
...: self.func = func
...: def dumps(self):
...: print(self.func.__name__, ' ', self.func.__module__)
...: return pickle.dumps(self.func)
...:
In [3]: class B:
...: def __init__(self):
...: self.a = None
...: def a_method(self, cmd):
...: def run_command():
...: return subprocess.check_call(cmd, close_fds=True)
...: self.a = A(run_command)
...: return self.a.dumps()
...:
...:
In [4]: b = B()
In [5]: b.a_method('sleep 100')
run_command __main__
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-18-dce0583e3bfc> in <module>
----> 1 b.a_method('sleep 100')
<ipython-input-16-7100883a1a96> in a_method(self, cmd)
6 return subprocess.check_call(cmd, close_fds=True)
7 self.a = A(run_command)
----> 8 return self.a.dumps()
9
10
<ipython-input-15-073409fdb9f1> in dumps(self)
4 def dumps(self):
5 print(self.func.__name__, ' ', self.func.__module__)
----> 6 return pickle.dumps(self.func)
7
AttributeError: Can't pickle local object 'B.a_method.<locals>.run_command'
```
i think the airflow_run func define in execute_async may case the pickle error.
so, i change the code and scheduler work well:
```
def airflow_run(command):
return subprocess.check_call(command, close_fds=True)
class DaskExecutor(BaseExecutor):
...
def execute_async(self, key, command, queue=None, executor_config=None):
if queue is not None:
warnings.warn(
'DaskExecutor does not support queues. '
'All tasks will be run in the same cluster'
)
if command[0:2] != ["airflow", "run"]:
raise ValueError('The command must start with ["airflow", "run"].')
future = self.client.submit(airflow_run, command, pure=False)
self.futures[future] = key
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aa1371 commented on issue #11451: AttributeError: Can't pickle local object 'DaskExecutor.execute_async..airflow_run'
Posted by GitBox <gi...@apache.org>.
aa1371 commented on issue #11451:
URL: https://github.com/apache/airflow/issues/11451#issuecomment-939201903
@esc's fix works for me. However, any reason we need to use this closure?
https://github.com/apache/airflow/blob/1df9a512c284f2585ce01bb77ac91d550f6ccaa3/airflow/executors/dask_executor.py#L82
Getting rid of the closure and submitting the check_call directly to client.submit seems to work and doesn't run into the serialization issue:
Current:
```
def airflow_run():
return subprocess.check_call(command, close_fds=True)
...
future = self.client.submit(airflow_run, pure=False, resources=resources)
```
Proposed:
```
future = self.client.submit(subprocess.check_call, command, pure=False, resources=resources, close_fds=True)
```
cc @jedcunningham
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xawyong commented on issue #11451: AttributeError: Can't pickle local object 'DaskExecutor.execute_async..airflow_run'
Posted by GitBox <gi...@apache.org>.
xawyong commented on issue #11451:
URL: https://github.com/apache/airflow/issues/11451#issuecomment-755929744
use 2.0.0, after changed the code as @lafrinte proposed, got below error.
```
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-01-07 02:01:14,768] {scheduler_job.py:1241} INFO - Starting the scheduler
[2021-01-07 02:01:14,769] {scheduler_job.py:1246} INFO - Processing each file at most -1 times
/usr/local/lib/python3.7/site-packages/distributed/client.py:1128 VersionMismatchWarning: Mismatched versions found
+-------------+-----------+-----------------------+-----------------------+
| Package | client | scheduler | workers |
+-------------+-----------+-----------------------+-----------------------+
| distributed | 2020.12.0 | 2020.12.0+11.g4386b75 | 2020.12.0+11.g4386b75 |
+-------------+-----------+-----------------------+-----------------------+
[2021-01-07 02:01:14,838] {dag_processing.py:250} INFO - Launched DagFileProcessorManager with pid: 18779
[2021-01-07 02:01:14,841] {scheduler_job.py:1751} INFO - Resetting orphaned tasks for active dag runs
[2021-01-07 02:01:14,849] {settings.py:52} INFO - Configured default timezone Timezone('UTC')
[2021-01-07 02:02:04,469] {scheduler_job.py:938} INFO - 4 tasks up for execution:
<TaskInstance: example_bash_operator.runme_0 2021-01-07 07:02:03.530249+00:00 [scheduled]>
<TaskInstance: example_bash_operator.runme_1 2021-01-07 07:02:03.530249+00:00 [scheduled]>
<TaskInstance: example_bash_operator.runme_2 2021-01-07 07:02:03.530249+00:00 [scheduled]>
<TaskInstance: example_bash_operator.also_run_this 2021-01-07 07:02:03.530249+00:00 [scheduled]>
[2021-01-07 02:02:04,470] {scheduler_job.py:972} INFO - Figuring out tasks to run in Pool(name=default_pool) with 128 open slots and 4 task instances ready to be queued
[2021-01-07 02:02:04,470] {scheduler_job.py:999} INFO - DAG example_bash_operator has 0/16 running and queued tasks
[2021-01-07 02:02:04,471] {scheduler_job.py:999} INFO - DAG example_bash_operator has 1/16 running and queued tasks
[2021-01-07 02:02:04,471] {scheduler_job.py:999} INFO - DAG example_bash_operator has 2/16 running and queued tasks
[2021-01-07 02:02:04,471] {scheduler_job.py:999} INFO - DAG example_bash_operator has 3/16 running and queued tasks
[2021-01-07 02:02:04,471] {scheduler_job.py:1060} INFO - Setting the following tasks to queued state:
<TaskInstance: example_bash_operator.runme_0 2021-01-07 07:02:03.530249+00:00 [scheduled]>
<TaskInstance: example_bash_operator.runme_1 2021-01-07 07:02:03.530249+00:00 [scheduled]>
<TaskInstance: example_bash_operator.runme_2 2021-01-07 07:02:03.530249+00:00 [scheduled]>
<TaskInstance: example_bash_operator.also_run_this 2021-01-07 07:02:03.530249+00:00 [scheduled]>
[2021-01-07 02:02:04,473] {scheduler_job.py:1102} INFO - Sending TaskInstanceKey(dag_id='example_bash_operator', task_id='runme_0', execution_date=datetime.datetime(2021, 1, 7, 7, 2, 3, 530249, tzinfo=Timezone('UTC')), try_number=1) to executor with priority 3 and queue default
[2021-01-07 02:02:04,473] {base_executor.py:79} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-01-07T07:02:03.530249+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_bash_operator.py']
[2021-01-07 02:02:04,473] {scheduler_job.py:1102} INFO - Sending TaskInstanceKey(dag_id='example_bash_operator', task_id='runme_1', execution_date=datetime.datetime(2021, 1, 7, 7, 2, 3, 530249, tzinfo=Timezone('UTC')), try_number=1) to executor with priority 3 and queue default
[2021-01-07 02:02:04,474] {base_executor.py:79} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-01-07T07:02:03.530249+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_bash_operator.py']
[2021-01-07 02:02:04,474] {scheduler_job.py:1102} INFO - Sending TaskInstanceKey(dag_id='example_bash_operator', task_id='runme_2', execution_date=datetime.datetime(2021, 1, 7, 7, 2, 3, 530249, tzinfo=Timezone('UTC')), try_number=1) to executor with priority 3 and queue default
[2021-01-07 02:02:04,474] {base_executor.py:79} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-01-07T07:02:03.530249+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_bash_operator.py']
[2021-01-07 02:02:04,474] {scheduler_job.py:1102} INFO - Sending TaskInstanceKey(dag_id='example_bash_operator', task_id='also_run_this', execution_date=datetime.datetime(2021, 1, 7, 7, 2, 3, 530249, tzinfo=Timezone('UTC')), try_number=1) to executor with priority 2 and queue default
[2021-01-07 02:02:04,474] {base_executor.py:79} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-01-07T07:02:03.530249+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_bash_operator.py']
[2021-01-07 02:02:06,343] {dask_executor.py:94} ERROR - Failed to execute task: CalledProcessError(120, ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-01-07T07:02:03.530249+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_bash_operator.py'])
[2021-01-07 02:02:06,344] {dask_executor.py:94} ERROR - Failed to execute task: CalledProcessError(120, ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-01-07T07:02:03.530249+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_bash_operator.py'])
[2021-01-07 02:02:06,345] {dask_executor.py:94} ERROR - Failed to execute task: CalledProcessError(120, ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-01-07T07:02:03.530249+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_bash_operator.py'])
[2021-01-07 02:02:06,345] {scheduler_job.py:1200} INFO - Executor reports execution of example_bash_operator.runme_0 execution_date=2021-01-07 07:02:03.530249+00:00 exited with status failed for try_number 1
[2021-01-07 02:02:06,345] {scheduler_job.py:1200} INFO - Executor reports execution of example_bash_operator.runme_2 execution_date=2021-01-07 07:02:03.530249+00:00 exited with status failed for try_number 1
[2021-01-07 02:02:06,345] {scheduler_job.py:1200} INFO - Executor reports execution of example_bash_operator.also_run_this execution_date=2021-01-07 07:02:03.530249+00:00 exited with status failed for try_number 1
[2021-01-07 02:02:06,350] {scheduler_job.py:1229} ERROR - Executor reports task instance <TaskInstance: example_bash_operator.runme_0 2021-01-07 07:02:03.530249+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
[2021-01-07 02:02:06,351] {scheduler_job.py:1229} ERROR - Executor reports task instance <TaskInstance: example_bash_operator.runme_2 2021-01-07 07:02:03.530249+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
[2021-01-07 02:02:06,352] {scheduler_job.py:1229} ERROR - Executor reports task instance <TaskInstance: example_bash_operator.also_run_this 2021-01-07 07:02:03.530249+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
[2021-01-07 02:02:07,410] {dask_executor.py:94} ERROR - Failed to execute task: CalledProcessError(120, ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-01-07T07:02:03.530249+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_bash_operator.py'])
[2021-01-07 02:02:07,410] {scheduler_job.py:1200} INFO - Executor reports execution of example_bash_operator.runme_1 execution_date=2021-01-07 07:02:03.530249+00:00 exited with status failed for try_number 1
[2021-01-07 02:02:07,415] {scheduler_job.py:1229} ERROR - Executor reports task instance <TaskInstance: example_bash_operator.runme_1 2021-01-07 07:02:03.530249+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
Process ForkProcess-1:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 365, in _run_processor_manager
processor_manager.start()
File "/usr/local/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 596, in start
return self._run_parsing_loop()
File "/usr/local/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 659, in _run_parsing_loop
self._processors.pop(processor.file_path)
KeyError: '/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_bash_operator.py'
[2021-01-07 02:02:08,445] {dagrun.py:429} ERROR - Marking run <DagRun example_bash_operator @ 2021-01-07 07:02:03.530249+00:00: manual__2021-01-07T07:02:03.530249+00:00, externally triggered: True> failed
[2021-01-07 02:02:08,470] {dag_processing.py:399} WARNING - DagFileProcessorManager (PID=18779) exited with exit code 1 - re-launching
[2021-01-07 02:02:08,475] {dag_processing.py:250} INFO - Launched DagFileProcessorManager with pid: 21398
[2021-01-07 02:02:08,483] {settings.py:52} INFO - Configured default timezone Timezone('UTC')
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] griffinqiu commented on issue #11451: AttributeError: Can't pickle local object 'DaskExecutor.execute_async..airflow_run'
Posted by GitBox <gi...@apache.org>.
griffinqiu commented on issue #11451:
URL: https://github.com/apache/airflow/issues/11451#issuecomment-762571684
Same issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lafrinte commented on issue #11451: AttributeError: Can't pickle local object 'DaskExecutor.execute_async..airflow_run'
Posted by GitBox <gi...@apache.org>.
lafrinte commented on issue #11451:
URL: https://github.com/apache/airflow/issues/11451#issuecomment-713751469
> Thanks for opening your first issue here! Be sure to follow the issue template!
i modify the description and follow the template. wish your replay
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aa1371 removed a comment on issue #11451: AttributeError: Can't pickle local object 'DaskExecutor.execute_async..airflow_run'
Posted by GitBox <gi...@apache.org>.
aa1371 removed a comment on issue #11451:
URL: https://github.com/apache/airflow/issues/11451#issuecomment-939201903
@esc's fix works for me. However, any reason we need to use this closure?
https://github.com/apache/airflow/blob/1df9a512c284f2585ce01bb77ac91d550f6ccaa3/airflow/executors/dask_executor.py#L82
Getting rid of the closure and submitting the check_call directly to client.submit seems to work and doesn't run into the serialization issue:
Current:
```
def airflow_run():
return subprocess.check_call(command, close_fds=True)
...
future = self.client.submit(airflow_run, pure=False, resources=resources)
```
Proposed:
```
future = self.client.submit(subprocess.check_call, command, pure=False, resources=resources, close_fds=True)
```
cc @jedcunningham
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #11451: AttributeError: Can't pickle local object 'DaskExecutor.execute_async..airflow_run'
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #11451:
URL: https://github.com/apache/airflow/issues/11451#issuecomment-706834958
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aa1371 edited a comment on issue #11451: AttributeError: Can't pickle local object 'DaskExecutor.execute_async..airflow_run'
Posted by GitBox <gi...@apache.org>.
aa1371 edited a comment on issue #11451:
URL: https://github.com/apache/airflow/issues/11451#issuecomment-939202583
Could also just pass check_call directly to submit without needing to define the extra function. This works in a plugin I have defined.
```
future = self.client.submit(subprocess.check_call, command, pure=False, resources=resources, close_fds=True)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aa1371 commented on issue #11451: AttributeError: Can't pickle local object 'DaskExecutor.execute_async..airflow_run'
Posted by GitBox <gi...@apache.org>.
aa1371 commented on issue #11451:
URL: https://github.com/apache/airflow/issues/11451#issuecomment-939202583
Could also just pass check_call directly to submit without needing to define the extra function. This works in a plugin I have defined.
```
future = self.client.submit(subprocess.check_call, command, pure=False, resources=resources, close_fds=True)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] toughrogrammer commented on issue #11451: AttributeError: Can't pickle local object 'DaskExecutor.execute_async..airflow_run'
Posted by GitBox <gi...@apache.org>.
toughrogrammer commented on issue #11451:
URL: https://github.com/apache/airflow/issues/11451#issuecomment-753307462
I got similar error when I upgraded to 2.0.0
```
ERROR - Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File "/Users/loki/Library/Caches/pypoetry/virtualenvs/airflow-RLSyOqyF-py3.8/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1275, in _execute
self._run_scheduler_loop()
File "/Users/loki/Library/Caches/pypoetry/virtualenvs/airflow-RLSyOqyF-py3.8/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1379, in _run_scheduler_loop
self.executor.heartbeat()
File "/Users/loki/Library/Caches/pypoetry/virtualenvs/airflow-RLSyOqyF-py3.8/lib/python3.8/site-packages/airflow/executors/base_executor.py", line 155, in heartbeat
self.trigger_tasks(open_slots)
File "/Users/loki/Library/Caches/pypoetry/virtualenvs/airflow-RLSyOqyF-py3.8/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 260, in trigger_tasks
self._process_tasks(task_tuples_to_send)
File "/Users/loki/Library/Caches/pypoetry/virtualenvs/airflow-RLSyOqyF-py3.8/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 269, in _process_tasks
key_and_async_results = self._send_tasks_to_celery(task_tuples_to_send)
File "/Users/loki/Library/Caches/pypoetry/virtualenvs/airflow-RLSyOqyF-py3.8/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 327, in _send_tasks_to_celery
with Pool(processes=num_processes, initializer=reset_signals) as send_pool:
File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/pool.py", line 212, in __init__
self._repopulate_pool()
File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static
w.start()
File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Users/loki/.asdf/installs/python/3.8.5/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'CeleryExecutor._send_tasks_to_celery.<locals>.reset_signals'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] esc commented on issue #11451: AttributeError: Can't pickle local object 'DaskExecutor.execute_async..airflow_run'
Posted by GitBox <gi...@apache.org>.
esc commented on issue #11451:
URL: https://github.com/apache/airflow/issues/11451#issuecomment-800371224
I had the same issue and this fixed it for me:
https://github.com/uqfoundation/dill/issues/383#issuecomment-684979223
```
# workaround from https://github.com/uqfoundation/dill/issues/383
import dill
dill.extend(False)
import cloudpickle
dill.extend(True)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org