[GitHub] [airflow] sarutak commented on a change in pull request #7128: [AIRFLOW-6529] Pickle error occurs when the scheduler tries to run on macOS.

[GitBox <gi...@apache.org>]

sarutak commented on a change in pull request #7128: [AIRFLOW-6529] Pickle error occurs when the scheduler tries to run on macOS.
URL: https://github.com/apache/airflow/pull/7128#discussion_r365555136
 
 

 ##########
 File path: tests/test_utils/mock_executor.py
 ##########
 @@ -36,10 +36,17 @@ def __init__(self, do_update=True, *args, **kwargs):
         self.history = []
         # All the tasks, in a stable sort order
         self.sorted_tasks = []
-        self.mock_task_results = defaultdict(lambda: State.SUCCESS)
+
+        # If multiprocessing runs in spawn mode,
+        # arguments are to be pickled but lambda is not picclable.
+        # So we should pass self.success instead of lambda.
 
 Review comment:
   If we run `TestSchedulerJob.test_retry_still_in_executor` on macOS, we will get following stack-trace.
   
   ```
     File "/Users/kou/work/oss/airflow/airflow/jobs/scheduler_job.py", line 1499, in _execute
       self._execute_helper()
     File "/Users/kou/work/oss/airflow/airflow/jobs/scheduler_job.py", line 1532, in _execute_helper
       self.processor_agent.start()
     File "/Users/kou/work/oss/airflow/airflow/utils/dag_processing.py", line 353, in start
       self._process.start()
     File "/opt/python3/3.8.1/lib/python3.8/multiprocessing/process.py", line 121, in start
       self._popen = self._Popen(self)
     File "/opt/python3/3.8.1/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
       return _default_context.get_context().Process._Popen(process_obj)
     File "/opt/python3/3.8.1/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
       return Popen(process_obj)
     File "/opt/python3/3.8.1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
       super().__init__(process_obj)
     File "/opt/python3/3.8.1/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
       self._launch(process_obj)
     File "/opt/python3/3.8.1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
       reduction.dump(process_obj, fp)
     File "/opt/python3/3.8.1/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
       ForkingPickler(file, protocol).dump(obj)
   AttributeError: Can't pickle local object 'MockExecutor.__init__.<locals>.<lambda>'
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services