[airflow] branch main updated: Fix issue with unit tests in subprocesses on macOS (#34037)

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new 5833320291 Fix issue with unit tests in subprocesses on macOS (#34037)
5833320291 is described below

commit 5833320291ee1ed5670b63d24c8ed2287f4cffe3
Author: Daniel Dyląg <Bi...@users.noreply.github.com>
AuthorDate: Sun Sep 3 17:50:45 2023 +0200

    Fix issue with unit tests in subprocesses on macOS (#34037)
    
    * Fix issue with unit tests in subprocesses on Mac
    
    When unit tests are run on local virtual env on macOS, a lot of them fail because mocks do not work as expected when they are used in subprocesses.
    See https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
    "Changed in version 3.8: On macOS, the spawn start method is now the default (...)". On Unix systems "fork" is the default.
    
    When you mock a method with patch() in unittest.mock, this mock will be accessible in subprocess only if the start command was "fork" and not "spawn".
    I don't know how many tests were affected by it, for me, it was test_scheduler_job.py.
    
    Moved config to conftest.py and enable only in macOS
    ---------
    
    Co-authored-by: daniel.dylag <da...@gmail.com>
---
 tests/conftest.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/tests/conftest.py b/tests/conftest.py
index a652cb9cef..ac3c785708 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -18,6 +18,7 @@ from __future__ import annotations
 
 import json
 import os
+import platform
 import subprocess
 import sys
 from contextlib import ExitStack, suppress
@@ -42,6 +43,11 @@ os.environ["AWS_DEFAULT_REGION"] = os.environ.get("AWS_DEFAULT_REGION") or "us-e
 os.environ["CREDENTIALS_DIR"] = os.environ.get("CREDENTIALS_DIR") or "/files/airflow-breeze-config/keys"
 os.environ["AIRFLOW_ENABLE_AIP_44"] = os.environ.get("AIRFLOW_ENABLE_AIP_44") or "true"
 
+if platform.system() == "Darwin":
+    # mocks from unittest.mock work correctly in subprocesses only if they are created by "fork" method
+    # but macOS uses "spawn" by default
+    os.environ["AIRFLOW__CORE__MP_START_METHOD"] = "fork"
+
 from airflow import settings  # noqa: E402
 from airflow.models.tasklog import LogTemplate  # noqa: E402
 from tests.test_utils.db import clear_all  # noqa: E402