(airflow) branch main updated: Improve speed and stability of Python 3.12 tests in canary build (#38194)

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new dffc9e3fa8 Improve speed and stability of Python 3.12 tests in canary build (#38194)
dffc9e3fa8 is described below

commit dffc9e3fa83e3cd910a83e7c505fe23b26feddf2
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Fri Mar 15 23:32:34 2024 +0100

    Improve speed and stability of Python 3.12 tests in canary build (#38194)
    
    Python 3.12 introduced a new (much faster) way of tracking and
    monitoring execution of python code by tools like coverage tracking
    using sysmon (PEP 669). This however also apparently heavily impacted
    performance of coverage tracking for Python 3.12 when PEP 669 is not
    used. The coverage library since 7.4.0 has an experimental support
    for PEP 669 that can be enabled with COVERAGE_CORE=sysmon env variable
    and a number of users confirmed it fixes the problem.
    
    We are using 7.4.4 coverage already so we should enable this mode
    to speed up our coverage tracking. That should also allow us to
    remove databricks from excluded providers.
    
    See https://github.com/databricks/databricks-sql-python/issues/369
    for databricks case and https://github.com/nedbat/coveragepy/issues/1665
    for coverage bug.
---
 dev/breeze/src/airflow_breeze/commands/testing_commands.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/dev/breeze/src/airflow_breeze/commands/testing_commands.py b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
index eceaa0a22e..cb8bd5723e 100644
--- a/dev/breeze/src/airflow_breeze/commands/testing_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
@@ -841,6 +841,8 @@ def helm_tests(
         python_version=shell_params.python,
         helm_test_package=helm_test_package,
     )
+    if sys.version_info >= (3, 12):
+        env["COVERAGE_CORE"] = "sysmon"
     cmd = ["docker", "compose", "run", "--service-ports", "--rm", "airflow", *pytest_args, *extra_pytest_args]
     result = run_command(cmd, check=False, env=env, output_outside_the_group=True)
     fix_ownership_using_docker()