[buildstream] branch chandan/macos-multiprocessing created (now 029b43b)

[ro...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

root pushed a change to branch chandan/macos-multiprocessing
in repository https://gitbox.apache.org/repos/asf/buildstream.git.


      at 029b43b  Force multiprocessing method to be fork if available

This branch includes the following new commits:

     new 029b43b  Force multiprocessing method to be fork if available

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[buildstream] 01/01: Force multiprocessing method to be fork if available

[ro...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

root pushed a commit to branch chandan/macos-multiprocessing
in repository https://gitbox.apache.org/repos/asf/buildstream.git

commit 029b43b33ae8cb0563ad95b199928691980c8d05
Author: Chandan Singh <ch...@chandansingh.net>
AuthorDate: Mon Aug 24 22:53:30 2020 +0100

    Force multiprocessing method to be fork if available
    
    Since Python 3.8, the multiprocessing module defaults to the spawn
    method on MacOS, since the fork method is considered unsafe.
    
    However, BuildStream is not compatible with spawn mode so we
    force the default to fork (if available) which works for the most part.
    
    See
    https://lists.apache.org/thread.html/r08edd3dd10ce3039da5eb01f0722d25c44406ab8afadcc3ae78a5414%40%3Cdev.buildstream.apache.org%3E
    for some related discussion.
---
 src/buildstream/_frontend/cli.py | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/src/buildstream/_frontend/cli.py b/src/buildstream/_frontend/cli.py
index b0eacbb..8474d3d 100644
--- a/src/buildstream/_frontend/cli.py
+++ b/src/buildstream/_frontend/cli.py
@@ -1,3 +1,4 @@
+import multiprocessing
 import os
 import sys
 from functools import partial
@@ -12,6 +13,33 @@ from ..utils import UtilError
 
 
 ##################################################################
+#                   Setup multiprocessing                        #
+##################################################################
+
+# _setup_multiprocessing()
+#
+# Since Python 3.8, the multiprocessing module defaults to
+# the spawn method on MacOS, since the fork method is considered
+# unsafe.
+#
+# However, BuildStream is not compatible with spawn mode so we
+# force the default to fork (if available) which works for the most part.
+#
+# Args:
+#    force (bool): Whether to force our configuration; useful when being called
+#                  from the test suite.
+#
+def _setup_multiprocessing(force=False):
+    # Our tests will not invoke the CLI as a subprocess, but directly from
+    # Python via Cli.run(). So, we allow `force` to be true in this case, but
+    # it is not needed when using the CLI normally.
+    force = "BST_TEST_SUITE" in os.environ
+
+    if multiprocessing.get_start_method() == "spawn" and "fork" in multiprocessing.get_all_start_methods():
+        multiprocessing.set_start_method("fork", force=force)
+
+
+##################################################################
 #              Helper classes and methods for Click              #
 ##################################################################
 
@@ -332,6 +360,10 @@ def cli(context, **kwargs):
     user preferences configuration file.
     """
 
+    # Configuration for multiprocessing module, that we want to do
+    # as early as possible.
+    _setup_multiprocessing()
+
     from .app import App
 
     # Create the App, giving it the main arguments