You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by no...@apache.org on 2020/12/29 12:32:50 UTC
[buildstream] branch aevri/win32_minimal_seemstowork_20190829
created (now d2d5dfe)
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a change to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git.
at d2d5dfe TEMP: INFO log pickle length and time
This branch includes the following new commits:
new 0ca2aa6 optionpool: make OptionPool picklable
new 4302107 WIP: pickleable jobs
new 22d06c0 job: pickle child jobs on start if BST_TEST_SUITE
new e160f09 WIP: _plugincontext, lint fixes
new e9e61f3 WIP: jobpickler, _{,un}reduce_plugincontext
new 799cbf0 Revert "WIP: jobpickler, _{,un}reduce_plugincontext"
new 7726811 WIP: bst-job-replay
new 97cf623 element: platform independent _get_normal_name
new 28951e2 WIP: spawn instead of fork
new 7843544 WIP: _scheduler/jobs/job.py: sort imports
new 76efcbc WIP: win32: _platform/win32: add support for win32
new da883af WIP: win32: job: replace add_child_handler with thread
new cd8fd57 WIP: sandboxnone: use initial SandboxNone
new 0fa65b0 WIP: win32: job: signals / win32
new 1ea5516 WIP: win32: ImportElement - fix separators
new a8833a4 WIP: win32: don't set owner of files
new bdf3bad WIP: win32: sandboxnone: pass-through env for now
new eeeca14 WIP: win32: WINDOWS.md: repro instructions
new 00ed4db WIP: mv multiprocessing os-specifics to platform
new b852196 TEMP: testpickle
new 9288cbd TEMP: no test pickle
new d2d5dfe TEMP: INFO log pickle length and time
The 22 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[buildstream] 21/22: TEMP: no test pickle
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 9288cbd76b1995c634225624a28e78df94f94c5e
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Jul 9 13:02:22 2019 +0100
TEMP: no test pickle
---
src/buildstream/_scheduler/jobs/jobpickler.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/buildstream/_scheduler/jobs/jobpickler.py b/src/buildstream/_scheduler/jobs/jobpickler.py
index 2e0ee5c..fd7162a 100644
--- a/src/buildstream/_scheduler/jobs/jobpickler.py
+++ b/src/buildstream/_scheduler/jobs/jobpickler.py
@@ -140,10 +140,10 @@ def pickle_child_job(child_job, context):
pickler.dispatch_table[Loader] = _reduce_loader
pickler.dispatch_table[Messenger] = _reduce_messenger
- import buildstream.testpickle
- test_pickler = buildstream.testpickle.TestPickler()
- test_pickler.dispatch_table = pickler.dispatch_table.copy()
- test_pickler.test_dump(child_job)
+ # import buildstream.testpickle
+ # test_pickler = buildstream.testpickle.TestPickler()
+ # test_pickler.dispatch_table = pickler.dispatch_table.copy()
+ # test_pickler.test_dump(child_job)
pickler.dump(child_job)
data.seek(0)
[buildstream] 18/22: WIP: win32: WINDOWS.md: repro instructions
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit eeeca143a960f2440ef8b2e2551d370666068692
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Thu Apr 18 11:16:51 2019 +0100
WIP: win32: WINDOWS.md: repro instructions
---
WINDOWS.md | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 59 insertions(+)
diff --git a/WINDOWS.md b/WINDOWS.md
new file mode 100644
index 0000000..46e3655
--- /dev/null
+++ b/WINDOWS.md
@@ -0,0 +1,59 @@
+Running on Windows
+==================
+
+This is a temporary doc tied to the lifetime of the `aevri/win32` branch,
+intended to help you repro the results.
+
+Installation
+------------
+
+First, make sure you have Python 3 installed.
+
+Next, you probably want to create a venv to install BuildStream into, for
+experimentation.
+
+Then, clone and install BuildStream:
+
+ git clone --branch aevri/win32 https://gitlab.com/buildstream/buildstream.git
+ pip install -e ./buildstream
+
+Next, install some additional dependencies for proper display:
+
+ pip install colorama windows-curses
+
+Finally, make sure you have the build tools installed:
+
+- Download the installer from: https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2017
+- Run the installer.
+- Select to install "Visual C++ build tools". Possibly need to include these
+ optional items:
+ - Windows 10 SDK
+ - Visual C++ tools for CMake
+ - Testing tools core feature - Build Tools
+
+Hello World
+-----------
+
+Here is how to build the "Hello World" example.
+
+First, launch a "Developer Command Prompt for VS 2017". This ensures that you
+have the correct environment variables for building. The next instructions
+assume you are running inside this prompt.
+
+Next, make sure you have activated any virtual environment for BuildStream.
+
+Then, change directory to the buildstream git repository.
+
+Finally, build and run like so:
+
+ bst --help
+
+ cd doc\examples\running-commands
+
+ bst show hello.bst
+
+ bst build hello.bst
+
+ bst artifact checkout hello.bst --directory checkout
+ cd checkout
+ hello.exe
[buildstream] 02/22: WIP: pickleable jobs
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 43021072636526991f89edf336e21fd658c47452
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Wed Jul 10 09:24:30 2019 +0100
WIP: pickleable jobs
WIP: make some things pickleable
WIP: Make ElementFactory and SourceFactory picklable
WIP: job: pickling machinery for jobs
WIP: loader: don't pickle _fetch_subprojects
Note that a new circular dependency was introduced in:
737aea18 Handle subproject fetching in the Stream class
We can avoid the complications by not pickling it out to the child job.
WIP: move reducers to jobpickler
WIP: jobpickler: use proto serialization instead of pickle
WIP: jobpickler: pickle less
WIP: job: pickle apologies
---
src/buildstream/_elementfactory.py | 7 +-
src/buildstream/_plugincontext.py | 26 ++++-
src/buildstream/_project.py | 12 +-
src/buildstream/_scheduler/jobs/jobpickler.py | 151 ++++++++++++++++++++++++++
src/buildstream/_sourcefactory.py | 5 +-
src/buildstream/element.py | 1 +
src/buildstream/source.py | 2 +
7 files changed, 194 insertions(+), 10 deletions(-)
diff --git a/src/buildstream/_elementfactory.py b/src/buildstream/_elementfactory.py
index d6591bf..b2a7f73 100644
--- a/src/buildstream/_elementfactory.py
+++ b/src/buildstream/_elementfactory.py
@@ -33,9 +33,12 @@ class ElementFactory(PluginContext):
def __init__(self, plugin_base, *,
format_versions={},
- plugin_origins=None):
+ plugin_origins=None,
+ pass_=None):
- super().__init__(plugin_base, Element, [_site.element_plugins],
+ assert pass_ is not None
+
+ super().__init__(plugin_base, Element, [_site.element_plugins], 'element' + str(pass_),
plugin_origins=plugin_origins,
format_versions=format_versions)
diff --git a/src/buildstream/_plugincontext.py b/src/buildstream/_plugincontext.py
index 7fef9b9..fa07e7b 100644
--- a/src/buildstream/_plugincontext.py
+++ b/src/buildstream/_plugincontext.py
@@ -44,9 +44,11 @@ from . import _yaml
#
class PluginContext():
- def __init__(self, plugin_base, base_type, site_plugin_path, *,
+ def __init__(self, plugin_base, base_type, site_plugin_path, identifier, *,
plugin_origins=None, format_versions={}):
+ self._identifier = identifier
+
# The plugin kinds which were loaded
self.loaded_dependencies = []
@@ -59,10 +61,26 @@ class PluginContext():
# The PluginSource object
self._plugin_base = plugin_base
- self._site_source = plugin_base.make_plugin_source(searchpath=site_plugin_path)
+ self._site_plugin_path = site_plugin_path
+ self._site_source = plugin_base.make_plugin_source(
+ searchpath=self._site_plugin_path,
+ identifier='site_plugin-' + self._identifier)
self._alternate_sources = {}
self._format_versions = format_versions
+ def __getstate__(self):
+ import copy
+ state = copy.copy(self.__dict__)
+ del state['_site_source']
+ state['_types'] = {}
+ return state
+
+ def __setstate__(self, state):
+ self.__dict__ = state
+ self._site_source = self._plugin_base.make_plugin_source(
+ searchpath=self._site_plugin_path,
+ identifier='site_plugin-' + self._identifier)
+
# lookup():
#
# Fetches a type loaded from a plugin in this plugin context
@@ -80,7 +98,7 @@ class PluginContext():
def _get_local_plugin_source(self, path):
if ('local', path) not in self._alternate_sources:
# key by a tuple to avoid collision
- source = self._plugin_base.make_plugin_source(searchpath=[path])
+ source = self._plugin_base.make_plugin_source(searchpath=[path], identifier='local_plugin-' + path + '-' + self._identifier)
# Ensure that sources never get garbage collected,
# as they'll take the plugins with them.
self._alternate_sources[('local', path)] = source
@@ -121,7 +139,7 @@ class PluginContext():
# The plugin didn't have an accompanying YAML file
defaults = None
- source = self._plugin_base.make_plugin_source(searchpath=[os.path.dirname(location)])
+ source = self._plugin_base.make_plugin_source(searchpath=[os.path.dirname(location)], identifier='pip_plugin-' + self._identifier)
self._alternate_sources[('pip', package_name)] = source
else:
diff --git a/src/buildstream/_project.py b/src/buildstream/_project.py
index 5f433c0..74fca7f 100644
--- a/src/buildstream/_project.py
+++ b/src/buildstream/_project.py
@@ -97,6 +97,8 @@ class Project():
default_mirror=None, parent_loader=None,
search_for_project=True, fetch_subprojects=None):
+ self._pass = None
+
# The project name
self.name = None
@@ -617,6 +619,8 @@ class Project():
config_no_include = _yaml.node_copy(self._default_config_node)
_yaml.composite(config_no_include, project_conf_first_pass)
+ assert self._pass is None
+ self._pass = 1
self._load_pass(config_no_include, self.first_pass_config,
ignore_unknown=True)
@@ -641,6 +645,8 @@ class Project():
config = _yaml.node_copy(self._default_config_node)
_yaml.composite(config, project_conf_second_pass)
+ assert self._pass == 1
+ self._pass = 2
self._load_pass(config, self.config)
self._validate_node(config)
@@ -914,10 +920,12 @@ class Project():
pluginbase = PluginBase(package='buildstream.plugins')
output.element_factory = ElementFactory(pluginbase,
plugin_origins=plugin_element_origins,
- format_versions=element_format_versions)
+ format_versions=element_format_versions,
+ pass_=self._pass)
output.source_factory = SourceFactory(pluginbase,
plugin_origins=plugin_source_origins,
- format_versions=source_format_versions)
+ format_versions=source_format_versions,
+ pass_=self._pass)
# _store_origin()
#
diff --git a/src/buildstream/_scheduler/jobs/jobpickler.py b/src/buildstream/_scheduler/jobs/jobpickler.py
new file mode 100644
index 0000000..5c1742f
--- /dev/null
+++ b/src/buildstream/_scheduler/jobs/jobpickler.py
@@ -0,0 +1,151 @@
+#
+# Copyright (C) 2019 Bloomberg Finance LP
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
+#
+# Authors:
+# Angelos Evripiotis <je...@bloomberg.net>
+
+
+import copyreg
+import io
+import pickle
+
+from ..._protos.buildstream.v2.artifact_pb2 import Artifact as ArtifactProto
+
+# BuildStream toplevel imports
+from ... import Element, Source
+from ..._loader import Loader
+from ..._messenger import Messenger
+
+
+def _reduce_artifact_proto(instance):
+ assert isinstance(instance, ArtifactProto)
+ data = instance.SerializeToString()
+ return (_unreduce_artifact_proto, (data,))
+
+
+def _unreduce_artifact_proto(data):
+ instance = ArtifactProto()
+ instance.ParseFromString(data)
+ return instance
+
+
+def _reduce_loader(instance):
+ assert isinstance(instance, Loader)
+ state = instance.__dict__.copy()
+
+ # When pickling a Loader over to the ChildJob, we don't want to bring
+ # the whole Stream over with it. The _fetch_subprojects member is a method
+ # of the Stream. We also don't want to remove it in the main process. If we
+ # remove it in the child process then we will already be too late. The only
+ # time that seems just right is here, when preparing the child process'
+ # copy of the Loader.
+ #
+ del state['_fetch_subprojects']
+
+ return (Loader.__new__, (Loader,), state)
+
+
+def _reduce_messenger(instance):
+ assert isinstance(instance, Messenger)
+ state = instance.__dict__.copy()
+
+ # When pickling a Messenger over to the ChildJob, we don't want to bring
+ # the whole _message_handler over with it. We also don't want to remove it
+ # in the main process. If we remove it in the child process then we will
+ # already be too late. The only time that seems just right is here, when
+ # preparing the child process' copy of the Messenger.
+ #
+ # Another approach might be to use a context manager on the Messenger,
+ # which removes and restores the _message_handler. This wouldn't require
+ # access to private details of Messenger.
+ #
+ del state['_message_handler']
+
+ return (Messenger.__new__, (Messenger,), state)
+
+
+def _reduce_element(element):
+ assert isinstance(element, Element)
+ meta_kind = element._meta_kind
+ project = element._get_project()
+ factory = project.config.element_factory
+ args = (factory, meta_kind)
+ state = element.__dict__.copy()
+ state["_Element__reverse_dependencies"] = None
+ state["_Element__buildable_callback"] = None
+ return (_unreduce_plugin, args, state)
+
+
+def _reduce_source(source):
+ assert isinstance(source, Source)
+ meta_kind = source._meta_kind
+ project = source._get_project()
+ factory = project.config.source_factory
+ args = (factory, meta_kind)
+ return (_unreduce_plugin, args, source.__dict__.copy())
+
+
+def _unreduce_plugin(factory, meta_kind):
+ cls, _ = factory.lookup(meta_kind)
+ plugin = cls.__new__(cls)
+
+ # TODO: find a better way of persisting this factory, otherwise the plugin
+ # will become invalid.
+ plugin.factory = factory
+
+ return plugin
+
+
+def pickle_child_job(child_job, context):
+
+ # Note: Another way of doing this would be to let PluginBase do it's
+ # import-magic. We would achieve this by first pickling the factories, and
+ # the string names of their plugins. Unpickling the plugins in the child
+ # process would then "just work". There would be an additional cost of
+ # having to load every plugin kind, regardless of which ones are used.
+
+ projects = context.get_projects()
+ element_classes = [
+ cls
+ for p in projects
+ for cls, _ in p.config.element_factory._types.values()
+ ]
+ source_classes = [
+ cls
+ for p in projects
+ for cls, _ in p.config.source_factory._types.values()
+ ]
+
+ data = io.BytesIO()
+ pickler = pickle.Pickler(data)
+ pickler.dispatch_table = copyreg.dispatch_table.copy()
+ for cls in element_classes:
+ pickler.dispatch_table[cls] = _reduce_element
+ for cls in source_classes:
+ pickler.dispatch_table[cls] = _reduce_source
+ pickler.dispatch_table[ArtifactProto] = _reduce_artifact_proto
+ pickler.dispatch_table[Loader] = _reduce_loader
+ pickler.dispatch_table[Messenger] = _reduce_messenger
+
+ pickler.dump(child_job)
+ data.seek(0)
+
+ return data
+
+
+def unpickle_child_job(pickled):
+ child_job = pickle.load(pickled)
+ return child_job
diff --git a/src/buildstream/_sourcefactory.py b/src/buildstream/_sourcefactory.py
index 1d959a1..eca4b50 100644
--- a/src/buildstream/_sourcefactory.py
+++ b/src/buildstream/_sourcefactory.py
@@ -33,9 +33,10 @@ class SourceFactory(PluginContext):
def __init__(self, plugin_base, *,
format_versions={},
- plugin_origins=None):
+ plugin_origins=None,
+ pass_=None):
- super().__init__(plugin_base, Source, [_site.source_plugins],
+ super().__init__(plugin_base, Source, [_site.source_plugins], 'source' + str(pass_),
format_versions=format_versions,
plugin_origins=plugin_origins)
diff --git a/src/buildstream/element.py b/src/buildstream/element.py
index 758a0b9..7061a27 100644
--- a/src/buildstream/element.py
+++ b/src/buildstream/element.py
@@ -186,6 +186,7 @@ class Element(Plugin):
"""
def __init__(self, context, project, meta, plugin_conf):
+ self._meta_kind = meta.kind
self.__cache_key_dict = None # Dict for cache key calculation
self.__cache_key = None # Our cached cache key
diff --git a/src/buildstream/source.py b/src/buildstream/source.py
index f8b5d3f..202af83 100644
--- a/src/buildstream/source.py
+++ b/src/buildstream/source.py
@@ -312,6 +312,8 @@ class Source(Plugin):
super().__init__("{}-{}".format(meta.element_name, meta.element_index),
context, project, provenance, "source", unique_id=unique_id)
+ self._meta_kind = meta.kind
+
self.__element_name = meta.element_name # The name of the element owning this source
self.__element_index = meta.element_index # The index of the source in the owning element's source list
self.__element_kind = meta.element_kind # The kind of the element owning this source
[buildstream] 06/22: Revert "WIP: jobpickler, _{,
un}reduce_plugincontext"
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 799cbf0eea09974817d2ba607f023fa1cee8293f
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Wed Jul 10 08:58:29 2019 +0100
Revert "WIP: jobpickler, _{,un}reduce_plugincontext"
This reverts commit 8dc514026627c5dbacc47a970a5ec511786edf69.
---
src/buildstream/_plugincontext.py | 13 +++++++++++++
src/buildstream/_scheduler/jobs/jobpickler.py | 24 ------------------------
2 files changed, 13 insertions(+), 24 deletions(-)
diff --git a/src/buildstream/_plugincontext.py b/src/buildstream/_plugincontext.py
index c058c58..60d2475 100644
--- a/src/buildstream/_plugincontext.py
+++ b/src/buildstream/_plugincontext.py
@@ -68,6 +68,19 @@ class PluginContext():
self._alternate_sources = {}
self._format_versions = format_versions
+ def __getstate__(self):
+ import copy
+ state = copy.copy(self.__dict__)
+ del state['_site_source']
+ state['_types'] = {}
+ return state
+
+ def __setstate__(self, state):
+ self.__dict__ = state
+ self._site_source = self._plugin_base.make_plugin_source(
+ searchpath=self._site_plugin_path,
+ identifier='site_plugin-' + self._identifier)
+
# lookup():
#
# Fetches a type loaded from a plugin in this plugin context
diff --git a/src/buildstream/_scheduler/jobs/jobpickler.py b/src/buildstream/_scheduler/jobs/jobpickler.py
index 841037c..5c1742f 100644
--- a/src/buildstream/_scheduler/jobs/jobpickler.py
+++ b/src/buildstream/_scheduler/jobs/jobpickler.py
@@ -28,7 +28,6 @@ from ..._protos.buildstream.v2.artifact_pb2 import Artifact as ArtifactProto
from ... import Element, Source
from ..._loader import Loader
from ..._messenger import Messenger
-from ..._plugincontext import PluginContext
def _reduce_artifact_proto(instance):
@@ -54,7 +53,6 @@ def _reduce_loader(instance):
# time that seems just right is here, when preparing the child process'
# copy of the Loader.
#
- # TODO: move this reduce func to loader.py
del state['_fetch_subprojects']
return (Loader.__new__, (Loader,), state)
@@ -74,32 +72,11 @@ def _reduce_messenger(instance):
# which removes and restores the _message_handler. This wouldn't require
# access to private details of Messenger.
#
- # TODO: move this reduce func to messenger.py
del state['_message_handler']
return (Messenger.__new__, (Messenger,), state)
-def _reduce_plugincontext(instance):
- assert isinstance(instance, PluginContext)
- state = instance.__dict__.copy()
- del state['_site_source']
- state['_types'] = {}
-
- # TODO: move this reduce func to plugincontext.py
-
- return (_unreduce_plugincontext, (state,))
-
-
-def _unreduce_plugincontext(state):
- instance = PluginContext()
- instance.__dict__ = state
- instance._site_source = instance._plugin_base.make_plugin_source(
- searchpath=instance._site_plugin_path,
- identifier='site_plugin-' + instance._identifier)
- return instance
-
-
def _reduce_element(element):
assert isinstance(element, Element)
meta_kind = element._meta_kind
@@ -162,7 +139,6 @@ def pickle_child_job(child_job, context):
pickler.dispatch_table[ArtifactProto] = _reduce_artifact_proto
pickler.dispatch_table[Loader] = _reduce_loader
pickler.dispatch_table[Messenger] = _reduce_messenger
- pickler.dispatch_table[PluginContext] = _reduce_plugincontext
pickler.dump(child_job)
data.seek(0)
[buildstream] 17/22: WIP: win32: sandboxnone: pass-through env for
now
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit bdf3badccd4fa65edb4910852d917c5f7eb55f94
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Wed Apr 17 16:34:39 2019 +0100
WIP: win32: sandboxnone: pass-through env for now
Rely on the user running "Developer Command Prompt for VS2017", so that
we get the correct environment variables for building.
---
src/buildstream/sandbox/_sandboxnone.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/buildstream/sandbox/_sandboxnone.py b/src/buildstream/sandbox/_sandboxnone.py
index e95a7b9..f8a2bda 100644
--- a/src/buildstream/sandbox/_sandboxnone.py
+++ b/src/buildstream/sandbox/_sandboxnone.py
@@ -59,7 +59,8 @@ class SandboxNone(Sandbox):
path = pathlib.Path(self.get_directory()) / cwd
print('run', command, 'in', path)
- result = subprocess.run(command, cwd=path, env=env)
+ #result = subprocess.run(command, cwd=path, env=env)
+ result = subprocess.run(command, cwd=path)
# out = pathlib.Path(self.get_directory()) / 'buildstream-install'
# out.mkdir(exist_ok=True)
[buildstream] 04/22: WIP: _plugincontext, lint fixes
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit e160f0918cc5793e0bf38aee1d5119373ef65daf
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Jul 9 17:35:04 2019 +0100
WIP: _plugincontext, lint fixes
---
src/buildstream/_plugincontext.py | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/src/buildstream/_plugincontext.py b/src/buildstream/_plugincontext.py
index fa07e7b..60d2475 100644
--- a/src/buildstream/_plugincontext.py
+++ b/src/buildstream/_plugincontext.py
@@ -98,7 +98,10 @@ class PluginContext():
def _get_local_plugin_source(self, path):
if ('local', path) not in self._alternate_sources:
# key by a tuple to avoid collision
- source = self._plugin_base.make_plugin_source(searchpath=[path], identifier='local_plugin-' + path + '-' + self._identifier)
+ source = self._plugin_base.make_plugin_source(
+ searchpath=[path],
+ identifier='local_plugin-' + path + '-' + self._identifier,
+ )
# Ensure that sources never get garbage collected,
# as they'll take the plugins with them.
self._alternate_sources[('local', path)] = source
@@ -139,7 +142,10 @@ class PluginContext():
# The plugin didn't have an accompanying YAML file
defaults = None
- source = self._plugin_base.make_plugin_source(searchpath=[os.path.dirname(location)], identifier='pip_plugin-' + self._identifier)
+ source = self._plugin_base.make_plugin_source(
+ searchpath=[os.path.dirname(location)],
+ identifier='pip_plugin-' + self._identifier,
+ )
self._alternate_sources[('pip', package_name)] = source
else:
[buildstream] 09/22: WIP: spawn instead of fork
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 28951e285af67b98f657e31529963ddecc6dc9cd
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Jul 9 13:46:41 2019 +0100
WIP: spawn instead of fork
---
src/buildstream/_messenger.py | 12 +++++++-----
src/buildstream/_scheduler/jobs/job.py | 26 ++++++++++++++++----------
src/buildstream/_scheduler/scheduler.py | 4 ++++
3 files changed, 27 insertions(+), 15 deletions(-)
diff --git a/src/buildstream/_messenger.py b/src/buildstream/_messenger.py
index 7dec939..d74d391 100644
--- a/src/buildstream/_messenger.py
+++ b/src/buildstream/_messenger.py
@@ -179,7 +179,9 @@ class Messenger():
# we also do not allow it in the main process.
assert self._log_handle is None
assert self._log_filename is None
- assert not utils._is_main_process()
+
+ # Need to deal with global _main_pid var.
+ # assert not utils._is_main_process()
# Create the fully qualified logfile in the log directory,
# appending the pid and .log extension at the end.
@@ -255,10 +257,10 @@ class Messenger():
# If this message is associated with a plugin, print what
# we know about the plugin.
plugin_name = ""
- if message.unique_id:
- template += " {plugin}"
- plugin = Plugin._lookup(message.unique_id)
- plugin_name = plugin.name
+ # if message.unique_id:
+ # template += " {plugin}"
+ # plugin = Plugin._lookup(message.unique_id)
+ # plugin_name = plugin.name
template += ": {message}"
diff --git a/src/buildstream/_scheduler/jobs/job.py b/src/buildstream/_scheduler/jobs/job.py
index 47a8622..8365bd8 100644
--- a/src/buildstream/_scheduler/jobs/job.py
+++ b/src/buildstream/_scheduler/jobs/job.py
@@ -34,7 +34,7 @@ from ..._exceptions import ImplError, BstError, set_last_task_error, SkipJob
from ..._message import Message, MessageType, unconditional_messages
from ... import _signals, utils
-from .jobpickler import pickle_child_job
+from .jobpickler import pickle_child_job, unpickle_child_job
# Return code values shutdown of job handling child processes
@@ -89,6 +89,11 @@ class _MessageType(enum.Enum):
SUBCLASS_CUSTOM_MESSAGE = 5
+def _do_pickled_child_job(pickled, *child_args):
+ child_job = unpickle_child_job(pickled)
+ return child_job.child_action(*child_args)
+
+
# Job()
#
# The Job object represents a task that will run in parallel to the main
@@ -165,7 +170,7 @@ class Job():
#
def start(self):
- self._queue = multiprocessing.Queue()
+ self._queue = self._scheduler.manager.Queue()
self._tries += 1
self._parent_start_listening()
@@ -181,10 +186,11 @@ class Job():
self._task_id,
)
- if 'BST_TEST_SUITE' in os.environ:
- pickle_child_job(child_job, self._scheduler.context)
-
- self._process = Process(target=child_job.child_action, args=[self._queue])
+ pickled = pickle_child_job(child_job, self._scheduler.context)
+ self._process = Process(
+ target=_do_pickled_child_job,
+ args=[pickled, self._queue],
+ )
# Block signals which are handled in the main process such that
# the child process does not inherit the parent's state, but the main
@@ -537,8 +543,8 @@ class Job():
# http://bugs.python.org/issue3831
#
if not self._listening:
- self._scheduler.loop.add_reader(
- self._queue._reader.fileno(), self._parent_recv)
+ # self._scheduler.loop.add_reader(
+ # self._queue._reader.fileno(), self._parent_recv)
self._listening = True
# _parent_stop_listening()
@@ -547,7 +553,7 @@ class Job():
#
def _parent_stop_listening(self):
if self._listening:
- self._scheduler.loop.remove_reader(self._queue._reader.fileno())
+ # self._scheduler.loop.remove_reader(self._queue._reader.fileno())
self._listening = False
@@ -830,7 +836,7 @@ class ChildJob():
# exit_code (_ReturnCode): The exit code to exit with
#
def _child_shutdown(self, exit_code):
- self._queue.close()
+ # self._queue.close()
assert isinstance(exit_code, _ReturnCode)
sys.exit(int(exit_code))
diff --git a/src/buildstream/_scheduler/scheduler.py b/src/buildstream/_scheduler/scheduler.py
index 00d6114..ae20fcd 100644
--- a/src/buildstream/_scheduler/scheduler.py
+++ b/src/buildstream/_scheduler/scheduler.py
@@ -72,6 +72,10 @@ class Scheduler():
interrupt_callback=None,
ticker_callback=None):
+ import multiprocessing
+ multiprocessing.set_start_method('spawn')
+ self.manager = multiprocessing.Manager()
+
#
# Public members
#
[buildstream] 19/22: WIP: mv multiprocessing os-specifics to
platform
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 00ed4dbb100493fe3ab172b6eff35e7fa6ecef09
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Jun 25 15:46:37 2019 +0100
WIP: mv multiprocessing os-specifics to platform
---
src/buildstream/_platform/darwin.py | 12 ++++++++++++
src/buildstream/_platform/platform.py | 4 ++++
src/buildstream/_platform/win32.py | 11 +++++++++--
src/buildstream/_scheduler/jobs/job.py | 5 +++--
src/buildstream/_scheduler/scheduler.py | 4 ----
5 files changed, 28 insertions(+), 8 deletions(-)
diff --git a/src/buildstream/_platform/darwin.py b/src/buildstream/_platform/darwin.py
index 86820a3..4efa53a 100644
--- a/src/buildstream/_platform/darwin.py
+++ b/src/buildstream/_platform/darwin.py
@@ -15,6 +15,7 @@
# You should have received a copy of the GNU Lesser General Public
# License along with this library. If not, see <http://www.gnu.org/licenses/>.
+import multiprocessing
import os
import resource
@@ -28,6 +29,12 @@ class Darwin(Platform):
# This value comes from OPEN_MAX in syslimits.h
OPEN_MAX = 10240
+ def __init__(self):
+ super().__init__()
+ if None is multiprocessing.get_start_method(allow_none=True):
+ multiprocessing.set_start_method('spawn')
+ self._manager = None
+
def create_sandbox(self, *args, **kwargs):
kwargs['dummy_reason'] = \
"OSXFUSE is not supported and there are no supported sandbox " + \
@@ -62,3 +69,8 @@ class Darwin(Platform):
old_soft_limit, hard_limit = resource.getrlimit(resource.RLIMIT_NOFILE)
soft_limit = min(max(self.OPEN_MAX, old_soft_limit), hard_limit)
resource.setrlimit(resource.RLIMIT_NOFILE, (soft_limit, hard_limit))
+
+ def make_queue(self):
+ if self._manager is None:
+ self._manager = multiprocessing.Manager()
+ return self._manager.Queue()
diff --git a/src/buildstream/_platform/platform.py b/src/buildstream/_platform/platform.py
index 42c379e..d0a21a1 100644
--- a/src/buildstream/_platform/platform.py
+++ b/src/buildstream/_platform/platform.py
@@ -17,6 +17,7 @@
# Authors:
# Tristan Maat <tr...@codethink.co.uk>
+import multiprocessing
import os
import platform
import sys
@@ -133,6 +134,9 @@ class Platform():
uname_machine = platform.uname().machine
return Platform.canonicalize_arch(uname_machine)
+ def make_queue(self):
+ return multiprocessing.Queue()
+
##################################################################
# Sandbox functions #
##################################################################
diff --git a/src/buildstream/_platform/win32.py b/src/buildstream/_platform/win32.py
index db70e4a..8844973 100644
--- a/src/buildstream/_platform/win32.py
+++ b/src/buildstream/_platform/win32.py
@@ -1,4 +1,4 @@
-import os
+import multiprocessing
from .._exceptions import PlatformError
from ..sandbox import SandboxNone
@@ -9,8 +9,10 @@ from . import Platform
class Win32(Platform):
def __init__(self):
-
super().__init__()
+ if None is multiprocessing.get_start_method(allow_none=True):
+ multiprocessing.set_start_method('spawn')
+ self._manager = None
def create_sandbox(self, *args, **kwargs):
kwargs['dummy_reason'] = \
@@ -29,3 +31,8 @@ class Win32(Platform):
def maximize_open_file_limit(self):
pass
+
+ def make_queue(self):
+ if self._manager is None:
+ self._manager = multiprocessing.Manager()
+ return self._manager.Queue()
diff --git a/src/buildstream/_scheduler/jobs/job.py b/src/buildstream/_scheduler/jobs/job.py
index 9864e4a..f192886 100644
--- a/src/buildstream/_scheduler/jobs/job.py
+++ b/src/buildstream/_scheduler/jobs/job.py
@@ -33,6 +33,7 @@ import traceback
from ..._exceptions import ImplError, BstError, set_last_task_error, SkipJob
from ..._message import Message, MessageType, unconditional_messages
from ... import _signals, utils
+from ..._platform import Platform
from .jobpickler import pickle_child_job, unpickle_child_job
@@ -184,8 +185,8 @@ class Job():
# Starts the job.
#
def start(self):
-
- self._queue = self._scheduler.manager.Queue()
+ platform = Platform.get_platform()
+ self._queue = platform.make_queue()
self._tries += 1
self._parent_start_listening()
diff --git a/src/buildstream/_scheduler/scheduler.py b/src/buildstream/_scheduler/scheduler.py
index 7f876b5..86510c6 100644
--- a/src/buildstream/_scheduler/scheduler.py
+++ b/src/buildstream/_scheduler/scheduler.py
@@ -72,10 +72,6 @@ class Scheduler():
interrupt_callback=None,
ticker_callback=None):
- import multiprocessing
- multiprocessing.set_start_method('spawn')
- self.manager = multiprocessing.Manager()
-
#
# Public members
#
[buildstream] 11/22: WIP: win32: _platform/win32: add support for
win32
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 76efcbc4adb10396dd8f5628be1815a773193945
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Apr 2 13:14:49 2019 +0100
WIP: win32: _platform/win32: add support for win32
Copy the approach of 'Darwin' and provide a SandboxDummy.
---
src/buildstream/_platform/platform.py | 4 ++++
src/buildstream/_platform/win32.py | 31 +++++++++++++++++++++++++++++++
2 files changed, 35 insertions(+)
diff --git a/src/buildstream/_platform/platform.py b/src/buildstream/_platform/platform.py
index 5f2b708..42c379e 100644
--- a/src/buildstream/_platform/platform.py
+++ b/src/buildstream/_platform/platform.py
@@ -48,6 +48,8 @@ class Platform():
backend = 'linux'
elif sys.platform.startswith('darwin'):
backend = 'darwin'
+ elif sys.platform == 'win32':
+ backend = 'win32'
else:
backend = 'unix'
@@ -55,6 +57,8 @@ class Platform():
from .linux import Linux as PlatformImpl # pylint: disable=cyclic-import
elif backend == 'darwin':
from .darwin import Darwin as PlatformImpl # pylint: disable=cyclic-import
+ elif backend == 'win32':
+ from .win32 import Win32 as PlatformImpl # pylint: disable=cyclic-import
elif backend == 'unix':
from .unix import Unix as PlatformImpl # pylint: disable=cyclic-import
else:
diff --git a/src/buildstream/_platform/win32.py b/src/buildstream/_platform/win32.py
new file mode 100644
index 0000000..db70e4a
--- /dev/null
+++ b/src/buildstream/_platform/win32.py
@@ -0,0 +1,31 @@
+import os
+
+from .._exceptions import PlatformError
+from ..sandbox import SandboxNone
+
+from . import Platform
+
+
+class Win32(Platform):
+
+ def __init__(self):
+
+ super().__init__()
+
+ def create_sandbox(self, *args, **kwargs):
+ kwargs['dummy_reason'] = \
+ "There are no supported sandbox " + \
+ "technologies for Win32 at this time"
+ return SandboxNone(*args, **kwargs)
+
+ def check_sandbox_config(self, config):
+ # Check host os and architecture match
+ if config.build_os != self.get_host_os():
+ raise PlatformError("Configured and host OS don't match.")
+ elif config.build_arch != self.get_host_arch():
+ raise PlatformError("Configured and host architecture don't match.")
+
+ return True
+
+ def maximize_open_file_limit(self):
+ pass
[buildstream] 03/22: job: pickle child jobs on start if
BST_TEST_SUITE
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 22d06c0b9c853226292892d75d0214ee97eb6c9a
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Jul 9 13:39:17 2019 +0100
job: pickle child jobs on start if BST_TEST_SUITE
If we're running BuildStream tests then pickle child jobs.
This ensures that we keep things picklable, whilst we work towards being
able to support platforms that need to use the 'spawn' method of
starting processes.
---
src/buildstream/_scheduler/jobs/job.py | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/src/buildstream/_scheduler/jobs/job.py b/src/buildstream/_scheduler/jobs/job.py
index 87f4619..47a8622 100644
--- a/src/buildstream/_scheduler/jobs/job.py
+++ b/src/buildstream/_scheduler/jobs/job.py
@@ -34,6 +34,8 @@ from ..._exceptions import ImplError, BstError, set_last_task_error, SkipJob
from ..._message import Message, MessageType, unconditional_messages
from ... import _signals, utils
+from .jobpickler import pickle_child_job
+
# Return code values shutdown of job handling child processes
#
@@ -179,6 +181,9 @@ class Job():
self._task_id,
)
+ if 'BST_TEST_SUITE' in os.environ:
+ pickle_child_job(child_job, self._scheduler.context)
+
self._process = Process(target=child_job.child_action, args=[self._queue])
# Block signals which are handled in the main process such that
[buildstream] 10/22: WIP: _scheduler/jobs/job.py: sort imports
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 7843544c83b18820883b3b91340dfc7da2b3fc43
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Jul 9 11:44:48 2019 +0100
WIP: _scheduler/jobs/job.py: sort imports
---
src/buildstream/_scheduler/jobs/job.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/buildstream/_scheduler/jobs/job.py b/src/buildstream/_scheduler/jobs/job.py
index 8365bd8..cdd403a 100644
--- a/src/buildstream/_scheduler/jobs/job.py
+++ b/src/buildstream/_scheduler/jobs/job.py
@@ -20,14 +20,14 @@
# Tristan Maat <tr...@codethink.co.uk>
# System imports
+import asyncio
+import datetime
import enum
+import multiprocessing
import os
-import sys
import signal
-import datetime
+import sys
import traceback
-import asyncio
-import multiprocessing
# BuildStream toplevel imports
from ..._exceptions import ImplError, BstError, set_last_task_error, SkipJob
[buildstream] 13/22: WIP: sandboxnone: use initial SandboxNone
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit cd8fd5756214f582f719f13ee55f17bbfc62c4be
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Wed Apr 10 18:06:06 2019 +0100
WIP: sandboxnone: use initial SandboxNone
---
src/buildstream/_platform/darwin.py | 4 +-
src/buildstream/sandbox/__init__.py | 1 +
src/buildstream/sandbox/_sandboxnone.py | 67 +++++++++++++++++++++++++++++++++
3 files changed, 70 insertions(+), 2 deletions(-)
diff --git a/src/buildstream/_platform/darwin.py b/src/buildstream/_platform/darwin.py
index 282a5b4..86820a3 100644
--- a/src/buildstream/_platform/darwin.py
+++ b/src/buildstream/_platform/darwin.py
@@ -18,7 +18,7 @@
import os
import resource
-from ..sandbox import SandboxDummy
+from ..sandbox import SandboxNone
from .platform import Platform
@@ -32,7 +32,7 @@ class Darwin(Platform):
kwargs['dummy_reason'] = \
"OSXFUSE is not supported and there are no supported sandbox " + \
"technologies for MacOS at this time"
- return SandboxDummy(*args, **kwargs)
+ return SandboxNone(*args, **kwargs)
def check_sandbox_config(self, config):
# Accept all sandbox configs as it's irrelevant with the dummy sandbox (no Sandbox.run).
diff --git a/src/buildstream/sandbox/__init__.py b/src/buildstream/sandbox/__init__.py
index 5966d19..6544348 100644
--- a/src/buildstream/sandbox/__init__.py
+++ b/src/buildstream/sandbox/__init__.py
@@ -20,3 +20,4 @@
from .sandbox import Sandbox, SandboxFlags, SandboxCommandError
from ._sandboxremote import SandboxRemote
from ._sandboxdummy import SandboxDummy
+from ._sandboxnone import SandboxNone
diff --git a/src/buildstream/sandbox/_sandboxnone.py b/src/buildstream/sandbox/_sandboxnone.py
new file mode 100644
index 0000000..e95a7b9
--- /dev/null
+++ b/src/buildstream/sandbox/_sandboxnone.py
@@ -0,0 +1,67 @@
+#
+# Copyright (C) 2019 Bloomberg Finance LP
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
+#
+# Authors:
+# Angelos Evripiotis <je...@bloomberg.net>
+
+import pathlib
+import pprint
+import subprocess
+
+from .._exceptions import SandboxError
+from .sandbox import Sandbox
+
+
+class SandboxNone(Sandbox):
+
+ def __init__(self, *args, **kwargs):
+ # TODO: don't require a dict copy.
+ kwargs = kwargs.copy()
+ kwargs['allow_real_directory'] = True
+
+ super().__init__(*args, **kwargs)
+
+ uid = self._get_config().build_uid
+ gid = self._get_config().build_gid
+ if uid != 0 or gid != 0:
+ raise SandboxError("Chroot sandboxes cannot specify a non-root uid/gid "
+ "({},{} were supplied via config)".format(uid, gid))
+
+ self.mount_map = None
+
+ def _run(self, command, flags, *, cwd, env):
+
+ install_path = pathlib.Path(self.get_directory()) / 'buildstream-install'
+
+ env = env.copy()
+ env['BST_INSTALLPATH'] = str(install_path)
+
+ # TODO: figure out what to do with 'flags'.
+
+ # TODO: do this in a robust way.
+ if cwd.startswith("/"):
+ cwd = cwd[1:]
+
+ # pprint.pprint(env)
+
+ path = pathlib.Path(self.get_directory()) / cwd
+ print('run', command, 'in', path)
+ result = subprocess.run(command, cwd=path, env=env)
+
+ # out = pathlib.Path(self.get_directory()) / 'buildstream-install'
+ # out.mkdir(exist_ok=True)
+
+ return result.returncode
[buildstream] 01/22: optionpool: make OptionPool picklable
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 0ca2aa605b813d0012921ea82f0687db2b2f87e3
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Thu Jun 27 14:04:52 2019 +0100
optionpool: make OptionPool picklable
In order to enable the 'spawn' method of creating jobs, make sure that
OptionPool is pickable, otherwise it cannot be sent to the job process.
---
src/buildstream/_options/optionpool.py | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/src/buildstream/_options/optionpool.py b/src/buildstream/_options/optionpool.py
index de3af3e..d5dea39 100644
--- a/src/buildstream/_options/optionpool.py
+++ b/src/buildstream/_options/optionpool.py
@@ -56,6 +56,18 @@ class OptionPool():
self._environment = jinja2.Environment(undefined=jinja2.StrictUndefined)
self._environment.globals = []
+ def __getstate__(self):
+ state = self.__dict__.copy()
+ del state['_environment']
+ return state
+
+ def __setstate__(self, state):
+ self.__dict__.update(state)
+
+ # jinja2 environment, with default globals cleared out of the way
+ self._environment = jinja2.Environment(undefined=jinja2.StrictUndefined)
+ self._environment.globals = []
+
# load()
#
# Loads the options described in the project.conf
[buildstream] 14/22: WIP: win32: job: signals / win32
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 0fa65b0bf18c86dd0b61576b6ddab09836a62a9d
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Apr 2 13:27:37 2019 +0100
WIP: win32: job: signals / win32
---
src/buildstream/_scheduler/jobs/job.py | 12 ++++++------
src/buildstream/_scheduler/scheduler.py | 4 +++-
src/buildstream/_signals.py | 4 ++++
3 files changed, 13 insertions(+), 7 deletions(-)
diff --git a/src/buildstream/_scheduler/jobs/job.py b/src/buildstream/_scheduler/jobs/job.py
index 480a76c..9864e4a 100644
--- a/src/buildstream/_scheduler/jobs/job.py
+++ b/src/buildstream/_scheduler/jobs/job.py
@@ -211,7 +211,7 @@ class Job():
# the child process does not inherit the parent's state, but the main
# process will be notified of any signal after we launch the child.
#
- with _signals.blocked([signal.SIGINT, signal.SIGTSTP, signal.SIGTERM], ignore=False):
+ with _signals.blocked([signal.SIGINT, signal.SIGTERM], ignore=False):
self._process.start()
# Wait for the child task to complete.
@@ -692,15 +692,15 @@ class ChildJob():
# This avoids some SIGTSTP signals from grandchildren
# getting propagated up to the master process
- os.setsid()
+ #os.setsid()
# First set back to the default signal handlers for the signals
# we handle, and then clear their blocked state.
#
- signal_list = [signal.SIGTSTP, signal.SIGTERM]
- for sig in signal_list:
- signal.signal(sig, signal.SIG_DFL)
- signal.pthread_sigmask(signal.SIG_UNBLOCK, signal_list)
+ #signal_list = [signal.SIGTSTP, signal.SIGTERM]
+ #for sig in signal_list:
+ # signal.signal(sig, signal.SIG_DFL)
+ #signal.pthread_sigmask(signal.SIG_UNBLOCK, signal_list)
# Assign the queue we passed across the process boundaries
#
diff --git a/src/buildstream/_scheduler/scheduler.py b/src/buildstream/_scheduler/scheduler.py
index ae20fcd..7f876b5 100644
--- a/src/buildstream/_scheduler/scheduler.py
+++ b/src/buildstream/_scheduler/scheduler.py
@@ -202,7 +202,7 @@ class Scheduler():
# Block this until we're finished terminating jobs,
# this will remain blocked forever.
- signal.pthread_sigmask(signal.SIG_BLOCK, [signal.SIGINT])
+ #signal.pthread_sigmask(signal.SIG_BLOCK, [signal.SIGINT])
# jobs_suspended()
#
@@ -575,11 +575,13 @@ class Scheduler():
# Connects our signal handler event callbacks to the mainloop
#
def _connect_signals(self):
+ return
self.loop.add_signal_handler(signal.SIGINT, self._interrupt_event)
self.loop.add_signal_handler(signal.SIGTERM, self._terminate_event)
self.loop.add_signal_handler(signal.SIGTSTP, self._suspend_event)
def _disconnect_signals(self):
+ return
self.loop.remove_signal_handler(signal.SIGINT)
self.loop.remove_signal_handler(signal.SIGTSTP)
self.loop.remove_signal_handler(signal.SIGTERM)
diff --git a/src/buildstream/_signals.py b/src/buildstream/_signals.py
index 2df2c79..733f28e 100644
--- a/src/buildstream/_signals.py
+++ b/src/buildstream/_signals.py
@@ -135,6 +135,8 @@ def suspend_handler(sig, frame):
#
@contextmanager
def suspendable(suspend_callback, resume_callback):
+ yield
+ return
global suspendable_stack # pylint: disable=global-statement
outermost = bool(not suspendable_stack)
@@ -164,6 +166,8 @@ def suspendable(suspend_callback, resume_callback):
#
@contextmanager
def blocked(signal_list, ignore=True):
+ yield
+ return
with ExitStack() as stack:
[buildstream] 12/22: WIP: win32: job: replace add_child_handler
with thread
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit da883afc0077da39f451e5f9f32cfe20590d5807
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Apr 2 13:26:08 2019 +0100
WIP: win32: job: replace add_child_handler with thread
---
src/buildstream/_scheduler/jobs/job.py | 18 ++++++++++++++++--
1 file changed, 16 insertions(+), 2 deletions(-)
diff --git a/src/buildstream/_scheduler/jobs/job.py b/src/buildstream/_scheduler/jobs/job.py
index cdd403a..480a76c 100644
--- a/src/buildstream/_scheduler/jobs/job.py
+++ b/src/buildstream/_scheduler/jobs/job.py
@@ -47,6 +47,21 @@ class _ReturnCode(enum.IntEnum):
SKIPPED = 3
+def _call_on_waitpid_threadfun(running_loop, process, callback):
+ process.join()
+ running_loop.call_soon_threadsafe(callback, process.pid, process.exitcode)
+
+
+def call_on_waitpid(running_loop, pid, callback):
+ import threading
+ t = threading.Thread(
+ target=_call_on_waitpid_threadfun,
+ args=(running_loop, pid, callback)
+ )
+ t.start()
+ return t
+
+
# JobStatus:
#
# The job completion status, passed back through the
@@ -221,8 +236,7 @@ class Job():
# an event loop callback. Otherwise, if the job completes too fast, then
# the callback is called immediately.
#
- self._watcher = asyncio.get_child_watcher()
- self._watcher.add_child_handler(self._process.pid, self._parent_child_completed)
+ self._watcher = call_on_waitpid(self._scheduler.loop, self._process, self._parent_child_completed)
# terminate()
#
[buildstream] 07/22: WIP: bst-job-replay
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 77268116883f49937318388ee846beb292a236d6
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Jul 9 11:38:48 2019 +0100
WIP: bst-job-replay
---
setup.py | 3 ++-
src/buildstream/_jobreplay.py | 15 +++++++++++++++
src/buildstream/_scheduler/jobs/jobpickler.py | 4 ++++
3 files changed, 21 insertions(+), 1 deletion(-)
diff --git a/setup.py b/setup.py
index ab3c6f3..af81886 100755
--- a/setup.py
+++ b/setup.py
@@ -152,7 +152,8 @@ bst_install_entry_points = {
if not os.environ.get('BST_ARTIFACTS_ONLY', ''):
check_for_bwrap()
bst_install_entry_points['console_scripts'] += [
- 'bst = buildstream._frontend:cli'
+ 'bst = buildstream._frontend:cli',
+ 'bst-job-replay = buildstream._jobreplay:cli',
]
#####################################################
diff --git a/src/buildstream/_jobreplay.py b/src/buildstream/_jobreplay.py
new file mode 100644
index 0000000..1c05324
--- /dev/null
+++ b/src/buildstream/_jobreplay.py
@@ -0,0 +1,15 @@
+import multiprocessing
+
+import click
+
+from ._scheduler.jobs.job import _unpickle_child_job
+
+
+@click.command(name='bst-job-replay', short_help="Replay a bst job")
+@click.argument('replayfile', type=click.File("rb"))
+def cli(replayfile):
+ job = _unpickle_child_job(replayfile)
+ queue = multiprocessing.Queue()
+ job._queue = queue
+ job._scheduler_context.set_message_handler(job._child_message_handler)
+ job.child_process()
diff --git a/src/buildstream/_scheduler/jobs/jobpickler.py b/src/buildstream/_scheduler/jobs/jobpickler.py
index 5c1742f..3ce2806 100644
--- a/src/buildstream/_scheduler/jobs/jobpickler.py
+++ b/src/buildstream/_scheduler/jobs/jobpickler.py
@@ -143,6 +143,10 @@ def pickle_child_job(child_job, context):
pickler.dump(child_job)
data.seek(0)
+ path = f"{child_job.action_name}_{child_job._task_id}"
+ with open(path, "wb") as f:
+ f.write(data.getvalue())
+
return data
[buildstream] 05/22: WIP: jobpickler, _{,un}reduce_plugincontext
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit e9e61f34c9977b2004050de1b2fa108b6493e7c3
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Jul 9 17:35:32 2019 +0100
WIP: jobpickler, _{,un}reduce_plugincontext
---
src/buildstream/_plugincontext.py | 13 -------------
src/buildstream/_scheduler/jobs/jobpickler.py | 24 ++++++++++++++++++++++++
2 files changed, 24 insertions(+), 13 deletions(-)
diff --git a/src/buildstream/_plugincontext.py b/src/buildstream/_plugincontext.py
index 60d2475..c058c58 100644
--- a/src/buildstream/_plugincontext.py
+++ b/src/buildstream/_plugincontext.py
@@ -68,19 +68,6 @@ class PluginContext():
self._alternate_sources = {}
self._format_versions = format_versions
- def __getstate__(self):
- import copy
- state = copy.copy(self.__dict__)
- del state['_site_source']
- state['_types'] = {}
- return state
-
- def __setstate__(self, state):
- self.__dict__ = state
- self._site_source = self._plugin_base.make_plugin_source(
- searchpath=self._site_plugin_path,
- identifier='site_plugin-' + self._identifier)
-
# lookup():
#
# Fetches a type loaded from a plugin in this plugin context
diff --git a/src/buildstream/_scheduler/jobs/jobpickler.py b/src/buildstream/_scheduler/jobs/jobpickler.py
index 5c1742f..841037c 100644
--- a/src/buildstream/_scheduler/jobs/jobpickler.py
+++ b/src/buildstream/_scheduler/jobs/jobpickler.py
@@ -28,6 +28,7 @@ from ..._protos.buildstream.v2.artifact_pb2 import Artifact as ArtifactProto
from ... import Element, Source
from ..._loader import Loader
from ..._messenger import Messenger
+from ..._plugincontext import PluginContext
def _reduce_artifact_proto(instance):
@@ -53,6 +54,7 @@ def _reduce_loader(instance):
# time that seems just right is here, when preparing the child process'
# copy of the Loader.
#
+ # TODO: move this reduce func to loader.py
del state['_fetch_subprojects']
return (Loader.__new__, (Loader,), state)
@@ -72,11 +74,32 @@ def _reduce_messenger(instance):
# which removes and restores the _message_handler. This wouldn't require
# access to private details of Messenger.
#
+ # TODO: move this reduce func to messenger.py
del state['_message_handler']
return (Messenger.__new__, (Messenger,), state)
+def _reduce_plugincontext(instance):
+ assert isinstance(instance, PluginContext)
+ state = instance.__dict__.copy()
+ del state['_site_source']
+ state['_types'] = {}
+
+ # TODO: move this reduce func to plugincontext.py
+
+ return (_unreduce_plugincontext, (state,))
+
+
+def _unreduce_plugincontext(state):
+ instance = PluginContext()
+ instance.__dict__ = state
+ instance._site_source = instance._plugin_base.make_plugin_source(
+ searchpath=instance._site_plugin_path,
+ identifier='site_plugin-' + instance._identifier)
+ return instance
+
+
def _reduce_element(element):
assert isinstance(element, Element)
meta_kind = element._meta_kind
@@ -139,6 +162,7 @@ def pickle_child_job(child_job, context):
pickler.dispatch_table[ArtifactProto] = _reduce_artifact_proto
pickler.dispatch_table[Loader] = _reduce_loader
pickler.dispatch_table[Messenger] = _reduce_messenger
+ pickler.dispatch_table[PluginContext] = _reduce_plugincontext
pickler.dump(child_job)
data.seek(0)
[buildstream] 22/22: TEMP: INFO log pickle length and time
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit d2d5dfe12208a2dacb205602636da2397a060491
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Jul 9 13:07:56 2019 +0100
TEMP: INFO log pickle length and time
---
src/buildstream/_scheduler/jobs/job.py | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/src/buildstream/_scheduler/jobs/job.py b/src/buildstream/_scheduler/jobs/job.py
index f192886..a47dbd1 100644
--- a/src/buildstream/_scheduler/jobs/job.py
+++ b/src/buildstream/_scheduler/jobs/job.py
@@ -23,10 +23,12 @@
import asyncio
import datetime
import enum
+import io
import multiprocessing
import os
import signal
import sys
+import time
import traceback
# BuildStream toplevel imports
@@ -202,18 +204,30 @@ class Job():
self._task_id,
)
+ then = time.time()
pickled = pickle_child_job(child_job, self._scheduler.context)
+ now = time.time()
+ pickled.seek(0, io.SEEK_END)
+ self.message(MessageType.INFO, "pickled len: {:,}".format(pickled.tell()))
+ self.message(MessageType.INFO, "pickle time: {}s".format(round(now - then, 2)))
+ pickled.seek(0)
+ then = time.time()
self._process = Process(
target=_do_pickled_child_job,
args=[pickled, self._queue],
)
+ now = time.time()
+ self.message(MessageType.INFO, "make process: {}s".format(round(now - then, 2)))
# Block signals which are handled in the main process such that
# the child process does not inherit the parent's state, but the main
# process will be notified of any signal after we launch the child.
#
+ then = time.time()
with _signals.blocked([signal.SIGINT, signal.SIGTERM], ignore=False):
self._process.start()
+ now = time.time()
+ self.message(MessageType.INFO, "start process: {}s".format(round(now - then, 2)))
# Wait for the child task to complete.
#
[buildstream] 20/22: TEMP: testpickle
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit b852196b3ef696da62242379160c9c1e387cab16
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Jul 9 13:00:15 2019 +0100
TEMP: testpickle
---
src/buildstream/_scheduler/jobs/jobpickler.py | 5 +
src/buildstream/testpickle.py | 224 ++++++++++++++++++++++++++
2 files changed, 229 insertions(+)
diff --git a/src/buildstream/_scheduler/jobs/jobpickler.py b/src/buildstream/_scheduler/jobs/jobpickler.py
index 3ce2806..2e0ee5c 100644
--- a/src/buildstream/_scheduler/jobs/jobpickler.py
+++ b/src/buildstream/_scheduler/jobs/jobpickler.py
@@ -140,6 +140,11 @@ def pickle_child_job(child_job, context):
pickler.dispatch_table[Loader] = _reduce_loader
pickler.dispatch_table[Messenger] = _reduce_messenger
+ import buildstream.testpickle
+ test_pickler = buildstream.testpickle.TestPickler()
+ test_pickler.dispatch_table = pickler.dispatch_table.copy()
+ test_pickler.test_dump(child_job)
+
pickler.dump(child_job)
data.seek(0)
diff --git a/src/buildstream/testpickle.py b/src/buildstream/testpickle.py
new file mode 100644
index 0000000..52f351b
--- /dev/null
+++ b/src/buildstream/testpickle.py
@@ -0,0 +1,224 @@
+import copyreg
+import io
+import pickle
+
+
+class _C:
+ def f(self):
+ pass
+
+
+class TestPickler:
+ def __init__(self):
+ self.dispatch_table = copyreg.dispatch_table.copy()
+ self.visited = set()
+
+ def test_dump(self, obj):
+ import bdb
+ try:
+ self._test_pickle(obj)
+ except bdb.BdbQuit:
+ raise
+ except Exception:
+ breakpoint()
+ raise
+
+ def _pickle(self, obj):
+ data = io.BytesIO()
+ pickler = pickle.Pickler(data)
+ pickler.dispatch_table = self.dispatch_table.copy()
+ pickler.dump(obj)
+
+ def _test_pickle(self, x, indent=0):
+
+ if indent > 50:
+ print()
+ print('*' * 80)
+ print("Indent level too high:", indent)
+ print('*' * 80)
+ print()
+ raise Exception("Indent level too high:", indent)
+
+ def prefix_print(*messages):
+ print(". " * indent + f"({type(x).__name__}):", *messages)
+
+ if id(x) in self.visited:
+ prefix_print(".. skipping already visited")
+ return
+
+ self.visited.add(id(x))
+
+ import bdb
+ try:
+ self._pickle(x)
+ except bdb.BdbQuit:
+ raise
+ except Exception as e:
+ prefix_print(f'({x}): does not pickle, recursing.', str(e), repr(e), ':.:')
+ else:
+ prefix_print(f'({x}): does pickle, skipping.')
+ return
+
+ if type(x) in self.dispatch_table:
+ unreducer, args, state = self.dispatch_table[type(x)](x)
+ prefix_print(f'({type(x)}): is in the dispatch_table, pickling reduced state.')
+ self._test_pickle(state, indent + 1)
+ return
+
+ if type(x) == type(_C().f):
+ prefix_print(f'method {x.__func__.__name__}')
+ try:
+ if x.__self__ is None:
+ value = x.__class__
+ else:
+ value = x.__self__
+ self._test_pickle(value, indent + 1)
+ except:
+ prefix_print(f"while pickling item method {x.__func__.__name__}: '{x}'.")
+ raise
+
+ if type(x).__name__ in ['method', 'instancemethod']:
+ prefix_print(".. skipping method")
+ return
+
+ if type(x).__name__ in ['list', 'tuple', 'set']:
+ prefix_print('... len', len(x))
+ for key, value in enumerate(x):
+ prefix_print(f'[{key}]')
+ try:
+ self._test_pickle(value, indent + 1)
+ except:
+ prefix_print(f"while pickling item {key}: {type(x).__name__}: '{x}'.")
+ raise
+ return
+
+ # if type(x).__name__ == 'function':
+ # prefix_print("function?")
+ # raise Exception()
+
+ # if type(x).__name__ == 'module':
+ # prefix_print(".. module")
+ # pickle_func(x)
+ # return
+
+ # TODO: make these work properly.
+ # if type(x).__name__ in ['SourceFactory', 'ElementFactory', 'Environment']:
+ # prefix_print(".. skipping")
+ # return
+ if type(x).__name__ in ['_UnixSelectorEventLoop', 'AuthenticationString', 'SyncManager']:
+ prefix_print(".. skipping")
+ return
+
+ if type(x).__name__ == 'dict':
+ prefix_print("...", x.keys())
+ for key, value in x.items():
+ prefix_print(f'[{key}]')
+ try:
+ self._test_pickle(value, indent + 1)
+ except:
+ prefix_print(f"while pickling ['{key}'].")
+ raise
+ return
+
+ # TODO: we need to make the generators work too, or ideally replace them.
+ # if type(x).__name__ == 'generator':
+ # prefix_print(".. skipping generator")
+ # return
+
+ # TODO: we need to make the weakrefs work properly.
+ if type(x).__name__ == 'weakref':
+ prefix_print(".. dereferencing weakref")
+ try:
+ self._test_pickle(x(), indent)
+ except:
+ prefix_print(f"while pickling weakref {x}.")
+ raise
+ return
+
+ try:
+ value = x.__getstate__()
+ except AttributeError:
+ pass
+ else:
+ prefix_print("... __getstate__")
+ try:
+ self._test_pickle(value, indent + 1)
+ except:
+ prefix_print(f"while pickling a __getstate__.")
+ raise
+ return
+
+ try:
+ x.__dict__
+ except AttributeError:
+ pass
+ else:
+ prefix_print("...", x.__dict__.keys())
+ for key, value in x.__dict__.items():
+ prefix_print(f'__dict__["{key}"]')
+ try:
+ self._test_pickle(value, indent + 1)
+ except:
+ prefix_print(f"while pickling member ['{key}'].")
+ raise
+ return
+
+ try:
+ x.__slots__
+ except AttributeError:
+ pass
+ else:
+ prefix_print("...", x.__slots__)
+ for key in x.__slots__:
+ value = getattr(x, key)
+ prefix_print(f'__slots__["{key}"]')
+ try:
+ self._test_pickle(value, indent + 1)
+ except:
+ prefix_print(f"while pickling member '{key}'.")
+ raise
+ return
+
+ print()
+ print('*' * 80)
+ print("!!! Unhandled pickle case:", x, type(x))
+ print('*' * 80)
+ print()
+ # raise Exception("Unhandled pickle case")
+ # prefix_print(x)
+ # self._pickle(x)
+
+
+def test_pickle_direct(x):
+ import io
+ import pickle
+ import multiprocessing.reduction
+
+ # Note that we should expect to see this complaint if we are not in a
+ # multiprocessing spawning_popen context, this will be fine when we're
+ # actually spawning:
+ #
+ # Pickling an AuthenticationString object is disallowed for
+ # security reasons.
+ #
+ # https://github.com/python/cpython/blob/master/Lib/multiprocessing/process.py#L335
+ #
+
+ # Suppress the complaint by pretending we're in a spawning context.
+ # https://github.com/python/cpython/blob/a8474d025cab794257d2fd0bea67840779b9351f/Lib/multiprocessing/popen_spawn_win32.py#L91
+ import multiprocessing.context
+ multiprocessing.context.set_spawning_popen("PPPPPopen")
+
+ data = io.BytesIO()
+
+ # Try to simulate what multiprocessing will do.
+ # https://github.com/python/cpython/blob/master/Lib/multiprocessing/reduction.py
+ try:
+ multiprocessing.reduction.dump(x, data)
+ except:
+ # breakpoint()
+ raise
+ finally:
+ multiprocessing.context.set_spawning_popen(None)
+
+ return data
[buildstream] 15/22: WIP: win32: ImportElement - fix separators
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 1ea5516769f63ee51a0a79cc13967eb7cbee1e3d
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Apr 2 13:36:29 2019 +0100
WIP: win32: ImportElement - fix separators
---
src/buildstream/plugins/elements/import.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/buildstream/plugins/elements/import.py b/src/buildstream/plugins/elements/import.py
index 61e353d..2861674 100644
--- a/src/buildstream/plugins/elements/import.py
+++ b/src/buildstream/plugins/elements/import.py
@@ -82,10 +82,10 @@ class ImportElement(Element):
outputdir = rootdir.descend('output', create=True)
# The directory to grab
- inputdir = inputdir.descend(*self.source.strip(os.sep).split(os.sep))
+ inputdir = inputdir.descend(*self.source.strip('/').split('/'))
# The output target directory
- outputdir = outputdir.descend(*self.target.strip(os.sep).split(os.sep), create=True)
+ outputdir = outputdir.descend(*self.target.strip('/').split('/'), create=True)
if inputdir.is_empty():
raise ElementError("{}: No files were found inside directory '{}'"
@@ -95,7 +95,7 @@ class ImportElement(Element):
outputdir.import_files(inputdir)
# And we're done
- return '/output'
+ return os.sep + 'output'
def generate_script(self):
build_root = self.get_variable('build-root')
[buildstream] 08/22: element: platform independent _get_normal_name
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 97cf62334d40fad31e650422c28f7074e908f8e0
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Tue Jun 25 14:23:43 2019 +0100
element: platform independent _get_normal_name
Note that windows has two path separators '/' and '\', handle both.
---
src/buildstream/element.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/src/buildstream/element.py b/src/buildstream/element.py
index 7061a27..7c70cfe 100644
--- a/src/buildstream/element.py
+++ b/src/buildstream/element.py
@@ -3196,7 +3196,10 @@ def _overlap_error_detail(f, forbidden_overlap_elements, elements):
# (str): The normalised element name
#
def _get_normal_name(element_name):
- return os.path.splitext(element_name.replace(os.sep, '-'))[0]
+ name_no_sep = element_name.replace(os.sep, '-')
+ if os.altsep is not None:
+ name_no_sep = name_no_sep.replace(os.altsep, '-')
+ return os.path.splitext(name_no_sep)[0]
# _compose_artifact_name():
[buildstream] 16/22: WIP: win32: don't set owner of files
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch aevri/win32_minimal_seemstowork_20190829
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit a8833a41dadd2e279e06937694ae88c9872fa0fc
Author: Angelos Evripiotis <je...@bloomberg.net>
AuthorDate: Wed Apr 17 16:33:44 2019 +0100
WIP: win32: don't set owner of files
---
src/buildstream/element.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/buildstream/element.py b/src/buildstream/element.py
index 7c70cfe..01af123 100644
--- a/src/buildstream/element.py
+++ b/src/buildstream/element.py
@@ -1493,7 +1493,7 @@ class Element(Plugin):
# Ensure deterministic mtime of sources at build time
vdirectory.set_deterministic_mtime()
# Ensure deterministic owners of sources at build time
- vdirectory.set_deterministic_user()
+ #vdirectory.set_deterministic_user()
# _set_required():
#