You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by "sophie-h (via GitHub)" <gi...@apache.org> on 2024/03/13 13:25:17 UTC
[PR] cargo: Add support for git dependencies [buildstream-plugins]
sophie-h opened a new pull request, #62:
URL: https://github.com/apache/buildstream-plugins/pull/62
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "gtristan (via GitHub)" <gi...@apache.org>.
gtristan commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1585903820
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -75,6 +85,10 @@
from buildstream import Source, SourceFetcher, SourceError
from buildstream import utils
+import dulwich
Review Comment:
The experimental repo is indeed the wild west, no objections to that.
The issue is that even with python libraries which conform well to pythonic standards of API compatibility, and follow semantic versioning, are not strict enough to be safe to use in a BuildStream plugin strictly speaking, since it is impossible to address both API versions separately (import names have not changed) or install both versions in a single python environment.
This will be equally true of course for the `bst-plugins-experimental` plugins, but we are not concerned with long term compatibility there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "AdrianVovk (via GitHub)" <gi...@apache.org>.
AdrianVovk commented on PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#issuecomment-1998912152
Ah and should probably add Fixes: #10 to the commit messages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1583365050
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +324,284 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ self.repo_mirrored = cargo.mirrored_git_url(repo)
+
+ self.mark_download_url(self.repo_mirrored)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+ repo_url = self._get_url(alias=alias_override)
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {repo_url}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(repo_url)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"This project requests crate {self.name} {self.version} from multiple sources, "
+ "which is incompatible with vendoring since cargo does not support it."
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml:
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
Review Comment:
Right. See https://github.com/python-poetry/tomlkit/blob/master/tomlkit/api.py#L65. load takes either binary or text mode file, dump takes always onl text mode. I suggest using text mode everywhere for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1583360827
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +324,284 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ self.repo_mirrored = cargo.mirrored_git_url(repo)
+
+ self.mark_download_url(self.repo_mirrored)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+ repo_url = self._get_url(alias=alias_override)
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {repo_url}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(repo_url)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"This project requests crate {self.name} {self.version} from multiple sources, "
+ "which is incompatible with vendoring since cargo does not support it."
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml:
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
Review Comment:
I suppose this should be "w", not "bw" based on https://gitlab.gnome.org/GNOME/gnome-build-meta/-/merge_requests/2825#note_2092004
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1530702093
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
FWIW: The typical case in GNOME would probably be that git dependencies pop up and disappear again because the project wants to test out some other components that don't have a release yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "AdrianVovk (via GitHub)" <gi...@apache.org>.
AdrianVovk commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1534728913
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
:+1: ok I like it.
Small concern; it should behave like keys. That way projects can configure a bunch of project-wide `git_mirrors` so it doesn't have to be specified in each element, but also if you _do_ need to configure something per-element you can just configure it
With your proposal right now, the set-up would need to look like:
```
sources:
- kind: cargo
git_mirrors:
(>):
- https://github.com/x/y: https://mirror.example.org/x/y
- https://github.com/a/b: my-alias:a/b
```
to avoid overwriting the project-wide config
Of course, domains don't fit into YAML syntax, so maybe what I'm asking for isn't possible :shrug:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "AdrianVovk (via GitHub)" <gi...@apache.org>.
AdrianVovk commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1536655173
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
Should probably be like this then
```yaml
git_mirrors:
https://github.com/x/y: https://mirror.example.org/x/y
https://github.com/a/b: my-alias:a/b
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "gtristan (via GitHub)" <gi...@apache.org>.
gtristan commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1574257654
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -75,6 +85,10 @@
from buildstream import Source, SourceFetcher, SourceError
from buildstream import utils
+import dulwich
Review Comment:
For buildstream-plugins, we prefer to have no dependencies outside of the standard library.
Some exceptions have been made for tomli and requests, which we expect to be API stable, any unstable APIs we rely on can cause BuildStream projects to be dependent on host specific configurations, and even worse, when junctions are involved this can result in being painted into a corner where multiple projects require different versions of the same external python libraries, and thus cannot be built.
This has caused much issues specifically with `dulwich` being used by the `git_repo` plugin in bst-plugins-experimental, which appears to be a wildly changing API and the specific version of dulwich to use is not very clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1530685691
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
Yeah, sounds legit. @abderrahim and @gtristan were reworking how URL marking works but it doesn't seem it helps here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1528756036
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +310,268 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {self.repo}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(self.repo)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"Cargo doens't support vendoring for identical crate version from different sources {self.name} {self.version}"
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml and "memers":
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
+ tomli_w.dump(crate["config"], f)
+
+ shutil.move(crate["path"], crate_target_dir)
+ else:
+ # No workspaces involed, just reploy complete dir as is
+ shutil.move(tmp_dir, crate_target_dir)
+
+ # Write .cargo-checksum.json required by cargo vendoring
+ with open(os.path.join(crate_target_dir, ".cargo-checksum.json"), "w") as f:
+ json.dump({"files": {}, "package": None}, f)
+
+ shutil.rmtree(tmp_dir, ignore_errors=True)
+
+ # is_cached()
+ #
+ # Get whether we have a local cached version of the git commit
+ #
+ # Returns:
+ # (bool): Whether we are cached or not
+ #
+ def is_cached(self):
+ with Repo(self._get_mirror_dir(), bare=True) as repo:
+ return self.commit.encode() in repo
+
+ # is_resolved()
+ #
+ # Get whether the current git repo is resolved
+ #
+ # Returns:
+ # (bool): Always true since we always have a commit
+ #
+ def is_resolved(self):
+ return True
+
+ ########################################################
+ # Private helpers #
+ ########################################################
+
+ # _get_mirror_dir()
+ #
+ # Gets the local mirror directory for this upstream git repository
+ #
+ def _get_mirror_dir(self):
+ if self.repo.endswith(".git"):
+ norm_url = self.repo[:-4]
+ else:
+ norm_url = self.repo
+
+ return os.path.join(
+ self.cargo.get_mirror_directory(),
+ utils.url_directory_name(norm_url) + ".git",
+ )
+
+ # _get_url()
+ #
+ # Gets the git URL to download this crate from
+ #
+ # Args:
+ # alias (str|None): The URL alias to apply, if any
+ #
+ # Returns:
+ # (str): The URL for this crate
+ #
+ def _get_url(self, alias=None):
+ return self.cargo.translate_url(self.cargo.repo, alias_override=alias)
+
+ # _mirror_repo()
+ #
+ # Returns the mirror repo, initialized if it doesn not exist yet
+ #
+ # Returns:
+ # (Repo): The mirror repo crate
+ #
+ def _mirror_repo(self):
+ try:
+ return Repo.init_bare(self._get_mirror_dir(), mkdir=True)
+ except FileExistsError:
+ return Repo(self._get_mirror_dir(), bare=True)
+
+
+# config_inherit_workspace()
+#
+# Adds inherited values to config
+#
+# Args:
+# config (dict): Crate config
+# workspace_config (dict): Workspace config
+def config_inherit_workspace(config, workspace_config):
+ workspace_deps = workspace_config.get("dependencies")
+ if workspace_deps is not None:
+ dependencies = []
+ for key in ["dependencies", "dev-dependencies", "build-dependencies"]:
+ if key in config:
+ dependencies.append(config[key])
+
+ if "traget" in config:
Review Comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1537588190
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -386,15 +704,24 @@ def track(self, *, previous_sources_dir):
return new_ref
def stage(self, directory):
-
# Stage the crates into the vendor directory
vendor_dir = os.path.join(directory, self.vendor_dir)
+
+ git_repos = set()
for crate in self.crates:
crate.stage(vendor_dir)
+ if isinstance(crate, CrateGit):
+ git_repos.add(crate.repo)
+
+ # Every git repo needs an extra entry in the config
+ # Using a set since the config entry cannot appear twice
+ vendor_cfg_git = ""
+ for repo in git_repos:
+ vendor_cfg_git += _git_vendor_config_template.format(git_repo=repo)
# Stage our vendor config
vendor_config = _default_vendor_config_template.format(
- vendorurl=self.translate_url(self.url), vendordir=self.vendor_dir
+ vendorurl=self.translate_url(self.url), vendordir=self.vendor_dir, git_vendors=vendor_cfg_git
Review Comment:
I'm still pretty sure this must be wrong. Maybe I will find the time to test this outside of STF.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "gtristan (via GitHub)" <gi...@apache.org>.
gtristan commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1574257654
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -75,6 +85,10 @@
from buildstream import Source, SourceFetcher, SourceError
from buildstream import utils
+import dulwich
Review Comment:
For buildstream-plugins, we prefer to have no dependencies outside of the standard library.
Some exceptions have been made for tomli and requests, which we expect to be API stable, any unstable APIs we rely on can cause BuildStream projects to be dependent on host specific configurations, and even worse, when junctions are involved this can result in being painted into a corner where multiple projects require different versions of the same external python libraries, and thus cannot be built in any single python environment.
This has caused much issues specifically with `dulwich` being used by the `git_repo` plugin in bst-plugins-experimental, which appears to be a wildly changing API and the specific version of dulwich to use is not very clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1585492629
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -75,6 +85,10 @@
from buildstream import Source, SourceFetcher, SourceError
from buildstream import utils
+import dulwich
Review Comment:
> For buildstream-plugins, we prefer to have no dependencies outside of the standard library.
Maybe we can just move this as a new plugin to the experimental plugins repo? The git plugin that is actually used is already there and uses dulwich as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1529220840
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
@sophie-h if you want to look at real-world example, https://gitlab.gnome.org/GNOME/gnome-build-meta/-/blob/master/include/aliases.yml#L12 -> https://gitlab.gnome.org/GNOME/gnome-build-meta/-/blob/master/include/mirrors.yml#L10. You as plugin author only need to provide ability to provide aliased URL's. BuildStream expands aliases based on user (or project) configuration and this may include mirrors. freedesktop-sdk and gnome attempt to mirror everything through lorry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1530650012
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -386,15 +704,24 @@ def track(self, *, previous_sources_dir):
return new_ref
def stage(self, directory):
-
# Stage the crates into the vendor directory
vendor_dir = os.path.join(directory, self.vendor_dir)
+
+ git_repos = set()
for crate in self.crates:
crate.stage(vendor_dir)
+ if isinstance(crate, CrateGit):
+ git_repos.add(crate.repo)
+
+ # Every git repo needs an extra entry in the config
+ # Using a set since the config entry cannot appear twice
+ vendor_cfg_git = ""
+ for repo in git_repos:
+ vendor_cfg_git += _git_vendor_config_template.format(git_repo=repo)
# Stage our vendor config
vendor_config = _default_vendor_config_template.format(
- vendorurl=self.translate_url(self.url), vendordir=self.vendor_dir
+ vendorurl=self.translate_url(self.url), vendordir=self.vendor_dir, git_vendors=vendor_cfg_git
Review Comment:
Has this ever been tested? Translating the URL here seems wrong at first glance. Cargo will still be looking up the original URL that's in the Cargo.lock file and doesn't know anything about buildstream using a mirror to download the files and also doesn't need to know. Or am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h closed pull request #62: cargo: Add support for git dependencies
URL: https://github.com/apache/buildstream-plugins/pull/62
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#issuecomment-1996071509
This works for me locally with making a `source checkout` and running cargo manually.
Would be great to get some feedback. Afterwards, I can do some more testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1528742357
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +310,268 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {self.repo}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(self.repo)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"Cargo doens't support vendoring for identical crate version from different sources {self.name} {self.version}"
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml and "memers":
Review Comment:
The second part should just be gone, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1528740566
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +310,268 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {self.repo}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(self.repo)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"Cargo doens't support vendoring for identical crate version from different sources {self.name} {self.version}"
Review Comment:
I went with a variation of the third option. I hope no one hits this issue in reality anyways.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h closed pull request #62: cargo: Add support for git dependencies
URL: https://github.com/apache/buildstream-plugins/pull/62
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1536647157
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
Oh, there is a consistent typo. URLs actually do work as keys so it should be
```yaml
git_mirrors:
https://github.com/x/y:
- https://mirror.example.org/x/y
https://github.com/a/b:
- my-alias:a/b
```
I guess I will try to implement this then next week
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1530049556
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
Aliasing indeed needs to be set on per-git repo level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1529326159
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
> if you want to look at real-world example
I don't see how this is related. The cargo plugin generates the list of dependencies when calling `source track` from the `Cargo.lock` file. There won't be any aliases used in the crate refs (dependencies).
As I understand @AdrianVovk's suggestion, this would change all git pulls to be redirected to a single git host, independent of the actual git host of the original URL. The aliasing mechanism would only be used for this one mirror URL. So that's something different I guess?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1528755151
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -367,17 +644,31 @@ def track(self, *, previous_sources_dir):
for package in lock["package"]:
if "source" not in package:
continue
- new_ref += [{"name": package["name"], "version": str(package["version"]), "sha": package.get("checksum")}]
+
+ ref = {}
+
+ if package["source"][:4] == "git+":
Review Comment:
> missing (=> "registry")
registry entries seem to start with `registry+`
I wanted to be safe to not introduce regressions by requiring this entry. But I can add the check.
> Does cargo support any other transports? Will it ever?
I'm not aware of any
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1583365050
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +324,284 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ self.repo_mirrored = cargo.mirrored_git_url(repo)
+
+ self.mark_download_url(self.repo_mirrored)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+ repo_url = self._get_url(alias=alias_override)
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {repo_url}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(repo_url)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"This project requests crate {self.name} {self.version} from multiple sources, "
+ "which is incompatible with vendoring since cargo does not support it."
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml:
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
Review Comment:
Right. See https://github.com/python-poetry/tomlkit/blob/master/tomlkit/api.py#L65. load takes either binary or text mode file, dump takes always only text mode. I suggest using text mode everywhere for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#issuecomment-2088402167
Closing in favor of https://gitlab.com/BuildStream/bst-plugins-experimental/-/merge_requests/312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1529220840
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
@sophie-h if you want to look at real-world example, https://gitlab.gnome.org/GNOME/gnome-build-meta/-/blob/master/include/aliases.yml#L12 -> https://gitlab.gnome.org/GNOME/gnome-build-meta/-/blob/master/include/mirrors.yml#L10. You as plugin author only need to provide ability to provide aliased URL's. BuildStream expands aliases based on user (or project) configuration and this may (but does not have to) include mirrors. freedesktop-sdk and gnome attempt to mirror everything through lorry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "AdrianVovk (via GitHub)" <gi...@apache.org>.
AdrianVovk commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1529374411
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
IMO you should, yes. BuildStream plugins are generally expected to provide some kind of support for source aliases so that it is possible to mirror the sources if necessary.
> this would change all git pulls to be redirected to a single git host
Yeah, as long as you have a single git host specified. If no host is specified, then we don't redirect.
I think that's an easier solution than finding some way to be able to redirect each cargo dependency individually...
> So that's something different I guess?
Nope, I'm just think what I'm proposing is an easier solution than finding some way to be able to redirect each cargo dependency individually, and it's good enough for the purposes of making it possible to mirror sources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1574354753
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -75,6 +85,10 @@
from buildstream import Source, SourceFetcher, SourceError
from buildstream import utils
+import dulwich
Review Comment:
The bigger problem to be honest is lack of tests with the plugins involved. Even if minimum version is set (like it is now), nothing proves it actually works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1585520796
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +324,284 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ self.repo_mirrored = cargo.mirrored_git_url(repo)
+
+ self.mark_download_url(self.repo_mirrored)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+ repo_url = self._get_url(alias=alias_override)
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {repo_url}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(repo_url)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"This project requests crate {self.name} {self.version} from multiple sources, "
+ "which is incompatible with vendoring since cargo does not support it."
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml:
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
Review Comment:
That was fallout from switching from tomli-w to tomlkit. It's fixed now.
> I suggest using text mode everywhere for consistency.
tomllib that's used for loading does only support binary mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1585540047
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +324,284 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ self.repo_mirrored = cargo.mirrored_git_url(repo)
+
+ self.mark_download_url(self.repo_mirrored)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+ repo_url = self._get_url(alias=alias_override)
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {repo_url}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(repo_url)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"This project requests crate {self.name} {self.version} from multiple sources, "
+ "which is incompatible with vendoring since cargo does not support it."
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml:
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
Review Comment:
> Read a TOML file. The first argument should be a readable and binary file object.
https://docs.python.org/3/library/tomllib.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "AdrianVovk (via GitHub)" <gi...@apache.org>.
AdrianVovk commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1530694211
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
Please consider the fact that rust projects have _lots_ of dependencies...
I'd rather not have to configure the alias mappings for dozens of rust crates each time I want to use the cargo plugin
Counterpoint: I guess rust projects don't have that many git dependencies necessarily, and we could configure the URL mapping globally, project-wide to avoid duplicating work?
Just something to keep in mind
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "AdrianVovk (via GitHub)" <gi...@apache.org>.
AdrianVovk commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1525705300
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +310,268 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {self.repo}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(self.repo)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"Cargo doens't support vendoring for identical crate version from different sources {self.name} {self.version}"
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml and "memers":
Review Comment:
typo
but also the condition doesn't make sense to me
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -367,17 +644,31 @@ def track(self, *, previous_sources_dir):
for package in lock["package"]:
if "source" not in package:
continue
- new_ref += [{"name": package["name"], "version": str(package["version"]), "sha": package.get("checksum")}]
+
+ ref = {}
+
+ if package["source"][:4] == "git+":
Review Comment:
perhaps it's worth splitting on the plus, then checking if it's git, missing (=> "registry"), or something else (=> error)
Does cargo support any other transports? Will it ever?
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +310,268 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {self.repo}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(self.repo)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"Cargo doens't support vendoring for identical crate version from different sources {self.name} {self.version}"
Review Comment:
This error message isn't particularly clear, IMO, but it's hard to come up with a one-liner that covers it in a non-confusing way...
- "Crate {self.name} {self.version} is vendored from multiple sources, which is unsupported"
- "Cannot stage {self.name} {self.version} because it was already staged by a different fetcher"
- "This project requests crate {self.name} {self.version} from multiple sources, which is incompatible with vendoring"
:shrug:
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +310,268 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {self.repo}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(self.repo)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"Cargo doens't support vendoring for identical crate version from different sources {self.name} {self.version}"
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml and "memers":
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
+ tomli_w.dump(crate["config"], f)
+
+ shutil.move(crate["path"], crate_target_dir)
+ else:
+ # No workspaces involed, just reploy complete dir as is
Review Comment:
typo
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +310,268 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {self.repo}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(self.repo)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"Cargo doens't support vendoring for identical crate version from different sources {self.name} {self.version}"
Review Comment:
This error message & check should probably be duplicated in the other fetcher too, because right now depending on the ordering of the `Cargo.lock` file you'll get inconsistent behavior: either an error message, or the git source will silently get overwritten by the tarball
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +310,268 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {self.repo}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(self.repo)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"Cargo doens't support vendoring for identical crate version from different sources {self.name} {self.version}"
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml and "memers":
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
+ tomli_w.dump(crate["config"], f)
+
+ shutil.move(crate["path"], crate_target_dir)
+ else:
+ # No workspaces involed, just reploy complete dir as is
+ shutil.move(tmp_dir, crate_target_dir)
+
+ # Write .cargo-checksum.json required by cargo vendoring
+ with open(os.path.join(crate_target_dir, ".cargo-checksum.json"), "w") as f:
+ json.dump({"files": {}, "package": None}, f)
+
+ shutil.rmtree(tmp_dir, ignore_errors=True)
+
+ # is_cached()
+ #
+ # Get whether we have a local cached version of the git commit
+ #
+ # Returns:
+ # (bool): Whether we are cached or not
+ #
+ def is_cached(self):
+ with Repo(self._get_mirror_dir(), bare=True) as repo:
+ return self.commit.encode() in repo
+
+ # is_resolved()
+ #
+ # Get whether the current git repo is resolved
+ #
+ # Returns:
+ # (bool): Always true since we always have a commit
+ #
+ def is_resolved(self):
+ return True
+
+ ########################################################
+ # Private helpers #
+ ########################################################
+
+ # _get_mirror_dir()
+ #
+ # Gets the local mirror directory for this upstream git repository
+ #
+ def _get_mirror_dir(self):
+ if self.repo.endswith(".git"):
+ norm_url = self.repo[:-4]
+ else:
+ norm_url = self.repo
+
+ return os.path.join(
+ self.cargo.get_mirror_directory(),
+ utils.url_directory_name(norm_url) + ".git",
+ )
+
+ # _get_url()
+ #
+ # Gets the git URL to download this crate from
+ #
+ # Args:
+ # alias (str|None): The URL alias to apply, if any
+ #
+ # Returns:
+ # (str): The URL for this crate
+ #
+ def _get_url(self, alias=None):
+ return self.cargo.translate_url(self.cargo.repo, alias_override=alias)
Review Comment:
self.cargo.repo seems incorrect. Maybe you meant self.repo?
You also don't use this function anyway... You should be using it when calling `dulwich.client.get_transport_and_path` above, and possibly elsewhere. Make sure to pass through the `alias_override`.
Once you start doing this, buildstream should start yelling at you about "unaliased sources". I described the fix in the comment about marking the URL
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
The whole alias witchcraft is very poorly documented (from the plugin side, anyway), and I have a tenuous grasp on it as best, mostly from my experience of guess & checking when writing my own modules. That said, I'll take a stab at explaining what's going on because I think I know what's going on here. Someone will have to fact-check me.
Basically you need to warn BuildStream @ element load/parse time about any URLs you will be translating for source aliases later on (i.e. at track time, or at fetch time, etc). So you mark the untranslated URL, and BuildStream won't be surprised by it later, when asked to resolve aliases.
Why bst needs this: I have no idea, sorry. I just know that it does.
The wrinkle here is that you don't actually translate the URL, which means that this is wholly incompatible with buildstream aliases and mirroring. You would be getting warnings about this ("Unaliased source!") but you completely bypass these warnings by never trying to translate the URL.
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +310,268 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {self.repo}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(self.repo)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"Cargo doens't support vendoring for identical crate version from different sources {self.name} {self.version}"
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml and "memers":
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
+ tomli_w.dump(crate["config"], f)
+
+ shutil.move(crate["path"], crate_target_dir)
+ else:
+ # No workspaces involed, just reploy complete dir as is
+ shutil.move(tmp_dir, crate_target_dir)
+
+ # Write .cargo-checksum.json required by cargo vendoring
+ with open(os.path.join(crate_target_dir, ".cargo-checksum.json"), "w") as f:
+ json.dump({"files": {}, "package": None}, f)
+
+ shutil.rmtree(tmp_dir, ignore_errors=True)
+
+ # is_cached()
+ #
+ # Get whether we have a local cached version of the git commit
+ #
+ # Returns:
+ # (bool): Whether we are cached or not
+ #
+ def is_cached(self):
+ with Repo(self._get_mirror_dir(), bare=True) as repo:
+ return self.commit.encode() in repo
+
+ # is_resolved()
+ #
+ # Get whether the current git repo is resolved
+ #
+ # Returns:
+ # (bool): Always true since we always have a commit
+ #
+ def is_resolved(self):
+ return True
+
+ ########################################################
+ # Private helpers #
+ ########################################################
+
+ # _get_mirror_dir()
+ #
+ # Gets the local mirror directory for this upstream git repository
+ #
+ def _get_mirror_dir(self):
+ if self.repo.endswith(".git"):
+ norm_url = self.repo[:-4]
+ else:
+ norm_url = self.repo
+
+ return os.path.join(
+ self.cargo.get_mirror_directory(),
+ utils.url_directory_name(norm_url) + ".git",
+ )
+
+ # _get_url()
+ #
+ # Gets the git URL to download this crate from
+ #
+ # Args:
+ # alias (str|None): The URL alias to apply, if any
+ #
+ # Returns:
+ # (str): The URL for this crate
+ #
+ def _get_url(self, alias=None):
+ return self.cargo.translate_url(self.cargo.repo, alias_override=alias)
+
+ # _mirror_repo()
+ #
+ # Returns the mirror repo, initialized if it doesn not exist yet
+ #
+ # Returns:
+ # (Repo): The mirror repo crate
+ #
+ def _mirror_repo(self):
+ try:
+ return Repo.init_bare(self._get_mirror_dir(), mkdir=True)
+ except FileExistsError:
+ return Repo(self._get_mirror_dir(), bare=True)
+
+
+# config_inherit_workspace()
+#
+# Adds inherited values to config
+#
+# Args:
+# config (dict): Crate config
+# workspace_config (dict): Workspace config
+def config_inherit_workspace(config, workspace_config):
+ workspace_deps = workspace_config.get("dependencies")
+ if workspace_deps is not None:
+ dependencies = []
+ for key in ["dependencies", "dev-dependencies", "build-dependencies"]:
+ if key in config:
+ dependencies.append(config[key])
+
+ if "traget" in config:
+ for target in config["target"].values():
+ if "dependencies" in target:
+ dependencies.append(target["dependencies"])
+
+ for deps in dependencies:
+ inherit_deps(deps, workspace_deps)
+
+ workspace_package = workspace_config.get("package")
+ if workspace_package is not None:
+ inherit_package(config["package"], workspace_package)
+
+
+# inherit_package()
+#
+# Adds inherited [package] entries to config
+#
+# Args:
+# items (dict): Package items
+# workspace_items (dict): Workspace items
+def inherit_package(items, workspace_items):
+ for key, value in items.items():
+ if isinstance(value, dict) and "workspace" in value:
+ workspace_value = workspace_items.get(key)
+ if workspace_value is None:
+ raise SourceError("Can't inherit package information from workspace: Value missing from workspace")
Review Comment:
As someone not familiar with the intricacies of Cargo, this error message makes no sense to me sorry.
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +310,268 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {self.repo}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(self.repo)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"Cargo doens't support vendoring for identical crate version from different sources {self.name} {self.version}"
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml and "memers":
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
+ tomli_w.dump(crate["config"], f)
+
+ shutil.move(crate["path"], crate_target_dir)
+ else:
+ # No workspaces involed, just reploy complete dir as is
+ shutil.move(tmp_dir, crate_target_dir)
+
+ # Write .cargo-checksum.json required by cargo vendoring
+ with open(os.path.join(crate_target_dir, ".cargo-checksum.json"), "w") as f:
+ json.dump({"files": {}, "package": None}, f)
+
+ shutil.rmtree(tmp_dir, ignore_errors=True)
+
+ # is_cached()
+ #
+ # Get whether we have a local cached version of the git commit
+ #
+ # Returns:
+ # (bool): Whether we are cached or not
+ #
+ def is_cached(self):
+ with Repo(self._get_mirror_dir(), bare=True) as repo:
+ return self.commit.encode() in repo
+
+ # is_resolved()
+ #
+ # Get whether the current git repo is resolved
+ #
+ # Returns:
+ # (bool): Always true since we always have a commit
+ #
+ def is_resolved(self):
+ return True
+
+ ########################################################
+ # Private helpers #
+ ########################################################
+
+ # _get_mirror_dir()
+ #
+ # Gets the local mirror directory for this upstream git repository
+ #
+ def _get_mirror_dir(self):
+ if self.repo.endswith(".git"):
+ norm_url = self.repo[:-4]
+ else:
+ norm_url = self.repo
+
+ return os.path.join(
+ self.cargo.get_mirror_directory(),
+ utils.url_directory_name(norm_url) + ".git",
+ )
+
+ # _get_url()
+ #
+ # Gets the git URL to download this crate from
+ #
+ # Args:
+ # alias (str|None): The URL alias to apply, if any
+ #
+ # Returns:
+ # (str): The URL for this crate
+ #
+ def _get_url(self, alias=None):
+ return self.cargo.translate_url(self.cargo.repo, alias_override=alias)
+
+ # _mirror_repo()
+ #
+ # Returns the mirror repo, initialized if it doesn not exist yet
+ #
+ # Returns:
+ # (Repo): The mirror repo crate
+ #
+ def _mirror_repo(self):
+ try:
+ return Repo.init_bare(self._get_mirror_dir(), mkdir=True)
+ except FileExistsError:
+ return Repo(self._get_mirror_dir(), bare=True)
+
+
+# config_inherit_workspace()
+#
+# Adds inherited values to config
+#
+# Args:
+# config (dict): Crate config
+# workspace_config (dict): Workspace config
+def config_inherit_workspace(config, workspace_config):
+ workspace_deps = workspace_config.get("dependencies")
+ if workspace_deps is not None:
+ dependencies = []
+ for key in ["dependencies", "dev-dependencies", "build-dependencies"]:
+ if key in config:
+ dependencies.append(config[key])
+
+ if "traget" in config:
Review Comment:
typo
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +310,268 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {self.repo}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(self.repo)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"Cargo doens't support vendoring for identical crate version from different sources {self.name} {self.version}"
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml and "memers":
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
+ tomli_w.dump(crate["config"], f)
+
+ shutil.move(crate["path"], crate_target_dir)
+ else:
+ # No workspaces involed, just reploy complete dir as is
+ shutil.move(tmp_dir, crate_target_dir)
+
+ # Write .cargo-checksum.json required by cargo vendoring
+ with open(os.path.join(crate_target_dir, ".cargo-checksum.json"), "w") as f:
+ json.dump({"files": {}, "package": None}, f)
+
+ shutil.rmtree(tmp_dir, ignore_errors=True)
+
+ # is_cached()
+ #
+ # Get whether we have a local cached version of the git commit
+ #
+ # Returns:
+ # (bool): Whether we are cached or not
+ #
+ def is_cached(self):
+ with Repo(self._get_mirror_dir(), bare=True) as repo:
+ return self.commit.encode() in repo
+
+ # is_resolved()
+ #
+ # Get whether the current git repo is resolved
+ #
+ # Returns:
+ # (bool): Always true since we always have a commit
+ #
+ def is_resolved(self):
+ return True
+
+ ########################################################
+ # Private helpers #
+ ########################################################
+
+ # _get_mirror_dir()
+ #
+ # Gets the local mirror directory for this upstream git repository
+ #
+ def _get_mirror_dir(self):
+ if self.repo.endswith(".git"):
+ norm_url = self.repo[:-4]
+ else:
+ norm_url = self.repo
+
+ return os.path.join(
+ self.cargo.get_mirror_directory(),
+ utils.url_directory_name(norm_url) + ".git",
+ )
+
+ # _get_url()
+ #
+ # Gets the git URL to download this crate from
+ #
+ # Args:
+ # alias (str|None): The URL alias to apply, if any
+ #
+ # Returns:
+ # (str): The URL for this crate
+ #
+ def _get_url(self, alias=None):
+ return self.cargo.translate_url(self.cargo.repo, alias_override=alias)
+
+ # _mirror_repo()
+ #
+ # Returns the mirror repo, initialized if it doesn not exist yet
+ #
+ # Returns:
+ # (Repo): The mirror repo crate
+ #
+ def _mirror_repo(self):
+ try:
+ return Repo.init_bare(self._get_mirror_dir(), mkdir=True)
+ except FileExistsError:
+ return Repo(self._get_mirror_dir(), bare=True)
+
+
+# config_inherit_workspace()
+#
+# Adds inherited values to config
+#
+# Args:
+# config (dict): Crate config
+# workspace_config (dict): Workspace config
+def config_inherit_workspace(config, workspace_config):
+ workspace_deps = workspace_config.get("dependencies")
+ if workspace_deps is not None:
+ dependencies = []
+ for key in ["dependencies", "dev-dependencies", "build-dependencies"]:
+ if key in config:
+ dependencies.append(config[key])
+
+ if "traget" in config:
+ for target in config["target"].values():
+ if "dependencies" in target:
+ dependencies.append(target["dependencies"])
+
+ for deps in dependencies:
+ inherit_deps(deps, workspace_deps)
+
+ workspace_package = workspace_config.get("package")
+ if workspace_package is not None:
+ inherit_package(config["package"], workspace_package)
+
+
+# inherit_package()
+#
+# Adds inherited [package] entries to config
Review Comment:
Function docs for this function, and similar ones (`config_inherit_workspace`, `inherit_deps`, etc), aren't very helpful.
> Adds inherited [package] entries to config
Inherited from where? Adds to what config?
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
The only real use-case I can think of for an alias here is just to outright mirror the entire dependency tree of the project. Here's the simplest way to support this, IMO:
The user will set this configuration in their project:
```yaml
- kind: cargo
git_mirror: somealias:
```
In your code you'd do something like this (pseudocode):
```python3
def __init__(...):
...
self.mark_download_url(cargo.git_mirror)
...
```
```python3
def _get_url(...):
if self.cargo.git_mirror is not None:
mirror = self.cargo.translate_url(self.cargo.git_mirror, ...) # Resolves aliases
if mirror == "": # Reasoning: this lets generic projects like freedesktop-sdk provide an alias for builders to override, without actually specifying any mirror of its own to use by default
return self.repo
protocol, url = split_protocol(self.repo) # ex: "https://github.com/foobar" => "https", "github.com/foobar"
return urljoin(mirror, "/", protocol, "/", url)
else:
return self.repo
```
Then make sure to use _get_url when giving a URL to dulwich to handle
So for example, if `git_mirror` is set to `our-private-gitlab:cargo-mirrors/` and `our-private-gitlab:` is set to `https://gitlab.example.com/`, then: https://github.com/someorg/somecrate will be translated into https://gitlab.example.com/cargo-mirrors/https/github.com/someorg/somecrate, and that's what we'll ultimately fetch.
freedesktop-sdk will set `git_mirror` to `cargo_git_mirror:` by default, and `cargo_git_mirror` will be set to "" (blank). Then https://github.com/someorg/somecrate will be left untouched: `self.cargo.git_mirror` is not none, but once translated it'll resolve to an empty string so it'll be ignored.
Some company that builds freedesktop-sdk can override `cargo_git_mirror` to be `https://gitlab.example.com/cargo-mirrors/` in their global buildstream.conf file, because they are in an environment where they need mirroring. This way https://github.com/someorg/somecrate will again be translated into https://gitlab.example.com/cargo-mirrors/https/github.com/someorg/somecrate, because this time `self.cargo.git_mirror` (again, set to `cargo_git_mirror:` by freedesktop-sdk) will get translated into `https://gitlab.example.com/cargo-mirrors/`
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +310,268 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {self.repo}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(self.repo)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"Cargo doens't support vendoring for identical crate version from different sources {self.name} {self.version}"
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml and "memers":
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
+ tomli_w.dump(crate["config"], f)
+
+ shutil.move(crate["path"], crate_target_dir)
+ else:
+ # No workspaces involed, just reploy complete dir as is
+ shutil.move(tmp_dir, crate_target_dir)
+
+ # Write .cargo-checksum.json required by cargo vendoring
+ with open(os.path.join(crate_target_dir, ".cargo-checksum.json"), "w") as f:
+ json.dump({"files": {}, "package": None}, f)
+
+ shutil.rmtree(tmp_dir, ignore_errors=True)
+
+ # is_cached()
+ #
+ # Get whether we have a local cached version of the git commit
+ #
+ # Returns:
+ # (bool): Whether we are cached or not
+ #
+ def is_cached(self):
+ with Repo(self._get_mirror_dir(), bare=True) as repo:
+ return self.commit.encode() in repo
+
+ # is_resolved()
+ #
+ # Get whether the current git repo is resolved
+ #
+ # Returns:
+ # (bool): Always true since we always have a commit
+ #
+ def is_resolved(self):
+ return True
+
+ ########################################################
+ # Private helpers #
+ ########################################################
+
+ # _get_mirror_dir()
+ #
+ # Gets the local mirror directory for this upstream git repository
+ #
+ def _get_mirror_dir(self):
+ if self.repo.endswith(".git"):
+ norm_url = self.repo[:-4]
+ else:
+ norm_url = self.repo
+
+ return os.path.join(
+ self.cargo.get_mirror_directory(),
+ utils.url_directory_name(norm_url) + ".git",
+ )
+
+ # _get_url()
+ #
+ # Gets the git URL to download this crate from
+ #
+ # Args:
+ # alias (str|None): The URL alias to apply, if any
+ #
+ # Returns:
+ # (str): The URL for this crate
+ #
+ def _get_url(self, alias=None):
+ return self.cargo.translate_url(self.cargo.repo, alias_override=alias)
+
+ # _mirror_repo()
+ #
+ # Returns the mirror repo, initialized if it doesn not exist yet
+ #
+ # Returns:
+ # (Repo): The mirror repo crate
+ #
+ def _mirror_repo(self):
+ try:
+ return Repo.init_bare(self._get_mirror_dir(), mkdir=True)
+ except FileExistsError:
+ return Repo(self._get_mirror_dir(), bare=True)
+
+
+# config_inherit_workspace()
+#
+# Adds inherited values to config
+#
+# Args:
+# config (dict): Crate config
+# workspace_config (dict): Workspace config
+def config_inherit_workspace(config, workspace_config):
+ workspace_deps = workspace_config.get("dependencies")
+ if workspace_deps is not None:
+ dependencies = []
+ for key in ["dependencies", "dev-dependencies", "build-dependencies"]:
+ if key in config:
+ dependencies.append(config[key])
+
+ if "traget" in config:
+ for target in config["target"].values():
+ if "dependencies" in target:
+ dependencies.append(target["dependencies"])
+
+ for deps in dependencies:
+ inherit_deps(deps, workspace_deps)
+
+ workspace_package = workspace_config.get("package")
+ if workspace_package is not None:
+ inherit_package(config["package"], workspace_package)
+
+
+# inherit_package()
+#
+# Adds inherited [package] entries to config
Review Comment:
Maybe there's some documentation of the Cargo.toml file that you can reference in these function comments, so it's clear what exactly you're doing and why? Right now the code is just twiddling some entries in some dictionaries, and unless I go looking for docs I have no way to know whether it's correct (which by the way I'm not doing for this review; I'll assume that you're manipulating the toml files correctly :smile:)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1528959924
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -367,17 +644,31 @@ def track(self, *, previous_sources_dir):
for package in lock["package"]:
if "source" not in package:
continue
- new_ref += [{"name": package["name"], "version": str(package["version"]), "sha": package.get("checksum")}]
+
+ ref = {}
+
+ if package["source"][:4] == "git+":
Review Comment:
Added a check
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
Should I implement this kind of mirror code?
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +310,268 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {self.repo}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(self.repo)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"Cargo doens't support vendoring for identical crate version from different sources {self.name} {self.version}"
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml and "memers":
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
+ tomli_w.dump(crate["config"], f)
+
+ shutil.move(crate["path"], crate_target_dir)
+ else:
+ # No workspaces involed, just reploy complete dir as is
+ shutil.move(tmp_dir, crate_target_dir)
+
+ # Write .cargo-checksum.json required by cargo vendoring
+ with open(os.path.join(crate_target_dir, ".cargo-checksum.json"), "w") as f:
+ json.dump({"files": {}, "package": None}, f)
+
+ shutil.rmtree(tmp_dir, ignore_errors=True)
+
+ # is_cached()
+ #
+ # Get whether we have a local cached version of the git commit
+ #
+ # Returns:
+ # (bool): Whether we are cached or not
+ #
+ def is_cached(self):
+ with Repo(self._get_mirror_dir(), bare=True) as repo:
+ return self.commit.encode() in repo
+
+ # is_resolved()
+ #
+ # Get whether the current git repo is resolved
+ #
+ # Returns:
+ # (bool): Always true since we always have a commit
+ #
+ def is_resolved(self):
+ return True
+
+ ########################################################
+ # Private helpers #
+ ########################################################
+
+ # _get_mirror_dir()
+ #
+ # Gets the local mirror directory for this upstream git repository
+ #
+ def _get_mirror_dir(self):
+ if self.repo.endswith(".git"):
+ norm_url = self.repo[:-4]
+ else:
+ norm_url = self.repo
+
+ return os.path.join(
+ self.cargo.get_mirror_directory(),
+ utils.url_directory_name(norm_url) + ".git",
+ )
+
+ # _get_url()
+ #
+ # Gets the git URL to download this crate from
+ #
+ # Args:
+ # alias (str|None): The URL alias to apply, if any
+ #
+ # Returns:
+ # (str): The URL for this crate
+ #
+ def _get_url(self, alias=None):
+ return self.cargo.translate_url(self.cargo.repo, alias_override=alias)
+
+ # _mirror_repo()
+ #
+ # Returns the mirror repo, initialized if it doesn not exist yet
+ #
+ # Returns:
+ # (Repo): The mirror repo crate
+ #
+ def _mirror_repo(self):
+ try:
+ return Repo.init_bare(self._get_mirror_dir(), mkdir=True)
+ except FileExistsError:
+ return Repo(self._get_mirror_dir(), bare=True)
+
+
+# config_inherit_workspace()
+#
+# Adds inherited values to config
+#
+# Args:
+# config (dict): Crate config
+# workspace_config (dict): Workspace config
+def config_inherit_workspace(config, workspace_config):
+ workspace_deps = workspace_config.get("dependencies")
+ if workspace_deps is not None:
+ dependencies = []
+ for key in ["dependencies", "dev-dependencies", "build-dependencies"]:
+ if key in config:
+ dependencies.append(config[key])
+
+ if "traget" in config:
+ for target in config["target"].values():
+ if "dependencies" in target:
+ dependencies.append(target["dependencies"])
+
+ for deps in dependencies:
+ inherit_deps(deps, workspace_deps)
+
+ workspace_package = workspace_config.get("package")
+ if workspace_package is not None:
+ inherit_package(config["package"], workspace_package)
+
+
+# inherit_package()
+#
+# Adds inherited [package] entries to config
+#
+# Args:
+# items (dict): Package items
+# workspace_items (dict): Workspace items
+def inherit_package(items, workspace_items):
+ for key, value in items.items():
+ if isinstance(value, dict) and "workspace" in value:
+ workspace_value = workspace_items.get(key)
+ if workspace_value is None:
+ raise SourceError("Can't inherit package information from workspace: Value missing from workspace")
Review Comment:
This message is more for debugging the plugin since the crate wouldn't have originally been able to build with this. But I tried to clarify it.
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +310,268 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {self.repo}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(self.repo)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"Cargo doens't support vendoring for identical crate version from different sources {self.name} {self.version}"
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml and "memers":
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
+ tomli_w.dump(crate["config"], f)
+
+ shutil.move(crate["path"], crate_target_dir)
+ else:
+ # No workspaces involed, just reploy complete dir as is
+ shutil.move(tmp_dir, crate_target_dir)
+
+ # Write .cargo-checksum.json required by cargo vendoring
+ with open(os.path.join(crate_target_dir, ".cargo-checksum.json"), "w") as f:
+ json.dump({"files": {}, "package": None}, f)
+
+ shutil.rmtree(tmp_dir, ignore_errors=True)
+
+ # is_cached()
+ #
+ # Get whether we have a local cached version of the git commit
+ #
+ # Returns:
+ # (bool): Whether we are cached or not
+ #
+ def is_cached(self):
+ with Repo(self._get_mirror_dir(), bare=True) as repo:
+ return self.commit.encode() in repo
+
+ # is_resolved()
+ #
+ # Get whether the current git repo is resolved
+ #
+ # Returns:
+ # (bool): Always true since we always have a commit
+ #
+ def is_resolved(self):
+ return True
+
+ ########################################################
+ # Private helpers #
+ ########################################################
+
+ # _get_mirror_dir()
+ #
+ # Gets the local mirror directory for this upstream git repository
+ #
+ def _get_mirror_dir(self):
+ if self.repo.endswith(".git"):
+ norm_url = self.repo[:-4]
+ else:
+ norm_url = self.repo
+
+ return os.path.join(
+ self.cargo.get_mirror_directory(),
+ utils.url_directory_name(norm_url) + ".git",
+ )
+
+ # _get_url()
+ #
+ # Gets the git URL to download this crate from
+ #
+ # Args:
+ # alias (str|None): The URL alias to apply, if any
+ #
+ # Returns:
+ # (str): The URL for this crate
+ #
+ def _get_url(self, alias=None):
+ return self.cargo.translate_url(self.cargo.repo, alias_override=alias)
+
+ # _mirror_repo()
+ #
+ # Returns the mirror repo, initialized if it doesn not exist yet
+ #
+ # Returns:
+ # (Repo): The mirror repo crate
+ #
+ def _mirror_repo(self):
+ try:
+ return Repo.init_bare(self._get_mirror_dir(), mkdir=True)
+ except FileExistsError:
+ return Repo(self._get_mirror_dir(), bare=True)
+
+
+# config_inherit_workspace()
+#
+# Adds inherited values to config
+#
+# Args:
+# config (dict): Crate config
+# workspace_config (dict): Workspace config
+def config_inherit_workspace(config, workspace_config):
+ workspace_deps = workspace_config.get("dependencies")
+ if workspace_deps is not None:
+ dependencies = []
+ for key in ["dependencies", "dev-dependencies", "build-dependencies"]:
+ if key in config:
+ dependencies.append(config[key])
+
+ if "traget" in config:
+ for target in config["target"].values():
+ if "dependencies" in target:
+ dependencies.append(target["dependencies"])
+
+ for deps in dependencies:
+ inherit_deps(deps, workspace_deps)
+
+ workspace_package = workspace_config.get("package")
+ if workspace_package is not None:
+ inherit_package(config["package"], workspace_package)
+
+
+# inherit_package()
+#
+# Adds inherited [package] entries to config
Review Comment:
I added a bunch of text plus links to the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1532483849
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
> ```yaml
> sources:
> - kind: cargo
> git_mirrors:
> - https://github.com/x/y: https://mirror.example.org/x/y
> - https://github.com/a/b: my-alias:a/b
> ```
What I have done there is not what mirrors do in practice. See this mirror entry:
```yaml
github:
- https://gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/github_com/
```
Maybe what's called `git_mirrors` above could be a string replace list such that
> ```yaml
> sources:
> - kind: cargo
> git_mirrors:
> - https://github.com/: github:
> ```
works too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#issuecomment-2052610970
Just a thought but it would be nice if the tomlkit dependency was optional and became fatal only if there actually are git crates. The existing dependency tomllib is in Python standard library while this one implies user installing packages which is not automatic when this is consumed as junction like freedesktop-sdk and gnome-build-meta do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1585532167
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +324,284 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ self.repo_mirrored = cargo.mirrored_git_url(repo)
+
+ self.mark_download_url(self.repo_mirrored)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+ repo_url = self._get_url(alias=alias_override)
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {repo_url}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(repo_url)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"This project requests crate {self.name} {self.version} from multiple sources, "
+ "which is incompatible with vendoring since cargo does not support it."
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml:
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
Review Comment:
This is not true based on brief glance of source code. It seems to support both. Maybe it's true on some older version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1524022895
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
I'm a little bit confused what this is for. I think the other fetcher just passes "cargo:cargo"? So this needs someone who understands this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#issuecomment-1998551234
Noticed that cargo workspace support was missing. Tested with gtk-rs and glycin for the workspaces feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1530213744
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
> Aliasing indeed needs to be set on per-git repo level.
Okay, so we just go through all git sources in `configure` and call `mark_download_url` for them and then just call `translate_url` on the complete URL. That's then the same as the `git_repo` plugin does, just that we have to potentially deal with multiple repos per `Source` object.
Sounds easy enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1532485520
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
That would allow us to mirror a specific repo or to just have a complete GitHub mirror.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1537585311
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
Okay, implemented this and tried to also document it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "sophie-h (via GitHub)" <gi...@apache.org>.
sophie-h commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1530673117
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +308,166 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {}
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ # TODO: Is this right?
+ self.mark_download_url(self.repo)
Review Comment:
Okay, that's not working since buildstream doesn't seem to have a feature to translate one URL to another. The only thing that's supported is adding an explicit prefix to a URL, which we can't do because the URLs are auto-generated by the plugin.
If that's correct, the only solution I can see is to add a list of URL mappings to the source config like
```yaml
sources:
- kind: cargo
git_mirrors:
- https://github.com/x/y: https://mirror.example.org/x/y
- https://github.com/a/b: my-alias:a/b
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cargo: Add support for git dependencies [buildstream-plugins]
Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on code in PR #62:
URL: https://github.com/apache/buildstream-plugins/pull/62#discussion_r1585532167
##########
src/buildstream_plugins/sources/cargo.py:
##########
@@ -294,6 +324,284 @@ def _get_mirror_file(self, sha=None):
return os.path.join(self._get_mirror_dir(), sha or self.sha)
+# Locks on repositories for write access
+REPO_LOCKS = {} # type: dict[str, threading.Lock]
+
+
+# CrateGit()
+#
+# Use a SourceFetcher class to be the per crate helper.
+#
+# This one is for crates fetched from git repositories.
+#
+# Args:
+# cargo (Cargo): The main Source implementation
+# name (str): The name of the crate to depend on
+# version (str): The version of the crate to depend on
+# repo (str): Repository URL
+# commit (str): Sha of the git commit
+class CrateGit(SourceFetcher):
+ def __init__(self, cargo, name, version, repo, commit):
+ super().__init__()
+
+ self.cargo = cargo
+ self.name = name
+ self.version = str(version)
+ self.repo = repo
+ self.commit = commit
+ self.repo_mirrored = cargo.mirrored_git_url(repo)
+
+ self.mark_download_url(self.repo_mirrored)
+
+ ########################################################
+ # SourceFetcher API method implementations #
+ ########################################################
+
+ def fetch(self, alias_override=None, **kwargs):
+ lock = REPO_LOCKS.setdefault(self._get_mirror_dir(), threading.Lock())
+ repo_url = self._get_url(alias=alias_override)
+
+ with lock, self._mirror_repo() as repo, self.cargo.timed_activity(f"Fetching from {repo_url}"):
+ # TODO: Auth not supported
+ client, path = dulwich.client.get_transport_and_path(repo_url)
+ remote_refs = client.fetch(
+ path,
+ repo,
+ determine_wants=lambda refs, depth=None: [self.commit.encode()],
+ depth=1,
+ )
+
+ ########################################################
+ # Helper APIs for the Cargo Source to use #
+ ########################################################
+
+ def ref_node(self):
+ return {"kind": "git", "name": self.name, "version": self.version, "repo": self.repo, "commit": self.commit}
+
+ # stage()
+ #
+ # A delegate method to do the work for a single git repo
+ # in Source.stage().
+ #
+ # Args:
+ # (directory): The vendor subdirectory to stage to
+ #
+ def stage(self, directory):
+ self.cargo.status(f"Checking out {self.commit}")
+
+ crate_target_dir = os.path.join(directory, f"{self.name}-{self.version}")
+ tmp_dir = os.path.join(directory, f"{self.name}-{self.version}-tmp")
+
+ try:
+ os.mkdir(tmp_dir)
+ except FileExistsError:
+ raise SourceError(
+ f"This project requests crate {self.name} {self.version} from multiple sources, "
+ "which is incompatible with vendoring since cargo does not support it."
+ )
+
+ with Repo(self._get_mirror_dir(), bare=True) as mirror:
+ with Repo.init(tmp_dir) as dest:
+ dest.object_store.add_object(mirror[self.commit.encode()])
+ dest.refs[b"HEAD"] = self.commit.encode()
+ dest.update_shallow([self.commit.encode()], [])
+
+ with Repo(tmp_dir, object_store=mirror.object_store) as dest:
+ dest.reset_index()
+
+ # Workspace handling
+ #
+ # When new workspace features are added it is worth checking if
+ # <https://github.com/flatpak/flatpak-builder-tools/blob/HEAD/cargo/flatpak-cargo-generator.py>
+ # has implemented them already. This implementation is inspired by the mentioned source.
+
+ with open(os.path.join(tmp_dir, "Cargo.toml"), "rb") as f:
+ root_toml = tomllib.load(f)
+
+ crates = {}
+
+ if "workspace" in root_toml:
+ # Find wanted crate inside workspace
+ for member in root_toml["workspace"].get("members", []):
+ for crate_toml_path in glob.glob(os.path.join(tmp_dir, member, "Cargo.toml")):
+ crate_path = os.path.normpath(os.path.dirname(crate_toml_path))
+
+ with open(crate_toml_path, "rb") as f:
+ crate_toml = tomllib.load(f)
+ crates[crate_toml["package"]["name"]] = {
+ "config": crate_toml,
+ "path": crate_path,
+ }
+
+ crate = crates[self.name]
+ # Apply information inherited from workspace Cargo.toml
+ config_inherit_workspace(crate["config"], root_toml["workspace"])
+
+ with open(os.path.join(crates[self.name]["path"], "Cargo.toml"), "bw") as f:
Review Comment:
This is not true based on brief glance of source code. It seems to support both. Maybe it's not true on some older version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org