You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/01/15 03:24:21 UTC
[GitHub] [tvm] areusch opened a new pull request #7289: Generate requirements.txt from Python spec
areusch opened a new pull request #7289:
URL: https://github.com/apache/tvm/pull/7289
This PR implements the [Consolidating Python Dependencies RFC](https://discuss.tvm.apache.org/t/rfc-consolidating-tvm-python-dependencies/8329).
As part of this RFC, we decided to try a solution halfway between burdening TVM with a full-blown dependency tool such as poetry and the approach we currently take (scattered Python deps). The initial patchset here contains the tool plus some examples of the dependencies. The remaining things needed to merge this PR:
- [ ] Write tests for the tool
- [ ] Finish consolidating various dependencies from the ci docker containers
I'm posting this now before I do any of the test work to get feedback on the approach now that there's something concrete to look at. I do think this tool is a bit more complex than it could be, and am concerned about maintaining it. In particular the logic needed to lint the dependencies list is complex. I don't see a solution that avoids joining a set of dependencies by feature against a constraints list as very workable given the current extras structure.
Another topic for discussion is the use of semver. Python does contain a compatibility operator we could use. We should probably debate between the two (let's use the forum for this), as it would remove some complexity here.
Comments would be quite welcomed!
@tqchen @tkonolige @mshawcroft @leandron @junrushao1994 @tmoreau89 @jwfromm @jroesch @mdw-octoml @zhiics
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] mdw-octoml commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
mdw-octoml commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r568062188
##########
File path: python/setup.py
##########
@@ -171,38 +171,25 @@ def get_package_data_files():
return ["relay/std/prelude.rly", "relay/std/core.rly"]
+# Temporarily add this directory to the path so we can import the requirements generator
+# tool.
+sys.path.insert(0, os.path.dirname(__file__))
+import gen_requirements
+sys.path.pop(0)
+
+requirements = gen_requirements.join_requirements()
Review comment:
Well, just a suggestion that when this is being invoked, and the requirements being generated automatically, it might be useful to have them dumped to stdout during the invocation (perhaps optionally) so someone debugging issues with Python deps can readily see what they were. Not sure if they are exposed in an obvious way elsewhere.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r568071995
##########
File path: tests/python/unittest/test_gen_requirements.py
##########
@@ -0,0 +1,220 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Tests for gen_requirements, found in python/."""
+
+import collections
+import contextlib
+import os
+import sys
+
+import tvm
+
+import pytest
+
+# Insert the parent dir to python/tvm into the import path, so that gen_requirements may be
+# imported.
+sys.path.insert(0, os.path.dirname(tvm.__file__))
+try:
+ import gen_requirements
+finally:
+ sys.path.pop(0)
+
+
+@contextlib.contextmanager
+def patch(obj, **kw):
+ old = {}
+ for prop_name, new in kw.items():
+ old[prop_name] = getattr(obj, prop_name)
+ setattr(obj, prop_name, new)
+ yield
+ for prop_name, value in old.items():
+ setattr(obj, prop_name, value)
+
+
+PROBLEM_REQUIREMENTS = [
+ ("extras-pre-core", ("", ["foo", 123])), # entry before core
+ (456, ("", ["foo", "bar"])), # invalid extras name, deps should not be processed
+ ("core", ("", ["foo"])), # ordinary core entry.
+ ("wrong-description-type", (None, ["foo"])), # wrong description type
+ ("bad-value", None), # value field is not a 2-tuple
+ ("bad-value-2", ("", ["foo"], 34)), # value field is not a 2-tuple
+ ("invalid", ("", ["qux"])), # duplicate invalid entry, all items valid.
+ ("extras-foo", ("", ["bar", "baz"])), # ordinary extras entry.
+ ("invalid", ("", ["baz", None, 123])), # valid extra name, invalid deps.
+ ("unsorted", ("", ["qux", "bar", "foo"])), # deps out of order
+ ("versioned_dep", ("", ["baz==1.2", "foo==^2.0", "buz<3", "bar>4"])),
+ ("duplicate_dep", ("", ["buz", "buz", "foo"])), # duplicate listed dependency
+ ("dev", ("", ["baz", "qux"])), # ordinary dev entry.
+ ("extras-post-dev", ("", ["bar", "buzz"])), # entry after dev
+]
+
+
+def test_validate_requirements():
+ with patch(gen_requirements, REQUIREMENTS_BY_PIECE=None):
+ assert gen_requirements.validate_requirements_by_piece() == [
+ "must be list or tuple, see None"
+ ]
+
+ with patch(gen_requirements, REQUIREMENTS_BY_PIECE=PROBLEM_REQUIREMENTS):
+ problems = gen_requirements.validate_requirements_by_piece()
+ assert problems == [
+ 'piece extras-pre-core: must list after "core" (core must be first)',
+ "piece extras-pre-core: deps should be a list of strings, got ['foo', 123]",
+ "piece 456: must be str",
+ "piece wrong-description-type: description should be a string, got None",
+ (
+ 'piece bad-value: should be formatted like ("bad-value", ("<requirements.txt '
+ 'comment>", ["dep1", "dep2", ...])). got: None'
+ ),
+ (
+ 'piece bad-value-2: should be formatted like ("bad-value-2", '
+ '("<requirements.txt comment>", ["dep1", "dep2", ...])). got: (\'\', '
+ "['foo'], 34)"
+ ),
+ "piece invalid: listed twice",
+ "piece invalid: deps should be a list of strings, got ['baz', None, 123]",
+ "piece unsorted: deps must be sorted. Correct order:\n ['bar', 'foo', 'qux']",
+ "piece versioned_dep: deps must be sorted. Correct order:\n ['bar>4', 'baz==1.2', 'buz<3', 'foo==^2.0']",
+ "piece versioned_dep: dependency baz==1.2 should not specify a version. Add it to CONSTRAINTS instead.",
+ "piece versioned_dep: dependency foo==^2.0 should not specify a version. Add it to CONSTRAINTS instead.",
+ "piece versioned_dep: dependency buz<3 should not specify a version. Add it to CONSTRAINTS instead.",
+ "piece versioned_dep: dependency bar>4 should not specify a version. Add it to CONSTRAINTS instead.",
+ "piece duplicate_dep: dependency buz listed twice",
+ 'piece extras-post-dev: must list before "dev" (dev must be last)',
+ 'pieces other than "core" and "dev" must appear in alphabetical order: '
+ "['bad-value', 'bad-value-2', 'duplicate_dep', 'extras-foo', 'extras-post-dev', "
+ "'extras-pre-core', 'invalid', 'invalid', 'unsorted', 'versioned_dep', "
+ "'wrong-description-type']",
+ ]
+
+
+TEST_REQUIREMENTS_BY_PIECE = (
+ ("core", ("core tvm requirements", ("bar", "foo", "non-constrained"))),
+ ("extra-one", ("requirements for one feature", ("baz", "qux"))),
+ ("extra-two", ("requirements for two feature", ("buz", "qux", "semver-minor", "semver-patch"))),
+ ("dev", ("requirements for dev", ("buz", "oof", "rab"))),
+)
+
+
+def test_validate_constraints():
+ with patch(
+ gen_requirements,
+ REQUIREMENTS_BY_PIECE=TEST_REQUIREMENTS_BY_PIECE,
+ CONSTRAINTS=(
+ ("unlisted", "~=3"),
Review comment:
same issue though
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r558477221
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
+
+ # Development requirements
+ ("dev", ["matplotlib", "pillow"]),
+]
+
+# Maps a named Python package (which should appear in REQUIREMENTS_BY_PIECE above) to a
+# semver or pip version constraint. Semver constraints are translated into requirements.txt
+# constraints.
+CONSTRAINTS = [
+ ("onnx", ">=1.7.0"),
+ ("onnxruntime", ">=1.0.0"),
+ ("pillow", "<7"),
+ ("synr", ">=0.2.1"),
+ ("tensorflow", ">=2.1.0"),
+ ("tflite", ">=2.1.0"),
+ ("torch", "^1.7.0"),
+ ("torchvision", ">=0.5.0"),
+]
+
+################################################################################
+# End of configuration options.
+################################################################################
+
+
+
+
+# Required keys in REQUIREMENTS_BY_PIECE.
+REQUIRED_PIECES = ["core", "dev"]
+
+# Regex to validates piece names.
+PIECE_REGEX = re.compile(r"^[a-z0-9][a-z0-9-]*", re.IGNORECASE)
+
+# Regex to match a constraint specification. Multiple constraints are not supported.
+CONSTRAINT_REGEX = re.compile(r"(?:\^|\<|(?:<=)|(?:==)|(?:>=)|\>)[^<>=\^,]+")
+
+# Regex for parsing semantic versions. See
+# https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string
+SEMVER_REGEX = re.compile(r"^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$")
+
+
+def validate_requirements_by_piece():
+ problems = []
+
+ unseen_required_pieces = set(REQUIRED_PIECES)
+ seen_pieces = set()
+
+ # Ensure that core is listed first and dev is listed last.
+ saw_core = False
+ saw_dev = False
+
+ if not isinstance(REQUIREMENTS_BY_PIECE, (list, tuple)):
+ problems.append(f"must be list or tuple, see {REQUIREMENTS_BY_PIECE!r}")
+
+ for piece, deps in REQUIREMENTS_BY_PIECE:
+ if not isinstance(piece, str):
+ problems.append(f"key {piece!r}: must be str, see {piece!r}")
+ continue
+
+ if piece in unseen_required_pieces:
+ unseen_required_pieces.remove(piece)
+
+ piece_lower = piece.lower()
+ if piece_lower in seen_pieces:
+ problems.append(f"piece {piece}: listed twice")
+
+ seen_pieces.add(piece_lower)
+
+ if not saw_core and piece != "core":
+ problems.append(f'must list "core" before {piece}')
+ elif piece == "core":
+ saw_core = True
+
+ if saw_dev:
+ problems.append(f'must list "dev" last')
+ elif piece == "dev":
+ saw_dev = True
+
+ if not isinstance(deps, (list, tuple)):
+ problems.append(f'value for key "{piece}" should be list or tuple, see {deps!r}')
+ continue
+
+ if list(sorted(deps)) != list(deps):
+ problems.append(f"deps for key {piece} must be sorted. Correct order:\n {list(sorted(deps))!r}")
+
+ piece_deps = set()
+ for d in deps:
+ if CONSTRAINT_REGEX.search(d):
+ problems.append(f"for piece {piece}: dependency {d} should not specify a version. "
+ "Add it to CONSTRAINTS instead.")
+
+ if d.lower() in piece_deps:
+ problems.append(f'for piece {piece}: dependency {d} listed twice')
+
+ piece_deps.add(d.lower())
+
+ extras_pieces = [k for (k, _) in REQUIREMENTS_BY_PIECE if k not in ("dev", "core")]
+ sorted_extras_pieces = list(sorted(extras_pieces))
+ if sorted_extras_pieces != list(extras_pieces):
+ problems.append('pieces other than "core" and "dev" must appear in alphabetical order'
+ f"{sorted_extras_pieces}")
+
+ return problems
+
+
+def parse_semver(package, constraint, problems):
+ """Parse a semantic versioning constraint of the form "^X.[.Y[.Z[...]]]]"
+
+ Parameters
+ ----------
+ package : str
+ Name of the package specifying this constraint, for reporting problems.
+ constraint : str
+ The semver constraint. Must start with "^"
+ problems : List[str]
+ A list of strings describing problems that have occurred validating the configuration.
+ Problems encountered while validating constraint are appended to this list.
+
+ Returns
+ -------
+ List[str], int :
+ A 2-tuple. The first element is a list containing an entry for each component in the
+ semver string (components separated by "."). The second element is the index of the
+ component in the list which must not change to meet the semver constraint.
+
+ See "Caret requirements" at https://python-poetry.org/docs/versions/.
+ """
+ m = SEMVER_REGEX.match(constraint[1:])
+ if not m:
+ problems.append(f"{package}: invalid semver constraint {constraint}")
+ return [], 0
+
+ min_ver_parts = list(m.groups())
+
+ # Remove the end
+ while min_ver_parts and min_ver_parts[-1] is None or not min_ver_parts[-1].strip():
+ min_ver_parts.pop()
+
+ for i, p in enumerate(min_ver_parts):
+ try:
+ x = int(p.strip())
+ except ValueError as p:
+ if i == len(parts) - 1 and i > 0:
+ # If a pre-release tag is specified (i.e. 1.2.0pre3), allow.
+ return min_ver_parts, i - 1
+
+ problems.append(
+ f"{package}: semver constraint with non-numeric leading component: {constraint}")
+ return [], 0
+
+ if x:
+ return min_ver_parts, i
+
+ else:
+ # All 0's
+ return min_ver_parts, 0
+
+
+def validate_constraints():
+ problems = []
+
+ if not isinstance(CONSTRAINTS, (list, tuple)):
+ problems.append(f"must be list or tuple, see: {CONSTRAINTS!r}")
+
+ seen_packages = set()
+ all_deps = set()
+ for _, deps in REQUIREMENTS_BY_PIECE:
+ for d in deps:
+ all_deps.add(d.lower())
+
+ for package, constraint in CONSTRAINTS:
+ if package in seen_packages:
+ problems.append(f"{package}: specified twice")
+
+ if package.lower() not in all_deps:
+ problems.append(f"{package}: not specified in REQUIREMENTS_BY_PIECE")
+
+ if not CONSTRAINT_REGEX.match(constraint):
+ problems.append(f'{package}: constraint "{constraint}" does not look like a valid constraint')
+
+ if constraint.startswith("^"):
+ parse_semver(package, constraint, problems)
+
+ all_constrained_packages = [p for (p, _) in CONSTRAINTS]
+ sorted_constrained_packages = list(sorted(all_constrained_packages))
+ if sorted_constrained_packages != all_constrained_packages:
+ problems.append("CONSTRAINTS entries should be in this sorted order: "
+ f'{sorted_constrained_packages}')
+
+ return problems
+
+
+class ValidationError(Exception):
+ """Raised when a validation error occurs."""
+
+ @staticmethod
+ def format_problems(config, problems):
+ formatted = []
+ for p in problems:
+ assert isinstance(p, str), f"problems element not a str: {p}"
+ formatted.append('\n'.join(
+ textwrap.wrap(f"{config}: {p}", width=80, initial_indent=" * ", subsequent_indent=" ")))
+
+ return '\n'.join(formatted)
+
+ def __init__(self, config, problems):
+ super(ValidationError, self).__init__(self.format_problems(config, problems))
+ self.problems = problems
+
+
+def validate_or_raise():
+ problems = validate_requirements_by_piece()
+ if problems:
+ raise ValidationError("REQUIREMENTS_BY_PIECE", problems)
+
+ problems = validate_constraints()
+ if problems:
+ raise ValidationError("CONSTRAINTS", problems)
+
+
+def semver_to_requirements(piece, dep, constraint, joined_deps):
+ problems = []
+ min_ver_parts, fixed_index = parse_semver(dep, constraint, problems)
+ assert not problems
+
+ # i is now the index of the version component which must not change.
+ try:
+ fixed_part = int(min_ver_parts[fixed_index])
+ except ValueError:
+ problems.append(
+ f"piece {piece}: dependency {dep} has semver constraint with non-integer leading parts: {ver}")
+ return
+
+ max_ver_parts = (min_ver_parts[:fixed_index] +
+ [str(int(min_ver_parts[fixed_index]) + 1)] +
+ ["0" for _ in min_ver_parts[fixed_index + 1:]])
+ joined_deps.append(f'{dep}>={".".join(min_ver_parts)},<{".".join(max_ver_parts)}')
+
+
+def join_requirements():
+ """Validate, then join REQUIRMENTS_BY_PIECE against CONSTRAINTS and return the result.
+
+ Returns
+ -------
+ An OrderedDict containing REQUIREMENTS_BY_PIECE, except any dependency mentioned in CONSTRAINTS
+ is replaced by a setuptools-compatible constraint.
+ """
+ validate_or_raise()
+
+ constraints_map = collections.OrderedDict(
+ [(p.lower(), c) for (p, c) in CONSTRAINTS])
+
+ to_return = collections.OrderedDict()
+ all_deps = set()
+ for piece, deps in REQUIREMENTS_BY_PIECE:
+ joined_deps = []
+ for d in deps:
+ constraint = constraints_map.get(d.lower())
+ if constraint is None:
+ joined_deps.append(d)
+ continue
+
+ if constraint[0] == "^":
+ # Translate semver to requirements:
+ semver_to_requirements(piece, d, constraint, joined_deps)
+ else:
+ joined_deps.append(f"{d}{constraint}")
+
+ if piece != "dev":
+ all_deps.update(joined_deps)
+
+ to_return[piece] = joined_deps
+
+ to_return["all"] = all_deps
+
+ return to_return
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--lint", action="store_true",
+ help="Just lint dependencies, don't generate anything")
+ return parser.parse_args()
+
+
+def main():
+ args = parse_args()
+ try:
+ joined_deps = join_requirements()
+ except ValidationError as e:
+ print(f"ERROR: invalid requirements configuration in {__file__}:", file=sys.stderr)
+ print(str(e), file=sys.stderr)
+ sys.exit(2)
+
+ if args.lint:
+ sys.exit(0)
+
+ output_dir = os.path.join(os.path.dirname(__file__), "requirements")
+ if not os.path.exists(output_dir):
+ os.makedirs(output_dir)
+ elif not os.path.isdir(output_dir):
+ print(f"ERROR: output directory {output_dir} exists but is not a dir. Delete it",
+ file=sys.stderr)
+ sys.exit(2)
+
+ for piece, deps in joined_deps.items():
+ with open(os.path.join(output_dir, f"{piece}.txt"), "w") as f:
+ for d in deps:
+ f.write(f"{d}{os.linesep}")
+
+
+if __name__ == "__main__":
+ main()
Review comment:
yes this is a great point. I sort of ran out of time before I posted the initial cut, so i figured i'd post and let people comment. I think the right thing to do is to move `join_requirements` + the requirements.txt writing into a top-level function and call _that_ function from both `gen_requirements.main` and `setup.py`. however, that does mean that if tools like `poetry` or others call `setup.py egg_info`, `requirements.txt` will be rewritten.
a developer might not expect that, though if it makes any appreciable change, they must be either syncing to main or modifying `gen_requirements.py`. i _think_ that's okay...what does everyone else think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r564706472
##########
File path: python/requirements/all.txt
##########
@@ -0,0 +1,16 @@
+numpy
Review comment:
I added a way to specify a comment to each piece and that's now placed at the top of each requirements.txt file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#issuecomment-761090330
thanks for the reviews so far everyone! one quick note:
the actual list of dependencies (and their version constraints) are still WIP. so definitely feel free to comment on things that are missing, but I know the list is wrong and I plan to fix the list and add tests if everyone's good w/ the approach.
i'll leave this open to collect reviews over the weekend and on Monday (holiday for me) and then i'll work to merge this mid-next-week if we are good w/ it.
@comaniac @tkonolige @leandron
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] leandron commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r558247110
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
Review comment:
Could you please also set `xgboost` as as dependency for tvmc? It is needed by `tvmc tune`, just noticed now.
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
+
+ # Development requirements
+ ("dev", ["matplotlib", "pillow"]),
+]
+
+# Maps a named Python package (which should appear in REQUIREMENTS_BY_PIECE above) to a
+# semver or pip version constraint. Semver constraints are translated into requirements.txt
+# constraints.
+CONSTRAINTS = [
+ ("onnx", ">=1.7.0"),
+ ("onnxruntime", ">=1.0.0"),
+ ("pillow", "<7"),
+ ("synr", ">=0.2.1"),
+ ("tensorflow", ">=2.1.0"),
+ ("tflite", ">=2.1.0"),
+ ("torch", "^1.7.0"),
+ ("torchvision", ">=0.5.0"),
+]
+
+################################################################################
+# End of configuration options.
+################################################################################
+
+
+
+
+# Required keys in REQUIREMENTS_BY_PIECE.
+REQUIRED_PIECES = ["core", "dev"]
+
+# Regex to validates piece names.
+PIECE_REGEX = re.compile(r"^[a-z0-9][a-z0-9-]*", re.IGNORECASE)
+
+# Regex to match a constraint specification. Multiple constraints are not supported.
+CONSTRAINT_REGEX = re.compile(r"(?:\^|\<|(?:<=)|(?:==)|(?:>=)|\>)[^<>=\^,]+")
+
+# Regex for parsing semantic versions. See
+# https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string
+SEMVER_REGEX = re.compile(r"^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$")
+
+
+def validate_requirements_by_piece():
+ problems = []
+
+ unseen_required_pieces = set(REQUIRED_PIECES)
+ seen_pieces = set()
+
+ # Ensure that core is listed first and dev is listed last.
+ saw_core = False
+ saw_dev = False
+
+ if not isinstance(REQUIREMENTS_BY_PIECE, (list, tuple)):
+ problems.append(f"must be list or tuple, see {REQUIREMENTS_BY_PIECE!r}")
+
+ for piece, deps in REQUIREMENTS_BY_PIECE:
+ if not isinstance(piece, str):
+ problems.append(f"key {piece!r}: must be str, see {piece!r}")
+ continue
+
+ if piece in unseen_required_pieces:
+ unseen_required_pieces.remove(piece)
+
+ piece_lower = piece.lower()
+ if piece_lower in seen_pieces:
+ problems.append(f"piece {piece}: listed twice")
+
+ seen_pieces.add(piece_lower)
+
+ if not saw_core and piece != "core":
+ problems.append(f'must list "core" before {piece}')
+ elif piece == "core":
+ saw_core = True
+
+ if saw_dev:
+ problems.append(f'must list "dev" last')
+ elif piece == "dev":
+ saw_dev = True
+
+ if not isinstance(deps, (list, tuple)):
+ problems.append(f'value for key "{piece}" should be list or tuple, see {deps!r}')
+ continue
+
+ if list(sorted(deps)) != list(deps):
+ problems.append(f"deps for key {piece} must be sorted. Correct order:\n {list(sorted(deps))!r}")
+
+ piece_deps = set()
+ for d in deps:
+ if CONSTRAINT_REGEX.search(d):
+ problems.append(f"for piece {piece}: dependency {d} should not specify a version. "
+ "Add it to CONSTRAINTS instead.")
+
+ if d.lower() in piece_deps:
+ problems.append(f'for piece {piece}: dependency {d} listed twice')
+
+ piece_deps.add(d.lower())
+
+ extras_pieces = [k for (k, _) in REQUIREMENTS_BY_PIECE if k not in ("dev", "core")]
+ sorted_extras_pieces = list(sorted(extras_pieces))
+ if sorted_extras_pieces != list(extras_pieces):
+ problems.append('pieces other than "core" and "dev" must appear in alphabetical order'
+ f"{sorted_extras_pieces}")
+
+ return problems
+
+
+def parse_semver(package, constraint, problems):
+ """Parse a semantic versioning constraint of the form "^X.[.Y[.Z[...]]]]"
+
+ Parameters
+ ----------
+ package : str
+ Name of the package specifying this constraint, for reporting problems.
+ constraint : str
+ The semver constraint. Must start with "^"
+ problems : List[str]
+ A list of strings describing problems that have occurred validating the configuration.
+ Problems encountered while validating constraint are appended to this list.
+
+ Returns
+ -------
+ List[str], int :
+ A 2-tuple. The first element is a list containing an entry for each component in the
+ semver string (components separated by "."). The second element is the index of the
+ component in the list which must not change to meet the semver constraint.
+
+ See "Caret requirements" at https://python-poetry.org/docs/versions/.
+ """
+ m = SEMVER_REGEX.match(constraint[1:])
+ if not m:
+ problems.append(f"{package}: invalid semver constraint {constraint}")
+ return [], 0
+
+ min_ver_parts = list(m.groups())
+
+ # Remove the end
+ while min_ver_parts and min_ver_parts[-1] is None or not min_ver_parts[-1].strip():
+ min_ver_parts.pop()
+
+ for i, p in enumerate(min_ver_parts):
+ try:
+ x = int(p.strip())
+ except ValueError as p:
+ if i == len(parts) - 1 and i > 0:
+ # If a pre-release tag is specified (i.e. 1.2.0pre3), allow.
+ return min_ver_parts, i - 1
+
+ problems.append(
+ f"{package}: semver constraint with non-numeric leading component: {constraint}")
+ return [], 0
+
+ if x:
+ return min_ver_parts, i
+
+ else:
+ # All 0's
+ return min_ver_parts, 0
+
+
+def validate_constraints():
+ problems = []
+
+ if not isinstance(CONSTRAINTS, (list, tuple)):
+ problems.append(f"must be list or tuple, see: {CONSTRAINTS!r}")
+
+ seen_packages = set()
+ all_deps = set()
+ for _, deps in REQUIREMENTS_BY_PIECE:
+ for d in deps:
+ all_deps.add(d.lower())
+
+ for package, constraint in CONSTRAINTS:
+ if package in seen_packages:
+ problems.append(f"{package}: specified twice")
+
+ if package.lower() not in all_deps:
+ problems.append(f"{package}: not specified in REQUIREMENTS_BY_PIECE")
+
+ if not CONSTRAINT_REGEX.match(constraint):
+ problems.append(f'{package}: constraint "{constraint}" does not look like a valid constraint')
+
+ if constraint.startswith("^"):
+ parse_semver(package, constraint, problems)
+
+ all_constrained_packages = [p for (p, _) in CONSTRAINTS]
+ sorted_constrained_packages = list(sorted(all_constrained_packages))
+ if sorted_constrained_packages != all_constrained_packages:
+ problems.append("CONSTRAINTS entries should be in this sorted order: "
+ f'{sorted_constrained_packages}')
+
+ return problems
+
+
+class ValidationError(Exception):
+ """Raised when a validation error occurs."""
+
+ @staticmethod
+ def format_problems(config, problems):
+ formatted = []
+ for p in problems:
+ assert isinstance(p, str), f"problems element not a str: {p}"
+ formatted.append('\n'.join(
+ textwrap.wrap(f"{config}: {p}", width=80, initial_indent=" * ", subsequent_indent=" ")))
+
+ return '\n'.join(formatted)
+
+ def __init__(self, config, problems):
+ super(ValidationError, self).__init__(self.format_problems(config, problems))
+ self.problems = problems
+
+
+def validate_or_raise():
+ problems = validate_requirements_by_piece()
+ if problems:
+ raise ValidationError("REQUIREMENTS_BY_PIECE", problems)
+
+ problems = validate_constraints()
+ if problems:
+ raise ValidationError("CONSTRAINTS", problems)
+
+
+def semver_to_requirements(piece, dep, constraint, joined_deps):
+ problems = []
+ min_ver_parts, fixed_index = parse_semver(dep, constraint, problems)
+ assert not problems
+
+ # i is now the index of the version component which must not change.
Review comment:
This probably needs some updating to reflect the code.
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
+
+ # Development requirements
+ ("dev", ["matplotlib", "pillow"]),
+]
+
+# Maps a named Python package (which should appear in REQUIREMENTS_BY_PIECE above) to a
+# semver or pip version constraint. Semver constraints are translated into requirements.txt
+# constraints.
+CONSTRAINTS = [
+ ("onnx", ">=1.7.0"),
+ ("onnxruntime", ">=1.0.0"),
+ ("pillow", "<7"),
+ ("synr", ">=0.2.1"),
+ ("tensorflow", ">=2.1.0"),
+ ("tflite", ">=2.1.0"),
+ ("torch", "^1.7.0"),
+ ("torchvision", ">=0.5.0"),
+]
+
+################################################################################
+# End of configuration options.
+################################################################################
+
+
+
+
+# Required keys in REQUIREMENTS_BY_PIECE.
+REQUIRED_PIECES = ["core", "dev"]
+
+# Regex to validates piece names.
+PIECE_REGEX = re.compile(r"^[a-z0-9][a-z0-9-]*", re.IGNORECASE)
+
+# Regex to match a constraint specification. Multiple constraints are not supported.
+CONSTRAINT_REGEX = re.compile(r"(?:\^|\<|(?:<=)|(?:==)|(?:>=)|\>)[^<>=\^,]+")
+
+# Regex for parsing semantic versions. See
+# https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string
+SEMVER_REGEX = re.compile(r"^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$")
+
+
+def validate_requirements_by_piece():
+ problems = []
+
+ unseen_required_pieces = set(REQUIRED_PIECES)
+ seen_pieces = set()
+
+ # Ensure that core is listed first and dev is listed last.
+ saw_core = False
+ saw_dev = False
+
+ if not isinstance(REQUIREMENTS_BY_PIECE, (list, tuple)):
+ problems.append(f"must be list or tuple, see {REQUIREMENTS_BY_PIECE!r}")
+
+ for piece, deps in REQUIREMENTS_BY_PIECE:
+ if not isinstance(piece, str):
+ problems.append(f"key {piece!r}: must be str, see {piece!r}")
+ continue
+
+ if piece in unseen_required_pieces:
+ unseen_required_pieces.remove(piece)
+
+ piece_lower = piece.lower()
+ if piece_lower in seen_pieces:
+ problems.append(f"piece {piece}: listed twice")
+
+ seen_pieces.add(piece_lower)
+
+ if not saw_core and piece != "core":
+ problems.append(f'must list "core" before {piece}')
+ elif piece == "core":
+ saw_core = True
+
+ if saw_dev:
+ problems.append(f'must list "dev" last')
+ elif piece == "dev":
+ saw_dev = True
+
+ if not isinstance(deps, (list, tuple)):
+ problems.append(f'value for key "{piece}" should be list or tuple, see {deps!r}')
+ continue
+
+ if list(sorted(deps)) != list(deps):
+ problems.append(f"deps for key {piece} must be sorted. Correct order:\n {list(sorted(deps))!r}")
+
+ piece_deps = set()
+ for d in deps:
+ if CONSTRAINT_REGEX.search(d):
+ problems.append(f"for piece {piece}: dependency {d} should not specify a version. "
+ "Add it to CONSTRAINTS instead.")
+
+ if d.lower() in piece_deps:
+ problems.append(f'for piece {piece}: dependency {d} listed twice')
+
+ piece_deps.add(d.lower())
+
+ extras_pieces = [k for (k, _) in REQUIREMENTS_BY_PIECE if k not in ("dev", "core")]
+ sorted_extras_pieces = list(sorted(extras_pieces))
+ if sorted_extras_pieces != list(extras_pieces):
+ problems.append('pieces other than "core" and "dev" must appear in alphabetical order'
+ f"{sorted_extras_pieces}")
+
+ return problems
+
+
+def parse_semver(package, constraint, problems):
+ """Parse a semantic versioning constraint of the form "^X.[.Y[.Z[...]]]]"
+
+ Parameters
+ ----------
+ package : str
+ Name of the package specifying this constraint, for reporting problems.
+ constraint : str
+ The semver constraint. Must start with "^"
+ problems : List[str]
+ A list of strings describing problems that have occurred validating the configuration.
+ Problems encountered while validating constraint are appended to this list.
+
+ Returns
+ -------
+ List[str], int :
+ A 2-tuple. The first element is a list containing an entry for each component in the
+ semver string (components separated by "."). The second element is the index of the
+ component in the list which must not change to meet the semver constraint.
+
+ See "Caret requirements" at https://python-poetry.org/docs/versions/.
+ """
+ m = SEMVER_REGEX.match(constraint[1:])
+ if not m:
+ problems.append(f"{package}: invalid semver constraint {constraint}")
+ return [], 0
+
+ min_ver_parts = list(m.groups())
+
+ # Remove the end
+ while min_ver_parts and min_ver_parts[-1] is None or not min_ver_parts[-1].strip():
+ min_ver_parts.pop()
+
+ for i, p in enumerate(min_ver_parts):
+ try:
+ x = int(p.strip())
+ except ValueError as p:
+ if i == len(parts) - 1 and i > 0:
+ # If a pre-release tag is specified (i.e. 1.2.0pre3), allow.
+ return min_ver_parts, i - 1
+
+ problems.append(
+ f"{package}: semver constraint with non-numeric leading component: {constraint}")
+ return [], 0
+
+ if x:
+ return min_ver_parts, i
+
+ else:
+ # All 0's
+ return min_ver_parts, 0
+
+
+def validate_constraints():
+ problems = []
+
+ if not isinstance(CONSTRAINTS, (list, tuple)):
+ problems.append(f"must be list or tuple, see: {CONSTRAINTS!r}")
+
+ seen_packages = set()
+ all_deps = set()
+ for _, deps in REQUIREMENTS_BY_PIECE:
+ for d in deps:
+ all_deps.add(d.lower())
+
+ for package, constraint in CONSTRAINTS:
+ if package in seen_packages:
+ problems.append(f"{package}: specified twice")
+
+ if package.lower() not in all_deps:
+ problems.append(f"{package}: not specified in REQUIREMENTS_BY_PIECE")
+
+ if not CONSTRAINT_REGEX.match(constraint):
+ problems.append(f'{package}: constraint "{constraint}" does not look like a valid constraint')
+
+ if constraint.startswith("^"):
+ parse_semver(package, constraint, problems)
+
+ all_constrained_packages = [p for (p, _) in CONSTRAINTS]
+ sorted_constrained_packages = list(sorted(all_constrained_packages))
+ if sorted_constrained_packages != all_constrained_packages:
+ problems.append("CONSTRAINTS entries should be in this sorted order: "
+ f'{sorted_constrained_packages}')
+
+ return problems
+
+
+class ValidationError(Exception):
+ """Raised when a validation error occurs."""
+
+ @staticmethod
+ def format_problems(config, problems):
+ formatted = []
+ for p in problems:
+ assert isinstance(p, str), f"problems element not a str: {p}"
+ formatted.append('\n'.join(
+ textwrap.wrap(f"{config}: {p}", width=80, initial_indent=" * ", subsequent_indent=" ")))
+
+ return '\n'.join(formatted)
+
+ def __init__(self, config, problems):
+ super(ValidationError, self).__init__(self.format_problems(config, problems))
+ self.problems = problems
+
+
+def validate_or_raise():
+ problems = validate_requirements_by_piece()
+ if problems:
+ raise ValidationError("REQUIREMENTS_BY_PIECE", problems)
+
+ problems = validate_constraints()
+ if problems:
+ raise ValidationError("CONSTRAINTS", problems)
+
+
+def semver_to_requirements(piece, dep, constraint, joined_deps):
+ problems = []
+ min_ver_parts, fixed_index = parse_semver(dep, constraint, problems)
+ assert not problems
+
+ # i is now the index of the version component which must not change.
+ try:
+ fixed_part = int(min_ver_parts[fixed_index])
+ except ValueError:
+ problems.append(
+ f"piece {piece}: dependency {dep} has semver constraint with non-integer leading parts: {ver}")
+ return
+
+ max_ver_parts = (min_ver_parts[:fixed_index] +
+ [str(int(min_ver_parts[fixed_index]) + 1)] +
+ ["0" for _ in min_ver_parts[fixed_index + 1:]])
+ joined_deps.append(f'{dep}>={".".join(min_ver_parts)},<{".".join(max_ver_parts)}')
+
+
+def join_requirements():
+ """Validate, then join REQUIRMENTS_BY_PIECE against CONSTRAINTS and return the result.
+
+ Returns
+ -------
+ An OrderedDict containing REQUIREMENTS_BY_PIECE, except any dependency mentioned in CONSTRAINTS
+ is replaced by a setuptools-compatible constraint.
+ """
+ validate_or_raise()
+
+ constraints_map = collections.OrderedDict(
+ [(p.lower(), c) for (p, c) in CONSTRAINTS])
+
+ to_return = collections.OrderedDict()
+ all_deps = set()
+ for piece, deps in REQUIREMENTS_BY_PIECE:
+ joined_deps = []
+ for d in deps:
+ constraint = constraints_map.get(d.lower())
+ if constraint is None:
+ joined_deps.append(d)
+ continue
+
+ if constraint[0] == "^":
+ # Translate semver to requirements:
+ semver_to_requirements(piece, d, constraint, joined_deps)
+ else:
+ joined_deps.append(f"{d}{constraint}")
+
+ if piece != "dev":
+ all_deps.update(joined_deps)
+
+ to_return[piece] = joined_deps
+
+ to_return["all"] = all_deps
+
+ return to_return
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--lint", action="store_true",
+ help="Just lint dependencies, don't generate anything")
+ return parser.parse_args()
+
+
+def main():
+ args = parse_args()
+ try:
+ joined_deps = join_requirements()
+ except ValidationError as e:
+ print(f"ERROR: invalid requirements configuration in {__file__}:", file=sys.stderr)
+ print(str(e), file=sys.stderr)
+ sys.exit(2)
+
+ if args.lint:
+ sys.exit(0)
+
+ output_dir = os.path.join(os.path.dirname(__file__), "requirements")
+ if not os.path.exists(output_dir):
+ os.makedirs(output_dir)
+ elif not os.path.isdir(output_dir):
+ print(f"ERROR: output directory {output_dir} exists but is not a dir. Delete it",
+ file=sys.stderr)
+ sys.exit(2)
+
+ for piece, deps in joined_deps.items():
+ with open(os.path.join(output_dir, f"{piece}.txt"), "w") as f:
+ for d in deps:
+ f.write(f"{d}{os.linesep}")
+
+
+if __name__ == "__main__":
+ main()
Review comment:
I noticed this script is used to generate the files under `./requirements` and also used _inline_ within `setup.py`. When a dependency is updated, it will immediately reflect in setup.py - this is good.
However, how do we make sure that when a new dependency is introduced/changed/updated/fixed, the `txt` files are also correctly updated?
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
Review comment:
Can you set xgboost also as required by tvmc? It is offered as an option for `tvmc tune`, maybe it is a bug currently, but I just noticed it now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r566336625
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
+
+ # Development requirements
+ ("dev", ["matplotlib", "pillow"]),
+]
+
+# Maps a named Python package (which should appear in REQUIREMENTS_BY_PIECE above) to a
+# semver or pip version constraint. Semver constraints are translated into requirements.txt
+# constraints.
+CONSTRAINTS = [
+ ("onnx", ">=1.7.0"),
+ ("onnxruntime", ">=1.0.0"),
+ ("pillow", "<7"),
+ ("synr", ">=0.2.1"),
+ ("tensorflow", ">=2.1.0"),
+ ("tflite", ">=2.1.0"),
+ ("torch", "^1.7.0"),
+ ("torchvision", ">=0.5.0"),
+]
+
+################################################################################
+# End of configuration options.
+################################################################################
+
+
+
+
+# Required keys in REQUIREMENTS_BY_PIECE.
+REQUIRED_PIECES = ["core", "dev"]
+
+# Regex to validates piece names.
+PIECE_REGEX = re.compile(r"^[a-z0-9][a-z0-9-]*", re.IGNORECASE)
+
+# Regex to match a constraint specification. Multiple constraints are not supported.
+CONSTRAINT_REGEX = re.compile(r"(?:\^|\<|(?:<=)|(?:==)|(?:>=)|\>)[^<>=\^,]+")
+
+# Regex for parsing semantic versions. See
+# https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string
+SEMVER_REGEX = re.compile(r"^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$")
+
+
+def validate_requirements_by_piece():
+ problems = []
+
+ unseen_required_pieces = set(REQUIRED_PIECES)
+ seen_pieces = set()
+
+ # Ensure that core is listed first and dev is listed last.
+ saw_core = False
+ saw_dev = False
+
+ if not isinstance(REQUIREMENTS_BY_PIECE, (list, tuple)):
+ problems.append(f"must be list or tuple, see {REQUIREMENTS_BY_PIECE!r}")
+
+ for piece, deps in REQUIREMENTS_BY_PIECE:
+ if not isinstance(piece, str):
+ problems.append(f"key {piece!r}: must be str, see {piece!r}")
+ continue
+
+ if piece in unseen_required_pieces:
+ unseen_required_pieces.remove(piece)
+
+ piece_lower = piece.lower()
+ if piece_lower in seen_pieces:
+ problems.append(f"piece {piece}: listed twice")
+
+ seen_pieces.add(piece_lower)
+
+ if not saw_core and piece != "core":
+ problems.append(f'must list "core" before {piece}')
+ elif piece == "core":
+ saw_core = True
+
+ if saw_dev:
+ problems.append(f'must list "dev" last')
+ elif piece == "dev":
+ saw_dev = True
+
+ if not isinstance(deps, (list, tuple)):
+ problems.append(f'value for key "{piece}" should be list or tuple, see {deps!r}')
+ continue
+
+ if list(sorted(deps)) != list(deps):
+ problems.append(f"deps for key {piece} must be sorted. Correct order:\n {list(sorted(deps))!r}")
+
+ piece_deps = set()
+ for d in deps:
+ if CONSTRAINT_REGEX.search(d):
+ problems.append(f"for piece {piece}: dependency {d} should not specify a version. "
+ "Add it to CONSTRAINTS instead.")
+
+ if d.lower() in piece_deps:
+ problems.append(f'for piece {piece}: dependency {d} listed twice')
+
+ piece_deps.add(d.lower())
+
+ extras_pieces = [k for (k, _) in REQUIREMENTS_BY_PIECE if k not in ("dev", "core")]
+ sorted_extras_pieces = list(sorted(extras_pieces))
+ if sorted_extras_pieces != list(extras_pieces):
+ problems.append('pieces other than "core" and "dev" must appear in alphabetical order'
+ f"{sorted_extras_pieces}")
+
+ return problems
+
+
+def parse_semver(package, constraint, problems):
+ """Parse a semantic versioning constraint of the form "^X.[.Y[.Z[...]]]]"
+
+ Parameters
+ ----------
+ package : str
+ Name of the package specifying this constraint, for reporting problems.
+ constraint : str
+ The semver constraint. Must start with "^"
+ problems : List[str]
+ A list of strings describing problems that have occurred validating the configuration.
+ Problems encountered while validating constraint are appended to this list.
+
+ Returns
+ -------
+ List[str], int :
+ A 2-tuple. The first element is a list containing an entry for each component in the
+ semver string (components separated by "."). The second element is the index of the
+ component in the list which must not change to meet the semver constraint.
+
+ See "Caret requirements" at https://python-poetry.org/docs/versions/.
+ """
+ m = SEMVER_REGEX.match(constraint[1:])
+ if not m:
+ problems.append(f"{package}: invalid semver constraint {constraint}")
+ return [], 0
+
+ min_ver_parts = list(m.groups())
+
+ # Remove the end
+ while min_ver_parts and min_ver_parts[-1] is None or not min_ver_parts[-1].strip():
+ min_ver_parts.pop()
+
+ for i, p in enumerate(min_ver_parts):
+ try:
+ x = int(p.strip())
+ except ValueError as p:
+ if i == len(parts) - 1 and i > 0:
+ # If a pre-release tag is specified (i.e. 1.2.0pre3), allow.
+ return min_ver_parts, i - 1
+
+ problems.append(
+ f"{package}: semver constraint with non-numeric leading component: {constraint}")
+ return [], 0
+
+ if x:
+ return min_ver_parts, i
+
+ else:
+ # All 0's
+ return min_ver_parts, 0
+
+
+def validate_constraints():
+ problems = []
+
+ if not isinstance(CONSTRAINTS, (list, tuple)):
+ problems.append(f"must be list or tuple, see: {CONSTRAINTS!r}")
+
+ seen_packages = set()
+ all_deps = set()
+ for _, deps in REQUIREMENTS_BY_PIECE:
+ for d in deps:
+ all_deps.add(d.lower())
+
+ for package, constraint in CONSTRAINTS:
+ if package in seen_packages:
+ problems.append(f"{package}: specified twice")
+
+ if package.lower() not in all_deps:
+ problems.append(f"{package}: not specified in REQUIREMENTS_BY_PIECE")
+
+ if not CONSTRAINT_REGEX.match(constraint):
+ problems.append(f'{package}: constraint "{constraint}" does not look like a valid constraint')
+
+ if constraint.startswith("^"):
+ parse_semver(package, constraint, problems)
+
+ all_constrained_packages = [p for (p, _) in CONSTRAINTS]
+ sorted_constrained_packages = list(sorted(all_constrained_packages))
+ if sorted_constrained_packages != all_constrained_packages:
+ problems.append("CONSTRAINTS entries should be in this sorted order: "
+ f'{sorted_constrained_packages}')
+
+ return problems
+
+
+class ValidationError(Exception):
+ """Raised when a validation error occurs."""
+
+ @staticmethod
+ def format_problems(config, problems):
+ formatted = []
+ for p in problems:
+ assert isinstance(p, str), f"problems element not a str: {p}"
+ formatted.append('\n'.join(
+ textwrap.wrap(f"{config}: {p}", width=80, initial_indent=" * ", subsequent_indent=" ")))
+
+ return '\n'.join(formatted)
+
+ def __init__(self, config, problems):
+ super(ValidationError, self).__init__(self.format_problems(config, problems))
+ self.problems = problems
+
+
+def validate_or_raise():
+ problems = validate_requirements_by_piece()
+ if problems:
+ raise ValidationError("REQUIREMENTS_BY_PIECE", problems)
+
+ problems = validate_constraints()
+ if problems:
+ raise ValidationError("CONSTRAINTS", problems)
+
+
+def semver_to_requirements(piece, dep, constraint, joined_deps):
+ problems = []
+ min_ver_parts, fixed_index = parse_semver(dep, constraint, problems)
+ assert not problems
+
+ # i is now the index of the version component which must not change.
+ try:
+ fixed_part = int(min_ver_parts[fixed_index])
+ except ValueError:
+ problems.append(
+ f"piece {piece}: dependency {dep} has semver constraint with non-integer leading parts: {ver}")
+ return
+
+ max_ver_parts = (min_ver_parts[:fixed_index] +
+ [str(int(min_ver_parts[fixed_index]) + 1)] +
+ ["0" for _ in min_ver_parts[fixed_index + 1:]])
+ joined_deps.append(f'{dep}>={".".join(min_ver_parts)},<{".".join(max_ver_parts)}')
+
+
+def join_requirements():
+ """Validate, then join REQUIRMENTS_BY_PIECE against CONSTRAINTS and return the result.
+
+ Returns
+ -------
+ An OrderedDict containing REQUIREMENTS_BY_PIECE, except any dependency mentioned in CONSTRAINTS
+ is replaced by a setuptools-compatible constraint.
+ """
+ validate_or_raise()
+
+ constraints_map = collections.OrderedDict(
+ [(p.lower(), c) for (p, c) in CONSTRAINTS])
+
+ to_return = collections.OrderedDict()
+ all_deps = set()
+ for piece, deps in REQUIREMENTS_BY_PIECE:
+ joined_deps = []
+ for d in deps:
+ constraint = constraints_map.get(d.lower())
+ if constraint is None:
+ joined_deps.append(d)
+ continue
+
+ if constraint[0] == "^":
+ # Translate semver to requirements:
+ semver_to_requirements(piece, d, constraint, joined_deps)
+ else:
+ joined_deps.append(f"{d}{constraint}")
+
+ if piece != "dev":
+ all_deps.update(joined_deps)
+
+ to_return[piece] = joined_deps
+
+ to_return["all"] = all_deps
+
+ return to_return
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--lint", action="store_true",
+ help="Just lint dependencies, don't generate anything")
+ return parser.parse_args()
+
+
+def main():
+ args = parse_args()
+ try:
+ joined_deps = join_requirements()
+ except ValidationError as e:
+ print(f"ERROR: invalid requirements configuration in {__file__}:", file=sys.stderr)
+ print(str(e), file=sys.stderr)
+ sys.exit(2)
+
+ if args.lint:
+ sys.exit(0)
+
+ output_dir = os.path.join(os.path.dirname(__file__), "requirements")
+ if not os.path.exists(output_dir):
+ os.makedirs(output_dir)
+ elif not os.path.isdir(output_dir):
+ print(f"ERROR: output directory {output_dir} exists but is not a dir. Delete it",
+ file=sys.stderr)
+ sys.exit(2)
+
+ for piece, deps in joined_deps.items():
+ with open(os.path.join(output_dir, f"{piece}.txt"), "w") as f:
+ for d in deps:
+ f.write(f"{d}{os.linesep}")
+
+
+if __name__ == "__main__":
+ main()
Review comment:
in thinking about this further, i'm starting to lean towards not updating the requirements.txt when `setup.py egg_info` is invoked. here's why:
- given the discussion about [constraint types](https://discuss.tvm.apache.org/t/rfc-consolidating-tvm-python-dependencies/8329/27) in the RFC, the functional constraints aren't themselves particularly useful in `pip install -r`. you need additional constraints to indicate which version of packages are used by the CI.
- in order to generate those we need another RFC. presumably, once we have those, we can change the language about installing Python dependencies in the Install From Source tutorial to tell you to use the CI-tested `requirements.txt`.
- The CI-tested requirements.txt needs to be generated with slightly different inputs (I.e. the requirements here + the output of `pip freeze` from various CI containers). I think we would create a slightly different tool or modify this one to include configuration on how to translate `pip freeze` into looser requirements.txt usable for everyday work.
- until we've completed all of this, I don't think we'd recommend people use these requirements. if they want them, it's pretty easy to generate by just running `python gen_requirements.txt`. I don't intend to actually check in the files in `python/requirements` to the repo (rather add .gitignore), so they wouldn't be out of date for most people.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r568070669
##########
File path: python/setup.py
##########
@@ -171,38 +171,25 @@ def get_package_data_files():
return ["relay/std/prelude.rly", "relay/std/core.rly"]
+# Temporarily add this directory to the path so we can import the requirements generator
+# tool.
+sys.path.insert(0, os.path.dirname(__file__))
+import gen_requirements
+sys.path.pop(0)
+
+requirements = gen_requirements.join_requirements()
Review comment:
I guess, I don't typically expect to see the requirements printed when I run `python setup.py`. are there cases where you do? I could add a comment saying: to produce requirements.txt files, run `python gen_requirements.py` above this line, which is kind of the typical case i'd expect someone to be looking for them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] mdw-octoml commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
mdw-octoml commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r568063457
##########
File path: tests/python/unittest/test_gen_requirements.py
##########
@@ -0,0 +1,220 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Tests for gen_requirements, found in python/."""
+
+import collections
+import contextlib
+import os
+import sys
+
+import tvm
+
+import pytest
+
+# Insert the parent dir to python/tvm into the import path, so that gen_requirements may be
+# imported.
+sys.path.insert(0, os.path.dirname(tvm.__file__))
+try:
+ import gen_requirements
+finally:
+ sys.path.pop(0)
+
+
+@contextlib.contextmanager
+def patch(obj, **kw):
+ old = {}
+ for prop_name, new in kw.items():
+ old[prop_name] = getattr(obj, prop_name)
+ setattr(obj, prop_name, new)
+ yield
+ for prop_name, value in old.items():
+ setattr(obj, prop_name, value)
+
+
+PROBLEM_REQUIREMENTS = [
+ ("extras-pre-core", ("", ["foo", 123])), # entry before core
+ (456, ("", ["foo", "bar"])), # invalid extras name, deps should not be processed
+ ("core", ("", ["foo"])), # ordinary core entry.
+ ("wrong-description-type", (None, ["foo"])), # wrong description type
+ ("bad-value", None), # value field is not a 2-tuple
+ ("bad-value-2", ("", ["foo"], 34)), # value field is not a 2-tuple
+ ("invalid", ("", ["qux"])), # duplicate invalid entry, all items valid.
+ ("extras-foo", ("", ["bar", "baz"])), # ordinary extras entry.
+ ("invalid", ("", ["baz", None, 123])), # valid extra name, invalid deps.
+ ("unsorted", ("", ["qux", "bar", "foo"])), # deps out of order
+ ("versioned_dep", ("", ["baz==1.2", "foo==^2.0", "buz<3", "bar>4"])),
+ ("duplicate_dep", ("", ["buz", "buz", "foo"])), # duplicate listed dependency
+ ("dev", ("", ["baz", "qux"])), # ordinary dev entry.
+ ("extras-post-dev", ("", ["bar", "buzz"])), # entry after dev
+]
+
+
+def test_validate_requirements():
+ with patch(gen_requirements, REQUIREMENTS_BY_PIECE=None):
+ assert gen_requirements.validate_requirements_by_piece() == [
+ "must be list or tuple, see None"
+ ]
+
+ with patch(gen_requirements, REQUIREMENTS_BY_PIECE=PROBLEM_REQUIREMENTS):
+ problems = gen_requirements.validate_requirements_by_piece()
+ assert problems == [
+ 'piece extras-pre-core: must list after "core" (core must be first)',
+ "piece extras-pre-core: deps should be a list of strings, got ['foo', 123]",
+ "piece 456: must be str",
+ "piece wrong-description-type: description should be a string, got None",
+ (
+ 'piece bad-value: should be formatted like ("bad-value", ("<requirements.txt '
+ 'comment>", ["dep1", "dep2", ...])). got: None'
+ ),
+ (
+ 'piece bad-value-2: should be formatted like ("bad-value-2", '
+ '("<requirements.txt comment>", ["dep1", "dep2", ...])). got: (\'\', '
+ "['foo'], 34)"
+ ),
+ "piece invalid: listed twice",
+ "piece invalid: deps should be a list of strings, got ['baz', None, 123]",
+ "piece unsorted: deps must be sorted. Correct order:\n ['bar', 'foo', 'qux']",
+ "piece versioned_dep: deps must be sorted. Correct order:\n ['bar>4', 'baz==1.2', 'buz<3', 'foo==^2.0']",
+ "piece versioned_dep: dependency baz==1.2 should not specify a version. Add it to CONSTRAINTS instead.",
+ "piece versioned_dep: dependency foo==^2.0 should not specify a version. Add it to CONSTRAINTS instead.",
+ "piece versioned_dep: dependency buz<3 should not specify a version. Add it to CONSTRAINTS instead.",
+ "piece versioned_dep: dependency bar>4 should not specify a version. Add it to CONSTRAINTS instead.",
+ "piece duplicate_dep: dependency buz listed twice",
+ 'piece extras-post-dev: must list before "dev" (dev must be last)',
+ 'pieces other than "core" and "dev" must appear in alphabetical order: '
+ "['bad-value', 'bad-value-2', 'duplicate_dep', 'extras-foo', 'extras-post-dev', "
+ "'extras-pre-core', 'invalid', 'invalid', 'unsorted', 'versioned_dep', "
+ "'wrong-description-type']",
+ ]
+
+
+TEST_REQUIREMENTS_BY_PIECE = (
+ ("core", ("core tvm requirements", ("bar", "foo", "non-constrained"))),
+ ("extra-one", ("requirements for one feature", ("baz", "qux"))),
+ ("extra-two", ("requirements for two feature", ("buz", "qux", "semver-minor", "semver-patch"))),
+ ("dev", ("requirements for dev", ("buz", "oof", "rab"))),
+)
+
+
+def test_validate_constraints():
+ with patch(
+ gen_requirements,
+ REQUIREMENTS_BY_PIECE=TEST_REQUIREMENTS_BY_PIECE,
+ CONSTRAINTS=(
+ ("unlisted", "~=3"),
Review comment:
Same comment as above, I think it would be better if the problems were kept along with the input constraints that are being tested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r564773562
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
Review comment:
thanks! added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tkonolige commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
tkonolige commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r558447476
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
Review comment:
I think you need tornado and cloudpickle too. They are requirements for tuning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] mdw-octoml commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
mdw-octoml commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r568062806
##########
File path: tests/python/unittest/test_gen_requirements.py
##########
@@ -0,0 +1,220 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Tests for gen_requirements, found in python/."""
+
+import collections
+import contextlib
+import os
+import sys
+
+import tvm
+
+import pytest
+
+# Insert the parent dir to python/tvm into the import path, so that gen_requirements may be
+# imported.
+sys.path.insert(0, os.path.dirname(tvm.__file__))
+try:
+ import gen_requirements
+finally:
+ sys.path.pop(0)
+
+
+@contextlib.contextmanager
+def patch(obj, **kw):
+ old = {}
+ for prop_name, new in kw.items():
+ old[prop_name] = getattr(obj, prop_name)
+ setattr(obj, prop_name, new)
+ yield
+ for prop_name, value in old.items():
+ setattr(obj, prop_name, value)
+
+
+PROBLEM_REQUIREMENTS = [
Review comment:
Nit: I think this test would be easier to maintain if the expected problem string were part of this list, rather than hardcoded in the test case in the same order below.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] leandron commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r564759432
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
Review comment:
@areusch the version restrictions comes from https://github.com/apache/tvm/issues/4953
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#issuecomment-770067836
@leandron @mdw-octoml @tkonolige @comaniac @tqchen please take another look when you have a minute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r564728329
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
Review comment:
@comaniac @tqchen any idea why it is restricted to that version?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r564706225
##########
File path: python/setup.py
##########
@@ -171,38 +171,25 @@ def get_package_data_files():
return ["relay/std/prelude.rly", "relay/std/core.rly"]
+# Temporarily add this directory to the path so we can import the requirements generator
+# tool.
+sys.path.insert(0, os.path.dirname(__file__))
+import gen_requirements
+sys.path.pop(0)
+
+requirements = gen_requirements.join_requirements()
Review comment:
what do you mean exactly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r564706852
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
+
+ # Development requirements
+ ("dev", ["matplotlib", "pillow"]),
+]
+
+# Maps a named Python package (which should appear in REQUIREMENTS_BY_PIECE above) to a
+# semver or pip version constraint. Semver constraints are translated into requirements.txt
+# constraints.
+CONSTRAINTS = [
+ ("onnx", ">=1.7.0"),
+ ("onnxruntime", ">=1.0.0"),
+ ("pillow", "<7"),
+ ("synr", ">=0.2.1"),
+ ("tensorflow", ">=2.1.0"),
+ ("tflite", ">=2.1.0"),
+ ("torch", "^1.7.0"),
+ ("torchvision", ">=0.5.0"),
+]
+
+################################################################################
+# End of configuration options.
+################################################################################
+
+
+
+
+# Required keys in REQUIREMENTS_BY_PIECE.
+REQUIRED_PIECES = ["core", "dev"]
+
+# Regex to validates piece names.
+PIECE_REGEX = re.compile(r"^[a-z0-9][a-z0-9-]*", re.IGNORECASE)
+
+# Regex to match a constraint specification. Multiple constraints are not supported.
+CONSTRAINT_REGEX = re.compile(r"(?:\^|\<|(?:<=)|(?:==)|(?:>=)|\>)[^<>=\^,]+")
+
+# Regex for parsing semantic versions. See
+# https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string
+SEMVER_REGEX = re.compile(r"^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$")
+
+
+def validate_requirements_by_piece():
+ problems = []
+
+ unseen_required_pieces = set(REQUIRED_PIECES)
+ seen_pieces = set()
+
+ # Ensure that core is listed first and dev is listed last.
+ saw_core = False
+ saw_dev = False
+
+ if not isinstance(REQUIREMENTS_BY_PIECE, (list, tuple)):
+ problems.append(f"must be list or tuple, see {REQUIREMENTS_BY_PIECE!r}")
+
+ for piece, deps in REQUIREMENTS_BY_PIECE:
+ if not isinstance(piece, str):
+ problems.append(f"key {piece!r}: must be str, see {piece!r}")
+ continue
+
+ if piece in unseen_required_pieces:
+ unseen_required_pieces.remove(piece)
+
+ piece_lower = piece.lower()
+ if piece_lower in seen_pieces:
+ problems.append(f"piece {piece}: listed twice")
+
+ seen_pieces.add(piece_lower)
+
+ if not saw_core and piece != "core":
+ problems.append(f'must list "core" before {piece}')
+ elif piece == "core":
+ saw_core = True
+
+ if saw_dev:
+ problems.append(f'must list "dev" last')
+ elif piece == "dev":
+ saw_dev = True
+
+ if not isinstance(deps, (list, tuple)):
+ problems.append(f'value for key "{piece}" should be list or tuple, see {deps!r}')
+ continue
+
+ if list(sorted(deps)) != list(deps):
+ problems.append(f"deps for key {piece} must be sorted. Correct order:\n {list(sorted(deps))!r}")
+
+ piece_deps = set()
+ for d in deps:
+ if CONSTRAINT_REGEX.search(d):
+ problems.append(f"for piece {piece}: dependency {d} should not specify a version. "
+ "Add it to CONSTRAINTS instead.")
+
+ if d.lower() in piece_deps:
+ problems.append(f'for piece {piece}: dependency {d} listed twice')
+
+ piece_deps.add(d.lower())
+
+ extras_pieces = [k for (k, _) in REQUIREMENTS_BY_PIECE if k not in ("dev", "core")]
+ sorted_extras_pieces = list(sorted(extras_pieces))
+ if sorted_extras_pieces != list(extras_pieces):
+ problems.append('pieces other than "core" and "dev" must appear in alphabetical order'
+ f"{sorted_extras_pieces}")
+
+ return problems
+
+
+def parse_semver(package, constraint, problems):
+ """Parse a semantic versioning constraint of the form "^X.[.Y[.Z[...]]]]"
+
+ Parameters
+ ----------
+ package : str
+ Name of the package specifying this constraint, for reporting problems.
+ constraint : str
+ The semver constraint. Must start with "^"
+ problems : List[str]
+ A list of strings describing problems that have occurred validating the configuration.
+ Problems encountered while validating constraint are appended to this list.
+
+ Returns
+ -------
+ List[str], int :
+ A 2-tuple. The first element is a list containing an entry for each component in the
+ semver string (components separated by "."). The second element is the index of the
+ component in the list which must not change to meet the semver constraint.
+
+ See "Caret requirements" at https://python-poetry.org/docs/versions/.
+ """
+ m = SEMVER_REGEX.match(constraint[1:])
+ if not m:
+ problems.append(f"{package}: invalid semver constraint {constraint}")
+ return [], 0
+
+ min_ver_parts = list(m.groups())
+
+ # Remove the end
+ while min_ver_parts and min_ver_parts[-1] is None or not min_ver_parts[-1].strip():
+ min_ver_parts.pop()
+
+ for i, p in enumerate(min_ver_parts):
+ try:
+ x = int(p.strip())
+ except ValueError as p:
+ if i == len(parts) - 1 and i > 0:
+ # If a pre-release tag is specified (i.e. 1.2.0pre3), allow.
+ return min_ver_parts, i - 1
+
+ problems.append(
+ f"{package}: semver constraint with non-numeric leading component: {constraint}")
+ return [], 0
+
+ if x:
+ return min_ver_parts, i
+
+ else:
+ # All 0's
+ return min_ver_parts, 0
+
+
+def validate_constraints():
+ problems = []
+
+ if not isinstance(CONSTRAINTS, (list, tuple)):
+ problems.append(f"must be list or tuple, see: {CONSTRAINTS!r}")
Review comment:
you still need to use mypy though right? I don't know that we enforce that in our pre-checkin lint right now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r558498417
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
Review comment:
right, I generally prefer yaml for a task like this but since this is the deps script, I don't think it should have external dependencies itself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] mdw-octoml commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
mdw-octoml commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r558479917
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
+
+ # Development requirements
+ ("dev", ["matplotlib", "pillow"]),
+]
+
+# Maps a named Python package (which should appear in REQUIREMENTS_BY_PIECE above) to a
+# semver or pip version constraint. Semver constraints are translated into requirements.txt
+# constraints.
+CONSTRAINTS = [
+ ("onnx", ">=1.7.0"),
+ ("onnxruntime", ">=1.0.0"),
+ ("pillow", "<7"),
+ ("synr", ">=0.2.1"),
+ ("tensorflow", ">=2.1.0"),
+ ("tflite", ">=2.1.0"),
+ ("torch", "^1.7.0"),
+ ("torchvision", ">=0.5.0"),
+]
+
+################################################################################
+# End of configuration options.
+################################################################################
+
+
+
+
+# Required keys in REQUIREMENTS_BY_PIECE.
+REQUIRED_PIECES = ["core", "dev"]
+
+# Regex to validates piece names.
+PIECE_REGEX = re.compile(r"^[a-z0-9][a-z0-9-]*", re.IGNORECASE)
+
+# Regex to match a constraint specification. Multiple constraints are not supported.
+CONSTRAINT_REGEX = re.compile(r"(?:\^|\<|(?:<=)|(?:==)|(?:>=)|\>)[^<>=\^,]+")
Review comment:
Have you considered using https://pypi.org/project/semver/ instead of rolling your own? Or is this script required to run with no deps?
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
+
+ # Development requirements
+ ("dev", ["matplotlib", "pillow"]),
+]
+
+# Maps a named Python package (which should appear in REQUIREMENTS_BY_PIECE above) to a
+# semver or pip version constraint. Semver constraints are translated into requirements.txt
+# constraints.
+CONSTRAINTS = [
+ ("onnx", ">=1.7.0"),
+ ("onnxruntime", ">=1.0.0"),
+ ("pillow", "<7"),
+ ("synr", ">=0.2.1"),
+ ("tensorflow", ">=2.1.0"),
+ ("tflite", ">=2.1.0"),
+ ("torch", "^1.7.0"),
+ ("torchvision", ">=0.5.0"),
+]
+
+################################################################################
+# End of configuration options.
+################################################################################
+
+
+
+
+# Required keys in REQUIREMENTS_BY_PIECE.
+REQUIRED_PIECES = ["core", "dev"]
+
+# Regex to validates piece names.
+PIECE_REGEX = re.compile(r"^[a-z0-9][a-z0-9-]*", re.IGNORECASE)
+
+# Regex to match a constraint specification. Multiple constraints are not supported.
+CONSTRAINT_REGEX = re.compile(r"(?:\^|\<|(?:<=)|(?:==)|(?:>=)|\>)[^<>=\^,]+")
+
+# Regex for parsing semantic versions. See
+# https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string
+SEMVER_REGEX = re.compile(r"^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$")
+
+
+def validate_requirements_by_piece():
Review comment:
For new code I would recommend using Python type annotations.
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
+
+ # Development requirements
+ ("dev", ["matplotlib", "pillow"]),
+]
+
+# Maps a named Python package (which should appear in REQUIREMENTS_BY_PIECE above) to a
+# semver or pip version constraint. Semver constraints are translated into requirements.txt
+# constraints.
+CONSTRAINTS = [
+ ("onnx", ">=1.7.0"),
+ ("onnxruntime", ">=1.0.0"),
+ ("pillow", "<7"),
+ ("synr", ">=0.2.1"),
+ ("tensorflow", ">=2.1.0"),
+ ("tflite", ">=2.1.0"),
+ ("torch", "^1.7.0"),
+ ("torchvision", ">=0.5.0"),
+]
+
+################################################################################
+# End of configuration options.
+################################################################################
+
+
+
+
+# Required keys in REQUIREMENTS_BY_PIECE.
+REQUIRED_PIECES = ["core", "dev"]
+
+# Regex to validates piece names.
+PIECE_REGEX = re.compile(r"^[a-z0-9][a-z0-9-]*", re.IGNORECASE)
+
+# Regex to match a constraint specification. Multiple constraints are not supported.
+CONSTRAINT_REGEX = re.compile(r"(?:\^|\<|(?:<=)|(?:==)|(?:>=)|\>)[^<>=\^,]+")
+
+# Regex for parsing semantic versions. See
+# https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string
+SEMVER_REGEX = re.compile(r"^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$")
+
+
+def validate_requirements_by_piece():
+ problems = []
+
+ unseen_required_pieces = set(REQUIRED_PIECES)
+ seen_pieces = set()
+
+ # Ensure that core is listed first and dev is listed last.
+ saw_core = False
+ saw_dev = False
+
+ if not isinstance(REQUIREMENTS_BY_PIECE, (list, tuple)):
+ problems.append(f"must be list or tuple, see {REQUIREMENTS_BY_PIECE!r}")
+
+ for piece, deps in REQUIREMENTS_BY_PIECE:
+ if not isinstance(piece, str):
+ problems.append(f"key {piece!r}: must be str, see {piece!r}")
+ continue
+
+ if piece in unseen_required_pieces:
+ unseen_required_pieces.remove(piece)
+
+ piece_lower = piece.lower()
+ if piece_lower in seen_pieces:
+ problems.append(f"piece {piece}: listed twice")
+
+ seen_pieces.add(piece_lower)
+
+ if not saw_core and piece != "core":
+ problems.append(f'must list "core" before {piece}')
+ elif piece == "core":
+ saw_core = True
+
+ if saw_dev:
+ problems.append(f'must list "dev" last')
+ elif piece == "dev":
+ saw_dev = True
+
+ if not isinstance(deps, (list, tuple)):
+ problems.append(f'value for key "{piece}" should be list or tuple, see {deps!r}')
+ continue
+
+ if list(sorted(deps)) != list(deps):
+ problems.append(f"deps for key {piece} must be sorted. Correct order:\n {list(sorted(deps))!r}")
+
+ piece_deps = set()
+ for d in deps:
+ if CONSTRAINT_REGEX.search(d):
+ problems.append(f"for piece {piece}: dependency {d} should not specify a version. "
+ "Add it to CONSTRAINTS instead.")
+
+ if d.lower() in piece_deps:
+ problems.append(f'for piece {piece}: dependency {d} listed twice')
+
+ piece_deps.add(d.lower())
+
+ extras_pieces = [k for (k, _) in REQUIREMENTS_BY_PIECE if k not in ("dev", "core")]
+ sorted_extras_pieces = list(sorted(extras_pieces))
+ if sorted_extras_pieces != list(extras_pieces):
+ problems.append('pieces other than "core" and "dev" must appear in alphabetical order'
+ f"{sorted_extras_pieces}")
+
+ return problems
+
+
+def parse_semver(package, constraint, problems):
+ """Parse a semantic versioning constraint of the form "^X.[.Y[.Z[...]]]]"
+
+ Parameters
+ ----------
+ package : str
+ Name of the package specifying this constraint, for reporting problems.
+ constraint : str
+ The semver constraint. Must start with "^"
+ problems : List[str]
+ A list of strings describing problems that have occurred validating the configuration.
+ Problems encountered while validating constraint are appended to this list.
+
+ Returns
+ -------
+ List[str], int :
+ A 2-tuple. The first element is a list containing an entry for each component in the
+ semver string (components separated by "."). The second element is the index of the
+ component in the list which must not change to meet the semver constraint.
+
+ See "Caret requirements" at https://python-poetry.org/docs/versions/.
+ """
+ m = SEMVER_REGEX.match(constraint[1:])
+ if not m:
+ problems.append(f"{package}: invalid semver constraint {constraint}")
+ return [], 0
+
+ min_ver_parts = list(m.groups())
+
+ # Remove the end
+ while min_ver_parts and min_ver_parts[-1] is None or not min_ver_parts[-1].strip():
+ min_ver_parts.pop()
+
+ for i, p in enumerate(min_ver_parts):
+ try:
+ x = int(p.strip())
+ except ValueError as p:
+ if i == len(parts) - 1 and i > 0:
+ # If a pre-release tag is specified (i.e. 1.2.0pre3), allow.
+ return min_ver_parts, i - 1
+
+ problems.append(
+ f"{package}: semver constraint with non-numeric leading component: {constraint}")
+ return [], 0
+
+ if x:
+ return min_ver_parts, i
+
+ else:
+ # All 0's
+ return min_ver_parts, 0
+
+
+def validate_constraints():
+ problems = []
+
+ if not isinstance(CONSTRAINTS, (list, tuple)):
+ problems.append(f"must be list or tuple, see: {CONSTRAINTS!r}")
Review comment:
This would be an unnecessary check if you used Python type annotations.
##########
File path: python/requirements/all.txt
##########
@@ -0,0 +1,16 @@
+numpy
Review comment:
I'd recommend adding comments at the top of each of these files to indicate what they are used for, since it may not always be obvious from the filename. PIP requirements files support `#` comments.https://pip.pypa.io/en/latest/reference/pip_install/#requirements-file-format
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
+
+ # Development requirements
+ ("dev", ["matplotlib", "pillow"]),
+]
+
+# Maps a named Python package (which should appear in REQUIREMENTS_BY_PIECE above) to a
+# semver or pip version constraint. Semver constraints are translated into requirements.txt
+# constraints.
+CONSTRAINTS = [
+ ("onnx", ">=1.7.0"),
+ ("onnxruntime", ">=1.0.0"),
+ ("pillow", "<7"),
+ ("synr", ">=0.2.1"),
+ ("tensorflow", ">=2.1.0"),
+ ("tflite", ">=2.1.0"),
+ ("torch", "^1.7.0"),
+ ("torchvision", ">=0.5.0"),
+]
+
+################################################################################
+# End of configuration options.
+################################################################################
+
+
+
+
+# Required keys in REQUIREMENTS_BY_PIECE.
+REQUIRED_PIECES = ["core", "dev"]
+
+# Regex to validates piece names.
+PIECE_REGEX = re.compile(r"^[a-z0-9][a-z0-9-]*", re.IGNORECASE)
+
+# Regex to match a constraint specification. Multiple constraints are not supported.
+CONSTRAINT_REGEX = re.compile(r"(?:\^|\<|(?:<=)|(?:==)|(?:>=)|\>)[^<>=\^,]+")
+
+# Regex for parsing semantic versions. See
+# https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string
+SEMVER_REGEX = re.compile(r"^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$")
+
+
+def validate_requirements_by_piece():
+ problems = []
+
+ unseen_required_pieces = set(REQUIRED_PIECES)
+ seen_pieces = set()
+
+ # Ensure that core is listed first and dev is listed last.
+ saw_core = False
+ saw_dev = False
+
+ if not isinstance(REQUIREMENTS_BY_PIECE, (list, tuple)):
+ problems.append(f"must be list or tuple, see {REQUIREMENTS_BY_PIECE!r}")
+
+ for piece, deps in REQUIREMENTS_BY_PIECE:
+ if not isinstance(piece, str):
+ problems.append(f"key {piece!r}: must be str, see {piece!r}")
+ continue
+
+ if piece in unseen_required_pieces:
+ unseen_required_pieces.remove(piece)
+
+ piece_lower = piece.lower()
+ if piece_lower in seen_pieces:
+ problems.append(f"piece {piece}: listed twice")
+
+ seen_pieces.add(piece_lower)
+
+ if not saw_core and piece != "core":
+ problems.append(f'must list "core" before {piece}')
+ elif piece == "core":
+ saw_core = True
+
+ if saw_dev:
+ problems.append(f'must list "dev" last')
+ elif piece == "dev":
+ saw_dev = True
+
+ if not isinstance(deps, (list, tuple)):
+ problems.append(f'value for key "{piece}" should be list or tuple, see {deps!r}')
+ continue
+
+ if list(sorted(deps)) != list(deps):
+ problems.append(f"deps for key {piece} must be sorted. Correct order:\n {list(sorted(deps))!r}")
+
+ piece_deps = set()
+ for d in deps:
+ if CONSTRAINT_REGEX.search(d):
+ problems.append(f"for piece {piece}: dependency {d} should not specify a version. "
+ "Add it to CONSTRAINTS instead.")
+
+ if d.lower() in piece_deps:
+ problems.append(f'for piece {piece}: dependency {d} listed twice')
+
+ piece_deps.add(d.lower())
+
+ extras_pieces = [k for (k, _) in REQUIREMENTS_BY_PIECE if k not in ("dev", "core")]
+ sorted_extras_pieces = list(sorted(extras_pieces))
+ if sorted_extras_pieces != list(extras_pieces):
+ problems.append('pieces other than "core" and "dev" must appear in alphabetical order'
+ f"{sorted_extras_pieces}")
+
+ return problems
+
+
+def parse_semver(package, constraint, problems):
+ """Parse a semantic versioning constraint of the form "^X.[.Y[.Z[...]]]]"
+
+ Parameters
+ ----------
+ package : str
+ Name of the package specifying this constraint, for reporting problems.
+ constraint : str
+ The semver constraint. Must start with "^"
+ problems : List[str]
+ A list of strings describing problems that have occurred validating the configuration.
+ Problems encountered while validating constraint are appended to this list.
+
+ Returns
+ -------
+ List[str], int :
+ A 2-tuple. The first element is a list containing an entry for each component in the
+ semver string (components separated by "."). The second element is the index of the
+ component in the list which must not change to meet the semver constraint.
+
+ See "Caret requirements" at https://python-poetry.org/docs/versions/.
+ """
+ m = SEMVER_REGEX.match(constraint[1:])
+ if not m:
+ problems.append(f"{package}: invalid semver constraint {constraint}")
+ return [], 0
+
+ min_ver_parts = list(m.groups())
+
+ # Remove the end
+ while min_ver_parts and min_ver_parts[-1] is None or not min_ver_parts[-1].strip():
+ min_ver_parts.pop()
+
+ for i, p in enumerate(min_ver_parts):
+ try:
+ x = int(p.strip())
+ except ValueError as p:
+ if i == len(parts) - 1 and i > 0:
+ # If a pre-release tag is specified (i.e. 1.2.0pre3), allow.
+ return min_ver_parts, i - 1
+
+ problems.append(
+ f"{package}: semver constraint with non-numeric leading component: {constraint}")
+ return [], 0
+
+ if x:
+ return min_ver_parts, i
+
+ else:
+ # All 0's
+ return min_ver_parts, 0
+
+
+def validate_constraints():
+ problems = []
+
+ if not isinstance(CONSTRAINTS, (list, tuple)):
+ problems.append(f"must be list or tuple, see: {CONSTRAINTS!r}")
+
+ seen_packages = set()
+ all_deps = set()
+ for _, deps in REQUIREMENTS_BY_PIECE:
+ for d in deps:
+ all_deps.add(d.lower())
+
+ for package, constraint in CONSTRAINTS:
+ if package in seen_packages:
+ problems.append(f"{package}: specified twice")
+
+ if package.lower() not in all_deps:
+ problems.append(f"{package}: not specified in REQUIREMENTS_BY_PIECE")
+
+ if not CONSTRAINT_REGEX.match(constraint):
+ problems.append(f'{package}: constraint "{constraint}" does not look like a valid constraint')
+
+ if constraint.startswith("^"):
+ parse_semver(package, constraint, problems)
+
+ all_constrained_packages = [p for (p, _) in CONSTRAINTS]
+ sorted_constrained_packages = list(sorted(all_constrained_packages))
+ if sorted_constrained_packages != all_constrained_packages:
+ problems.append("CONSTRAINTS entries should be in this sorted order: "
+ f'{sorted_constrained_packages}')
+
+ return problems
+
+
+class ValidationError(Exception):
+ """Raised when a validation error occurs."""
+
+ @staticmethod
+ def format_problems(config, problems):
+ formatted = []
+ for p in problems:
+ assert isinstance(p, str), f"problems element not a str: {p}"
+ formatted.append('\n'.join(
+ textwrap.wrap(f"{config}: {p}", width=80, initial_indent=" * ", subsequent_indent=" ")))
+
+ return '\n'.join(formatted)
+
+ def __init__(self, config, problems):
+ super(ValidationError, self).__init__(self.format_problems(config, problems))
+ self.problems = problems
+
+
+def validate_or_raise():
+ problems = validate_requirements_by_piece()
+ if problems:
+ raise ValidationError("REQUIREMENTS_BY_PIECE", problems)
+
+ problems = validate_constraints()
+ if problems:
+ raise ValidationError("CONSTRAINTS", problems)
+
+
+def semver_to_requirements(piece, dep, constraint, joined_deps):
+ problems = []
+ min_ver_parts, fixed_index = parse_semver(dep, constraint, problems)
+ assert not problems
+
+ # i is now the index of the version component which must not change.
+ try:
+ fixed_part = int(min_ver_parts[fixed_index])
+ except ValueError:
+ problems.append(
+ f"piece {piece}: dependency {dep} has semver constraint with non-integer leading parts: {ver}")
+ return
+
+ max_ver_parts = (min_ver_parts[:fixed_index] +
+ [str(int(min_ver_parts[fixed_index]) + 1)] +
+ ["0" for _ in min_ver_parts[fixed_index + 1:]])
+ joined_deps.append(f'{dep}>={".".join(min_ver_parts)},<{".".join(max_ver_parts)}')
+
+
+def join_requirements():
+ """Validate, then join REQUIRMENTS_BY_PIECE against CONSTRAINTS and return the result.
+
+ Returns
+ -------
+ An OrderedDict containing REQUIREMENTS_BY_PIECE, except any dependency mentioned in CONSTRAINTS
+ is replaced by a setuptools-compatible constraint.
+ """
+ validate_or_raise()
+
+ constraints_map = collections.OrderedDict(
+ [(p.lower(), c) for (p, c) in CONSTRAINTS])
+
+ to_return = collections.OrderedDict()
+ all_deps = set()
+ for piece, deps in REQUIREMENTS_BY_PIECE:
+ joined_deps = []
+ for d in deps:
+ constraint = constraints_map.get(d.lower())
+ if constraint is None:
+ joined_deps.append(d)
+ continue
+
+ if constraint[0] == "^":
+ # Translate semver to requirements:
+ semver_to_requirements(piece, d, constraint, joined_deps)
+ else:
+ joined_deps.append(f"{d}{constraint}")
+
+ if piece != "dev":
+ all_deps.update(joined_deps)
+
+ to_return[piece] = joined_deps
+
+ to_return["all"] = all_deps
+
+ return to_return
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--lint", action="store_true",
+ help="Just lint dependencies, don't generate anything")
+ return parser.parse_args()
+
+
+def main():
+ args = parse_args()
+ try:
+ joined_deps = join_requirements()
+ except ValidationError as e:
+ print(f"ERROR: invalid requirements configuration in {__file__}:", file=sys.stderr)
+ print(str(e), file=sys.stderr)
+ sys.exit(2)
+
+ if args.lint:
+ sys.exit(0)
Review comment:
Recommend `return`, not `exit`, since this is a function that could be (and, in fact, is!) invoked by another Python library -- same for the error cases, raise an Exception, don't exit.
##########
File path: python/setup.py
##########
@@ -171,38 +171,25 @@ def get_package_data_files():
return ["relay/std/prelude.rly", "relay/std/core.rly"]
+# Temporarily add this directory to the path so we can import the requirements generator
+# tool.
+sys.path.insert(0, os.path.dirname(__file__))
+import gen_requirements
+sys.path.pop(0)
+
+requirements = gen_requirements.join_requirements()
Review comment:
Do you want to print out some details on what the requirements identified are, to help with debugging?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao1994 merged pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
junrushao1994 merged pull request #7289:
URL: https://github.com/apache/tvm/pull/7289
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r564731209
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
Review comment:
Yes. Previously there were some reported issues with the older XGboost versions, so the required XGBoost version bumps from 0.9.0 to 1.1.0.
Here are two examples I could think of at this moment:
https://github.com/apache/tvm/issues/4953
https://discuss.tvm.apache.org/t/segfault-in-auto-tuning-tutorial-tune-relay-x86-py/5928/9
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
Review comment:
Yes. Previously there were some reported issues with the older XGboost versions, so the required XGBoost version bumped from 0.9.0 to 1.1.0.
Here are two examples I could think of at this moment:
https://github.com/apache/tvm/issues/4953
https://discuss.tvm.apache.org/t/segfault-in-auto-tuning-tutorial-tune-relay-x86-py/5928/9
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r564710839
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
Review comment:
@comaniac can you point me to something more concrete that specifies the min required version?
@leandron added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r564718576
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
Review comment:
The upstream python/setup.py suggests `xgboost>=1.1.0`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r568052566
##########
File path: python/.gitignore
##########
@@ -1,3 +1,4 @@
build
dist
*.cpp
+/requirements/*.txt
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r567180785
##########
File path: python/.gitignore
##########
@@ -1,3 +1,4 @@
build
dist
*.cpp
+/requirements/*.txt
Review comment:
```suggestion
requirements/*.txt
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] mdw-octoml commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
mdw-octoml commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r558478062
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
Review comment:
Nit: You might consider storing these lists in a .yaml file and loading it, which is easier to edit/maintain than directly editing the Python code itself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r564707316
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
+
+ # Development requirements
+ ("dev", ["matplotlib", "pillow"]),
+]
+
+# Maps a named Python package (which should appear in REQUIREMENTS_BY_PIECE above) to a
+# semver or pip version constraint. Semver constraints are translated into requirements.txt
+# constraints.
+CONSTRAINTS = [
+ ("onnx", ">=1.7.0"),
+ ("onnxruntime", ">=1.0.0"),
+ ("pillow", "<7"),
+ ("synr", ">=0.2.1"),
+ ("tensorflow", ">=2.1.0"),
+ ("tflite", ">=2.1.0"),
+ ("torch", "^1.7.0"),
+ ("torchvision", ">=0.5.0"),
+]
+
+################################################################################
+# End of configuration options.
+################################################################################
+
+
+
+
+# Required keys in REQUIREMENTS_BY_PIECE.
+REQUIRED_PIECES = ["core", "dev"]
+
+# Regex to validates piece names.
+PIECE_REGEX = re.compile(r"^[a-z0-9][a-z0-9-]*", re.IGNORECASE)
+
+# Regex to match a constraint specification. Multiple constraints are not supported.
+CONSTRAINT_REGEX = re.compile(r"(?:\^|\<|(?:<=)|(?:==)|(?:>=)|\>)[^<>=\^,]+")
+
+# Regex for parsing semantic versions. See
+# https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string
+SEMVER_REGEX = re.compile(r"^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$")
+
+
+def validate_requirements_by_piece():
Review comment:
I will add type annotations before submitting this PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#issuecomment-767738967
@comaniac @leandron @tqchen @mdw-octoml @tkonolige please take a look at the [latest update](https://discuss.tvm.apache.org/t/rfc-consolidating-tvm-python-dependencies/8329/27) on the RFC forum thread.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r566353673
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
+
+ # Development requirements
+ ("dev", ["matplotlib", "pillow"]),
+]
+
+# Maps a named Python package (which should appear in REQUIREMENTS_BY_PIECE above) to a
+# semver or pip version constraint. Semver constraints are translated into requirements.txt
+# constraints.
+CONSTRAINTS = [
+ ("onnx", ">=1.7.0"),
+ ("onnxruntime", ">=1.0.0"),
+ ("pillow", "<7"),
+ ("synr", ">=0.2.1"),
+ ("tensorflow", ">=2.1.0"),
+ ("tflite", ">=2.1.0"),
+ ("torch", "^1.7.0"),
+ ("torchvision", ">=0.5.0"),
+]
+
+################################################################################
+# End of configuration options.
+################################################################################
+
+
+
+
+# Required keys in REQUIREMENTS_BY_PIECE.
+REQUIRED_PIECES = ["core", "dev"]
+
+# Regex to validates piece names.
+PIECE_REGEX = re.compile(r"^[a-z0-9][a-z0-9-]*", re.IGNORECASE)
+
+# Regex to match a constraint specification. Multiple constraints are not supported.
+CONSTRAINT_REGEX = re.compile(r"(?:\^|\<|(?:<=)|(?:==)|(?:>=)|\>)[^<>=\^,]+")
+
+# Regex for parsing semantic versions. See
+# https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string
+SEMVER_REGEX = re.compile(r"^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$")
+
+
+def validate_requirements_by_piece():
+ problems = []
+
+ unseen_required_pieces = set(REQUIRED_PIECES)
+ seen_pieces = set()
+
+ # Ensure that core is listed first and dev is listed last.
+ saw_core = False
+ saw_dev = False
+
+ if not isinstance(REQUIREMENTS_BY_PIECE, (list, tuple)):
+ problems.append(f"must be list or tuple, see {REQUIREMENTS_BY_PIECE!r}")
+
+ for piece, deps in REQUIREMENTS_BY_PIECE:
+ if not isinstance(piece, str):
+ problems.append(f"key {piece!r}: must be str, see {piece!r}")
+ continue
+
+ if piece in unseen_required_pieces:
+ unseen_required_pieces.remove(piece)
+
+ piece_lower = piece.lower()
+ if piece_lower in seen_pieces:
+ problems.append(f"piece {piece}: listed twice")
+
+ seen_pieces.add(piece_lower)
+
+ if not saw_core and piece != "core":
+ problems.append(f'must list "core" before {piece}')
+ elif piece == "core":
+ saw_core = True
+
+ if saw_dev:
+ problems.append(f'must list "dev" last')
+ elif piece == "dev":
+ saw_dev = True
+
+ if not isinstance(deps, (list, tuple)):
+ problems.append(f'value for key "{piece}" should be list or tuple, see {deps!r}')
+ continue
+
+ if list(sorted(deps)) != list(deps):
+ problems.append(f"deps for key {piece} must be sorted. Correct order:\n {list(sorted(deps))!r}")
+
+ piece_deps = set()
+ for d in deps:
+ if CONSTRAINT_REGEX.search(d):
+ problems.append(f"for piece {piece}: dependency {d} should not specify a version. "
+ "Add it to CONSTRAINTS instead.")
+
+ if d.lower() in piece_deps:
+ problems.append(f'for piece {piece}: dependency {d} listed twice')
+
+ piece_deps.add(d.lower())
+
+ extras_pieces = [k for (k, _) in REQUIREMENTS_BY_PIECE if k not in ("dev", "core")]
+ sorted_extras_pieces = list(sorted(extras_pieces))
+ if sorted_extras_pieces != list(extras_pieces):
+ problems.append('pieces other than "core" and "dev" must appear in alphabetical order'
+ f"{sorted_extras_pieces}")
+
+ return problems
+
+
+def parse_semver(package, constraint, problems):
+ """Parse a semantic versioning constraint of the form "^X.[.Y[.Z[...]]]]"
+
+ Parameters
+ ----------
+ package : str
+ Name of the package specifying this constraint, for reporting problems.
+ constraint : str
+ The semver constraint. Must start with "^"
+ problems : List[str]
+ A list of strings describing problems that have occurred validating the configuration.
+ Problems encountered while validating constraint are appended to this list.
+
+ Returns
+ -------
+ List[str], int :
+ A 2-tuple. The first element is a list containing an entry for each component in the
+ semver string (components separated by "."). The second element is the index of the
+ component in the list which must not change to meet the semver constraint.
+
+ See "Caret requirements" at https://python-poetry.org/docs/versions/.
+ """
+ m = SEMVER_REGEX.match(constraint[1:])
+ if not m:
+ problems.append(f"{package}: invalid semver constraint {constraint}")
+ return [], 0
+
+ min_ver_parts = list(m.groups())
+
+ # Remove the end
+ while min_ver_parts and min_ver_parts[-1] is None or not min_ver_parts[-1].strip():
+ min_ver_parts.pop()
+
+ for i, p in enumerate(min_ver_parts):
+ try:
+ x = int(p.strip())
+ except ValueError as p:
+ if i == len(parts) - 1 and i > 0:
+ # If a pre-release tag is specified (i.e. 1.2.0pre3), allow.
+ return min_ver_parts, i - 1
+
+ problems.append(
+ f"{package}: semver constraint with non-numeric leading component: {constraint}")
+ return [], 0
+
+ if x:
+ return min_ver_parts, i
+
+ else:
+ # All 0's
+ return min_ver_parts, 0
+
+
+def validate_constraints():
+ problems = []
+
+ if not isinstance(CONSTRAINTS, (list, tuple)):
+ problems.append(f"must be list or tuple, see: {CONSTRAINTS!r}")
+
+ seen_packages = set()
+ all_deps = set()
+ for _, deps in REQUIREMENTS_BY_PIECE:
+ for d in deps:
+ all_deps.add(d.lower())
+
+ for package, constraint in CONSTRAINTS:
+ if package in seen_packages:
+ problems.append(f"{package}: specified twice")
+
+ if package.lower() not in all_deps:
+ problems.append(f"{package}: not specified in REQUIREMENTS_BY_PIECE")
+
+ if not CONSTRAINT_REGEX.match(constraint):
+ problems.append(f'{package}: constraint "{constraint}" does not look like a valid constraint')
+
+ if constraint.startswith("^"):
+ parse_semver(package, constraint, problems)
+
+ all_constrained_packages = [p for (p, _) in CONSTRAINTS]
+ sorted_constrained_packages = list(sorted(all_constrained_packages))
+ if sorted_constrained_packages != all_constrained_packages:
+ problems.append("CONSTRAINTS entries should be in this sorted order: "
+ f'{sorted_constrained_packages}')
+
+ return problems
+
+
+class ValidationError(Exception):
+ """Raised when a validation error occurs."""
+
+ @staticmethod
+ def format_problems(config, problems):
+ formatted = []
+ for p in problems:
+ assert isinstance(p, str), f"problems element not a str: {p}"
+ formatted.append('\n'.join(
+ textwrap.wrap(f"{config}: {p}", width=80, initial_indent=" * ", subsequent_indent=" ")))
+
+ return '\n'.join(formatted)
+
+ def __init__(self, config, problems):
+ super(ValidationError, self).__init__(self.format_problems(config, problems))
+ self.problems = problems
+
+
+def validate_or_raise():
+ problems = validate_requirements_by_piece()
+ if problems:
+ raise ValidationError("REQUIREMENTS_BY_PIECE", problems)
+
+ problems = validate_constraints()
+ if problems:
+ raise ValidationError("CONSTRAINTS", problems)
+
+
+def semver_to_requirements(piece, dep, constraint, joined_deps):
+ problems = []
+ min_ver_parts, fixed_index = parse_semver(dep, constraint, problems)
+ assert not problems
+
+ # i is now the index of the version component which must not change.
+ try:
+ fixed_part = int(min_ver_parts[fixed_index])
+ except ValueError:
+ problems.append(
+ f"piece {piece}: dependency {dep} has semver constraint with non-integer leading parts: {ver}")
+ return
+
+ max_ver_parts = (min_ver_parts[:fixed_index] +
+ [str(int(min_ver_parts[fixed_index]) + 1)] +
+ ["0" for _ in min_ver_parts[fixed_index + 1:]])
+ joined_deps.append(f'{dep}>={".".join(min_ver_parts)},<{".".join(max_ver_parts)}')
+
+
+def join_requirements():
+ """Validate, then join REQUIRMENTS_BY_PIECE against CONSTRAINTS and return the result.
+
+ Returns
+ -------
+ An OrderedDict containing REQUIREMENTS_BY_PIECE, except any dependency mentioned in CONSTRAINTS
+ is replaced by a setuptools-compatible constraint.
+ """
+ validate_or_raise()
+
+ constraints_map = collections.OrderedDict(
+ [(p.lower(), c) for (p, c) in CONSTRAINTS])
+
+ to_return = collections.OrderedDict()
+ all_deps = set()
+ for piece, deps in REQUIREMENTS_BY_PIECE:
+ joined_deps = []
+ for d in deps:
+ constraint = constraints_map.get(d.lower())
+ if constraint is None:
+ joined_deps.append(d)
+ continue
+
+ if constraint[0] == "^":
+ # Translate semver to requirements:
+ semver_to_requirements(piece, d, constraint, joined_deps)
+ else:
+ joined_deps.append(f"{d}{constraint}")
+
+ if piece != "dev":
+ all_deps.update(joined_deps)
+
+ to_return[piece] = joined_deps
+
+ to_return["all"] = all_deps
+
+ return to_return
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--lint", action="store_true",
+ help="Just lint dependencies, don't generate anything")
+ return parser.parse_args()
+
+
+def main():
+ args = parse_args()
+ try:
+ joined_deps = join_requirements()
+ except ValidationError as e:
+ print(f"ERROR: invalid requirements configuration in {__file__}:", file=sys.stderr)
+ print(str(e), file=sys.stderr)
+ sys.exit(2)
+
+ if args.lint:
+ sys.exit(0)
Review comment:
actually in setup.py we call `join_requirements`, so I think we are good here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r564708746
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
Review comment:
added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao1994 merged pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
junrushao1994 merged pull request #7289:
URL: https://github.com/apache/tvm/pull/7289
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r568071857
##########
File path: tests/python/unittest/test_gen_requirements.py
##########
@@ -0,0 +1,220 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Tests for gen_requirements, found in python/."""
+
+import collections
+import contextlib
+import os
+import sys
+
+import tvm
+
+import pytest
+
+# Insert the parent dir to python/tvm into the import path, so that gen_requirements may be
+# imported.
+sys.path.insert(0, os.path.dirname(tvm.__file__))
+try:
+ import gen_requirements
+finally:
+ sys.path.pop(0)
+
+
+@contextlib.contextmanager
+def patch(obj, **kw):
+ old = {}
+ for prop_name, new in kw.items():
+ old[prop_name] = getattr(obj, prop_name)
+ setattr(obj, prop_name, new)
+ yield
+ for prop_name, value in old.items():
+ setattr(obj, prop_name, value)
+
+
+PROBLEM_REQUIREMENTS = [
Review comment:
I guess then though, you're doing processing to construct the test stimulus, which you could screw up. plus, each line here could create multiple problem strings or contribute to composite problem strings such as the final one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r564709209
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
+
+ # Development requirements
+ ("dev", ["matplotlib", "pillow"]),
+]
+
+# Maps a named Python package (which should appear in REQUIREMENTS_BY_PIECE above) to a
+# semver or pip version constraint. Semver constraints are translated into requirements.txt
+# constraints.
+CONSTRAINTS = [
+ ("onnx", ">=1.7.0"),
+ ("onnxruntime", ">=1.0.0"),
+ ("pillow", "<7"),
+ ("synr", ">=0.2.1"),
+ ("tensorflow", ">=2.1.0"),
+ ("tflite", ">=2.1.0"),
+ ("torch", "^1.7.0"),
+ ("torchvision", ">=0.5.0"),
+]
+
+################################################################################
+# End of configuration options.
+################################################################################
+
+
+
+
+# Required keys in REQUIREMENTS_BY_PIECE.
+REQUIRED_PIECES = ["core", "dev"]
+
+# Regex to validates piece names.
+PIECE_REGEX = re.compile(r"^[a-z0-9][a-z0-9-]*", re.IGNORECASE)
+
+# Regex to match a constraint specification. Multiple constraints are not supported.
+CONSTRAINT_REGEX = re.compile(r"(?:\^|\<|(?:<=)|(?:==)|(?:>=)|\>)[^<>=\^,]+")
+
+# Regex for parsing semantic versions. See
+# https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string
+SEMVER_REGEX = re.compile(r"^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$")
+
+
+def validate_requirements_by_piece():
+ problems = []
+
+ unseen_required_pieces = set(REQUIRED_PIECES)
+ seen_pieces = set()
+
+ # Ensure that core is listed first and dev is listed last.
+ saw_core = False
+ saw_dev = False
+
+ if not isinstance(REQUIREMENTS_BY_PIECE, (list, tuple)):
+ problems.append(f"must be list or tuple, see {REQUIREMENTS_BY_PIECE!r}")
+
+ for piece, deps in REQUIREMENTS_BY_PIECE:
+ if not isinstance(piece, str):
+ problems.append(f"key {piece!r}: must be str, see {piece!r}")
+ continue
+
+ if piece in unseen_required_pieces:
+ unseen_required_pieces.remove(piece)
+
+ piece_lower = piece.lower()
+ if piece_lower in seen_pieces:
+ problems.append(f"piece {piece}: listed twice")
+
+ seen_pieces.add(piece_lower)
+
+ if not saw_core and piece != "core":
+ problems.append(f'must list "core" before {piece}')
+ elif piece == "core":
+ saw_core = True
+
+ if saw_dev:
+ problems.append(f'must list "dev" last')
+ elif piece == "dev":
+ saw_dev = True
+
+ if not isinstance(deps, (list, tuple)):
+ problems.append(f'value for key "{piece}" should be list or tuple, see {deps!r}')
+ continue
+
+ if list(sorted(deps)) != list(deps):
+ problems.append(f"deps for key {piece} must be sorted. Correct order:\n {list(sorted(deps))!r}")
+
+ piece_deps = set()
+ for d in deps:
+ if CONSTRAINT_REGEX.search(d):
+ problems.append(f"for piece {piece}: dependency {d} should not specify a version. "
+ "Add it to CONSTRAINTS instead.")
+
+ if d.lower() in piece_deps:
+ problems.append(f'for piece {piece}: dependency {d} listed twice')
+
+ piece_deps.add(d.lower())
+
+ extras_pieces = [k for (k, _) in REQUIREMENTS_BY_PIECE if k not in ("dev", "core")]
+ sorted_extras_pieces = list(sorted(extras_pieces))
+ if sorted_extras_pieces != list(extras_pieces):
+ problems.append('pieces other than "core" and "dev" must appear in alphabetical order'
+ f"{sorted_extras_pieces}")
+
+ return problems
+
+
+def parse_semver(package, constraint, problems):
+ """Parse a semantic versioning constraint of the form "^X.[.Y[.Z[...]]]]"
+
+ Parameters
+ ----------
+ package : str
+ Name of the package specifying this constraint, for reporting problems.
+ constraint : str
+ The semver constraint. Must start with "^"
+ problems : List[str]
+ A list of strings describing problems that have occurred validating the configuration.
+ Problems encountered while validating constraint are appended to this list.
+
+ Returns
+ -------
+ List[str], int :
+ A 2-tuple. The first element is a list containing an entry for each component in the
+ semver string (components separated by "."). The second element is the index of the
+ component in the list which must not change to meet the semver constraint.
+
+ See "Caret requirements" at https://python-poetry.org/docs/versions/.
+ """
+ m = SEMVER_REGEX.match(constraint[1:])
+ if not m:
+ problems.append(f"{package}: invalid semver constraint {constraint}")
+ return [], 0
+
+ min_ver_parts = list(m.groups())
+
+ # Remove the end
+ while min_ver_parts and min_ver_parts[-1] is None or not min_ver_parts[-1].strip():
+ min_ver_parts.pop()
+
+ for i, p in enumerate(min_ver_parts):
+ try:
+ x = int(p.strip())
+ except ValueError as p:
+ if i == len(parts) - 1 and i > 0:
+ # If a pre-release tag is specified (i.e. 1.2.0pre3), allow.
+ return min_ver_parts, i - 1
+
+ problems.append(
+ f"{package}: semver constraint with non-numeric leading component: {constraint}")
+ return [], 0
+
+ if x:
+ return min_ver_parts, i
+
+ else:
+ # All 0's
+ return min_ver_parts, 0
+
+
+def validate_constraints():
+ problems = []
+
+ if not isinstance(CONSTRAINTS, (list, tuple)):
+ problems.append(f"must be list or tuple, see: {CONSTRAINTS!r}")
+
+ seen_packages = set()
+ all_deps = set()
+ for _, deps in REQUIREMENTS_BY_PIECE:
+ for d in deps:
+ all_deps.add(d.lower())
+
+ for package, constraint in CONSTRAINTS:
+ if package in seen_packages:
+ problems.append(f"{package}: specified twice")
+
+ if package.lower() not in all_deps:
+ problems.append(f"{package}: not specified in REQUIREMENTS_BY_PIECE")
+
+ if not CONSTRAINT_REGEX.match(constraint):
+ problems.append(f'{package}: constraint "{constraint}" does not look like a valid constraint')
+
+ if constraint.startswith("^"):
+ parse_semver(package, constraint, problems)
+
+ all_constrained_packages = [p for (p, _) in CONSTRAINTS]
+ sorted_constrained_packages = list(sorted(all_constrained_packages))
+ if sorted_constrained_packages != all_constrained_packages:
+ problems.append("CONSTRAINTS entries should be in this sorted order: "
+ f'{sorted_constrained_packages}')
+
+ return problems
+
+
+class ValidationError(Exception):
+ """Raised when a validation error occurs."""
+
+ @staticmethod
+ def format_problems(config, problems):
+ formatted = []
+ for p in problems:
+ assert isinstance(p, str), f"problems element not a str: {p}"
+ formatted.append('\n'.join(
+ textwrap.wrap(f"{config}: {p}", width=80, initial_indent=" * ", subsequent_indent=" ")))
+
+ return '\n'.join(formatted)
+
+ def __init__(self, config, problems):
+ super(ValidationError, self).__init__(self.format_problems(config, problems))
+ self.problems = problems
+
+
+def validate_or_raise():
+ problems = validate_requirements_by_piece()
+ if problems:
+ raise ValidationError("REQUIREMENTS_BY_PIECE", problems)
+
+ problems = validate_constraints()
+ if problems:
+ raise ValidationError("CONSTRAINTS", problems)
+
+
+def semver_to_requirements(piece, dep, constraint, joined_deps):
+ problems = []
+ min_ver_parts, fixed_index = parse_semver(dep, constraint, problems)
+ assert not problems
+
+ # i is now the index of the version component which must not change.
Review comment:
I updated this code quite a bit, now this comment is obsolete
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r558462379
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
Review comment:
Where did you specify the required XGBoost version? We have to use the required higher version to avoid some known issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#discussion_r558498794
##########
File path: python/gen_requirements.py
##########
@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""TVM Python requriements.txt generator.
+
+This script generates a set of requirements.txt files (stored in `./requirements`) that describe
+TVM's Python dependencies.
+
+## Pieces
+
+TVM can be roughly broken into these named pieces along the lines of Python dependencies:
+
+- "core": A core piece, which is intended to be buildable with very few external dependencies. Users
+ can use Relay, compile models, and run autotuning with this part.
+- "importer-<tool>": Model importers, which convert models defined in various other tools (i.e.
+ TensorFlow, PyTorch, etc) into Relay models.
+- Extra features (i.e. XGBoost in AutoTVM). These enhance TVM's functionality, but aren't required
+ for basic operation.
+
+## What this tool does
+
+From these pieces, this tool builds:
+ - requirements/<name>.txt - Python dependencies for each named piece above, `<name>` is the same as
+ the quoted piece name.
+ - requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.
+ - requirements/dev.txt - Python dependencies needed to develop TVM, such as lint and test tools.
+
+The data representing each piece is contained in the two maps below.
+"""
+
+import argparse
+import collections
+import os
+import re
+import textwrap
+import sys
+
+# Maps named TVM piece (see description above) to a list of names of Python packages. Please use
+# alphabetical order for each package list, and do not add version constraints here!
+REQUIREMENTS_BY_PIECE = [
+ # Base requirements needed to install tvm with no extras.
+ ("core", [
+ "attrs",
+ "decorator",
+ "numpy",
+ "psutil",
+ "scipy",
+ "synr",
+ ]),
+
+ # Relay frontends.
+ ("importer-caffe2", ["torch"]),
+ ("importer-coreml", ["coremltools"]),
+ ("importer-darknet", ["opencv-python"]),
+ ("importer-keras", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-onnx", ["future", "onnx", "onnxruntime", "torch", "torchvision"]),
+ ("importer-pytorch", ["future", "torch", "torchvision"]),
+ ("importer-tensorflow", ["tensorflow", "tensorflow-estimator"]),
+ ("importer-tflite", ["tensorflow", "tensorflow-estimator", "tflite"]),
+
+ ("tvmc", ["onnx", "onnxruntime", "tensorflow", "tflite", "torch", "torchvision"]),
+
+ # XGBoost, useful for autotuning on some targets.
+ ("xgboost", ["torch"]),
+
+ # Development requirements
+ ("dev", ["matplotlib", "pillow"]),
+]
+
+# Maps a named Python package (which should appear in REQUIREMENTS_BY_PIECE above) to a
+# semver or pip version constraint. Semver constraints are translated into requirements.txt
+# constraints.
+CONSTRAINTS = [
+ ("onnx", ">=1.7.0"),
+ ("onnxruntime", ">=1.0.0"),
+ ("pillow", "<7"),
+ ("synr", ">=0.2.1"),
+ ("tensorflow", ">=2.1.0"),
+ ("tflite", ">=2.1.0"),
+ ("torch", "^1.7.0"),
+ ("torchvision", ">=0.5.0"),
+]
+
+################################################################################
+# End of configuration options.
+################################################################################
+
+
+
+
+# Required keys in REQUIREMENTS_BY_PIECE.
+REQUIRED_PIECES = ["core", "dev"]
+
+# Regex to validates piece names.
+PIECE_REGEX = re.compile(r"^[a-z0-9][a-z0-9-]*", re.IGNORECASE)
+
+# Regex to match a constraint specification. Multiple constraints are not supported.
+CONSTRAINT_REGEX = re.compile(r"(?:\^|\<|(?:<=)|(?:==)|(?:>=)|\>)[^<>=\^,]+")
Review comment:
yeah no non-stdlib deps in the deps generator script
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao1994 commented on pull request #7289: Generate requirements.txt from Python spec
Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on pull request #7289:
URL: https://github.com/apache/tvm/pull/7289#issuecomment-772193937
Thanks everyone! I am going to merge it in tomorrow if there are no further comments :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org