You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/07/01 15:32:24 UTC

[GitHub] [tvm] mikepapadim opened a new pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

mikepapadim opened a new pull request #8382:
URL: https://github.com/apache/tvm/pull/8382


   This PR adds a pre-commit hook configuration file.
   This configuration aims to encourage running the sanity checks on Jenkins, locally prior to committing. 
   
   How to install:
   ```pip install pre-commit```
   ```pre-commit install```
   
   
   This pre-commit configuration will run the following checks on each `git commit`:
   ```
   Check for added large files..........................(no files to check)Skipped
   Check for merge conflicts............................(no files to check)Skipped
   Check Yaml...........................................(no files to check)Skipped
   Fix End of Files.....................................(no files to check)Skipped
   Trim Trailing Whitespace.............................(no files to check)Skipped
   Running Minimal Unit-Tests ..............................................Passed
   Running Black............................................................Passed
   Checking File Types......................................................Passed
   Checking ASF License Headers ............................................Passed
   Linting the C++ code ................................(no files to check)Skipped
   Checking Clang format ...............................(no files to check)Skipped
   Type Checking with MyPY .................................................Passed
   ```
   
   Also, as WIP branches tend to fail tests on early stages of development the above-mentioned checks can be prevented with:
   ``` git commit --no-verify ``` or ```git commit -n ```
   
   **Requirements**: pip3 install cpplint pylint==2.4.4 mypy==0.902 black==20.8b1 pre-commit==2.3.0
   
   @hogepodge @areusch @jroesch @jcf94 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mikepapadim commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
mikepapadim commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-888243890


   I can change it to match `docker/lint.sh`. Also, regarding developer docs, is there a specific doc that this should be appropriate to be placed?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] tqchen commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-896098359


   @mikepapadim Let us consider skip the minimum tests by default for now since they are usually not part of the lint


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] junrushao1994 commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-891396226


   Hey any updates on this PR? It is super helpful functionality for developers, so is it good to merge now, or do we need some more extra docs for it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mikepapadim commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
mikepapadim commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-904952972


   This good to go @areusch @jroesch @mbs-octoml @tqchen 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] areusch commented on a change in pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#discussion_r663231702



##########
File path: .pre-commit-config.yaml
##########
@@ -0,0 +1,40 @@
+repos:
+    - repo: https://github.com/pre-commit/pre-commit-hooks
+      rev: v2.5.0
+      hooks:
+        - id: check-added-large-files
+        - id: check-merge-conflict
+        - id: check-yaml
+        - id: end-of-file-fixer
+    - repo: local
+      hooks:
+        -   id: run-file-types
+            name: Checking File Types...
+            entry: python3 tests/lint/check_file_type.py
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-headers-check
+            name: Checking ASF License Headers...
+            entry: tests/lint/check_asf_header.sh --local
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-black

Review comment:
       @mshawcroft so most python package dependencies are fairly loose in order to allow you to create mega-venv. imagine if a middleware required a precise version of a common library e.g. `attrs`. it would be very difficult to combine such middleware with other larger packages in the same venv while respecting the constraints of `install_requires`. therefore, many packages don't specify exact `install_requires`. you can [see](https://github.com/PyCQA/pylint/commit/fed98574f909bb34cff32b63db4e93d5271cee32#diff-fa602a8a75dc9dcc92261bac5f533c2a85e34fcceaff63b3a3a81d9acde2fc52) that when pylint updates its `astroid` dep, it merely pushes along the `>=` constraint, but now that astroid 2.6.2 is [released](https://pypi.org/project/astroid/#history), that old constraint is ambiguous.
   
   this lax specification has made python easy to use, but as a result, API mismatches pervade the ecosystem. In general, duck typing has minimized the impact of these--up to the point that you want to reproduce exact results machine-to-machine. at this point, you need a lockfile and potentially the new pip dep resolver (note that the `install_requires` constraints were [not even enforced](https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020) until last year). this is why lint is particularly sensitive to the exact set of packages installed in the venv, and why i don't currently recommend to use anything checked-in other than the docker container to repro ci lint results right now.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mikepapadim commented on a change in pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
mikepapadim commented on a change in pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#discussion_r662839717



##########
File path: .pre-commit-config.yaml
##########
@@ -0,0 +1,40 @@
+repos:
+    - repo: https://github.com/pre-commit/pre-commit-hooks
+      rev: v2.5.0
+      hooks:
+        - id: check-added-large-files
+        - id: check-merge-conflict
+        - id: check-yaml
+        - id: end-of-file-fixer
+    - repo: local
+      hooks:
+        -   id: run-file-types
+            name: Checking File Types...
+            entry: python3 tests/lint/check_file_type.py
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-headers-check
+            name: Checking ASF License Headers...
+            entry: tests/lint/check_asf_header.sh --local
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-black

Review comment:
       Indeed, this can work without getting involved with the docker images. 
   
   In my case, I got the from `pip freeze` the versions that docker uses and install them locally to have the same environment. 
   
   I agree, I am going to update the config file with pre-canned versions matching the lint environment(https://github.com/apache/tvm/blob/ab01abc22460c6d8dcd6ed75e77b7224364014d8/docker/Dockerfile.ci_lint#L32). 
   
   I think the goal of having such hook available is to reduce the fails in 1) Sanity checks stages and 2) the builds MacOs and Windows builds that are running the minimal unit tests.   
   
   @areusch I think the above configuration might allow to prevent the need of the docker.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] tqchen merged pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
tqchen merged pull request #8382:
URL: https://github.com/apache/tvm/pull/8382


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] areusch commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-895341780


   @mikepapadim @tqchen @junrushao1994 i'd still like us to use the `ci-lint` image to invoke each of these checks. doing this will ensure that the output of the pre-commit hook exactly matches the CI (or has the best chance of doing so). the best way to do this is with:
   `docker/lint.sh <check_name>`
   
   there is one wrinkle with this approach--it will fail spectacularly when you are inside a `git-worktree` other than your original git repo. @Lunderberg is resolving this limitation with #8670.
   
   do you mind changing the command line to `docker/lint.sh` for the pre-commit steps? then i'm good with merging this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mikepapadim commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
mikepapadim commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-895247542


   IMO it is good to go. The main aim is to force some minimal checks to run locally and practically being able to spot the two GH actions (windows and MacOS) fails locally as well as the sanity check stage. 
   
   Maybe we could add an accompanying md doc. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mikepapadim commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
mikepapadim commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-895933667


   @areusch Now it runs all linting through the docker image except the `all-platform-minimal-test`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mikepapadim commented on a change in pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
mikepapadim commented on a change in pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#discussion_r662839717



##########
File path: .pre-commit-config.yaml
##########
@@ -0,0 +1,40 @@
+repos:
+    - repo: https://github.com/pre-commit/pre-commit-hooks
+      rev: v2.5.0
+      hooks:
+        - id: check-added-large-files
+        - id: check-merge-conflict
+        - id: check-yaml
+        - id: end-of-file-fixer
+    - repo: local
+      hooks:
+        -   id: run-file-types
+            name: Checking File Types...
+            entry: python3 tests/lint/check_file_type.py
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-headers-check
+            name: Checking ASF License Headers...
+            entry: tests/lint/check_asf_header.sh --local
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-black

Review comment:
       Indeed, this can work without getting involved with the docker images. 
   
   In my case, I got the from `pip freeze` the versions that docker uses and install them locally to have the same environment. 
   
   I agree, I am going to update the config file with pre-canned versions matching the lint environment(https://github.com/apache/tvm/blob/ab01abc22460c6d8dcd6ed75e77b7224364014d8/docker/Dockerfile.ci_lint#L32). 
   
   I think the goal of having such hook available is to reduce the fails in 1) Sanity checks stages and 2) the builds MacOs and Windows builds that are running the minimal unit tests.   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mshawcroft commented on a change in pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
mshawcroft commented on a change in pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#discussion_r663134095



##########
File path: .pre-commit-config.yaml
##########
@@ -0,0 +1,40 @@
+repos:
+    - repo: https://github.com/pre-commit/pre-commit-hooks
+      rev: v2.5.0
+      hooks:
+        - id: check-added-large-files
+        - id: check-merge-conflict
+        - id: check-yaml
+        - id: end-of-file-fixer
+    - repo: local
+      hooks:
+        -   id: run-file-types
+            name: Checking File Types...
+            entry: python3 tests/lint/check_file_type.py
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-headers-check
+            name: Checking ASF License Headers...
+            entry: tests/lint/check_asf_header.sh --local
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-black

Review comment:
       My proposal above would leave the linter tools versioned via precommit only, the CI docker environment would call pre-commit, but would not need to install or version tools directly.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] tqchen commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-905781697


   Thanks @mikepapadim ! This is now merged


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mshawcroft commented on a change in pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
mshawcroft commented on a change in pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#discussion_r674271047



##########
File path: .pre-commit-config.yaml
##########
@@ -0,0 +1,40 @@
+repos:
+    - repo: https://github.com/pre-commit/pre-commit-hooks
+      rev: v2.5.0
+      hooks:
+        - id: check-added-large-files
+        - id: check-merge-conflict
+        - id: check-yaml
+        - id: end-of-file-fixer
+    - repo: local
+      hooks:
+        -   id: run-file-types
+            name: Checking File Types...
+            entry: python3 tests/lint/check_file_type.py
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-headers-check
+            name: Checking ASF License Headers...
+            entry: tests/lint/check_asf_header.sh --local
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-black

Review comment:
       @areusch can you share a link to a bug on the pylint project raised as a result of inadequate install_requires constraints?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mikepapadim commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
mikepapadim commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-884314208


   > @mikepapadim how do you want to proceed here?
   
   If we need this to be upstreamed, we can reduce the number of checks with only the ones having pre-canned versions matching the CI to avoid any extra overhead. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mikepapadim commented on a change in pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
mikepapadim commented on a change in pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#discussion_r662839717



##########
File path: .pre-commit-config.yaml
##########
@@ -0,0 +1,40 @@
+repos:
+    - repo: https://github.com/pre-commit/pre-commit-hooks
+      rev: v2.5.0
+      hooks:
+        - id: check-added-large-files
+        - id: check-merge-conflict
+        - id: check-yaml
+        - id: end-of-file-fixer
+    - repo: local
+      hooks:
+        -   id: run-file-types
+            name: Checking File Types...
+            entry: python3 tests/lint/check_file_type.py
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-headers-check
+            name: Checking ASF License Headers...
+            entry: tests/lint/check_asf_header.sh --local
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-black

Review comment:
       Indeed, this can work without getting involved with the docker images. 
   
   In my case, I got the from `pip freeze` the versions that docker uses and install them locally to have the same environment. 
   
   I agree, I am going to update the config file with pre-canned versions matching the lint environment. 
   
   I think the goal of having such hook available is to reduce the fails in 1) Sanity checks stages and 2) the builds MacOs and Windows builds that are running the minimal unit tests.   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] areusch commented on a change in pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#discussion_r663160299



##########
File path: .pre-commit-config.yaml
##########
@@ -0,0 +1,40 @@
+repos:
+    - repo: https://github.com/pre-commit/pre-commit-hooks
+      rev: v2.5.0
+      hooks:
+        - id: check-added-large-files
+        - id: check-merge-conflict
+        - id: check-yaml
+        - id: end-of-file-fixer
+    - repo: local
+      hooks:
+        -   id: run-file-types
+            name: Checking File Types...
+            entry: python3 tests/lint/check_file_type.py
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-headers-check
+            name: Checking ASF License Headers...
+            entry: tests/lint/check_asf_header.sh --local
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-black

Review comment:
       @mshawcroft after some reading, it appears that `pre-commit` will create a venv and effectively run `pip install <tool>=<version>`. i think this won't quite achieve the same thing as the docker container without specifying `additional_dependencies`--each of these named Python packages has dependencies that need to exactly match those in `ci-lint`. you could do that by cross-referencing `pip3 freeze` from ci-lint. [this blog](https://objectpartners.com/2020/06/15/consistent-python-environments-with-poetry-and-pre-commit-hooks/) discusses integrating `pre-commit` with a local venv, which goes a bit further towards this (though we are still blocked on `requirements.txt` due to CI container rebuild pipeline).
   
   i'm okay with committing something with limitations, just noting what they are, and i would prefer we document those limitations and/or a recommended strategy to work around those somewhere. i do think that as we address the `requirements.txt` shortcoming, we can most likely resolve this issue outside docker (python version may impact this as well).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] tqchen edited a comment on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-894634558


   If we feel it is in a good state. We can merge then send a followup PR to mention the tool in the Git tips part of our developer guide. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] areusch commented on a change in pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#discussion_r677036052



##########
File path: .pre-commit-config.yaml
##########
@@ -0,0 +1,40 @@
+repos:
+    - repo: https://github.com/pre-commit/pre-commit-hooks
+      rev: v2.5.0
+      hooks:
+        - id: check-added-large-files
+        - id: check-merge-conflict
+        - id: check-yaml
+        - id: end-of-file-fixer
+    - repo: local
+      hooks:
+        -   id: run-file-types
+            name: Checking File Types...
+            entry: python3 tests/lint/check_file_type.py
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-headers-check
+            name: Checking ASF License Headers...
+            entry: tests/lint/check_asf_header.sh --local
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-black

Review comment:
       i don't have one offhand, but if you look at the default pylint bug template, they want you to run `pylint --version` and that outputs the exact versions of python, astroid, and pylint used to trigger the bug. if we are considering the case where `pylint` is the lone thing installed into a venv, the bug would have to essentially be a released regression in `astroid`. if we're considering a case where other things dependent on e.g. `astroid` were installed (especially after `pylint` was installed), then i think it'd be easier to contrive a case (i _think _ if another package locked `astroid` to an arbitrary version, pip < 20.3 could downgrade it past `pylint`'s `install_requires`).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mshawcroft commented on a change in pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
mshawcroft commented on a change in pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#discussion_r663219991



##########
File path: .pre-commit-config.yaml
##########
@@ -0,0 +1,40 @@
+repos:
+    - repo: https://github.com/pre-commit/pre-commit-hooks
+      rev: v2.5.0
+      hooks:
+        - id: check-added-large-files
+        - id: check-merge-conflict
+        - id: check-yaml
+        - id: end-of-file-fixer
+    - repo: local
+      hooks:
+        -   id: run-file-types
+            name: Checking File Types...
+            entry: python3 tests/lint/check_file_type.py
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-headers-check
+            name: Checking ASF License Headers...
+            entry: tests/lint/check_asf_header.sh --local
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-black

Review comment:
       @areusch there should be no need to provide additional constraints on their individual dependencies, if there is, that suggests an upstream bug w.r.t install_requires, are you aware of a specific issue or just concerned that there might be an issue?
   
   I'd expect that having pre-commit specify the top level version for each lint tool is sufficient that is the normal way to use pre-commit.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] areusch commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-872352229


   thanks @mikepapadim this looks like a great addition! one question on the approach: typically in order to get accurate e.g. pylint results, you need to ensure you have exactly the version of pylint, astroid, etc. as are installed in the CI. we don't currently have a programmatic way to do that outside of the `ci-lint` docker container. 
   
   i made a script `docker/lint.sh` which invokes these commands in `ci-lint`--the problem with this one is that due to a limitation of the docker container mounts (see #6344), this doesn't work with git-subtree. it would be great if we could somehow introduce the docker container to this approach to make it foolproof. otherwise, it might weaken the pre-commit output if it's inaccurate.
   
   what are your thoughts? we don't have a good way to capture the python deps used in CI right now (there are some efforts underway, but it'll be a while); however, you could probably get what you need from `docker/bash.sh ci_lint pip3 freeze` if we wanted to go that approach. another approach is to push on #6344 again and try to fix the rust build issues.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] tqchen commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-894634558


   We can merge then send a followup PR to mention the tool in the Git tips part of our developer guide. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbs-octoml commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-904099250


   Hi Michalis, is this one ready to go in? I think if you push to get past the Jenkins hickup it should proceed through ci as usual.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] areusch commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-883738910


   @mikepapadim how do you want to proceed here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mshawcroft commented on a change in pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
mshawcroft commented on a change in pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#discussion_r662812459



##########
File path: .pre-commit-config.yaml
##########
@@ -0,0 +1,40 @@
+repos:
+    - repo: https://github.com/pre-commit/pre-commit-hooks
+      rev: v2.5.0
+      hooks:
+        - id: check-added-large-files
+        - id: check-merge-conflict
+        - id: check-yaml
+        - id: end-of-file-fixer
+    - repo: local
+      hooks:
+        -   id: run-file-types
+            name: Checking File Types...
+            entry: python3 tests/lint/check_file_type.py
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-headers-check
+            name: Checking ASF License Headers...
+            entry: tests/lint/check_asf_header.sh --local
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-black

Review comment:
       black, pylint, mypy, possibly others here that are in 'local' can be run directly from pre-commit using pre-canned configurations.  e.g.
   - repo: https://github.com/psf/black
     rev: 21.5b1
     hooks:
       - id: black
   
   Perhaps another approach here is to add those three here in precommit and replace the relevant set of tests/lint/*.sh scripts and supporting machinery and tool installs in the docker with a single tests/lint/pre-commit.sh such that pre-commit manages the versioning / installation and execution of those tools and the standard tvm CI build test environment just leverages pre-commit.   Doing this moves all the configuration of those tools to one place, it removes the need to have CI docker infrastructure manage linter versions, and means that for all the tooling fully managed by pre-commit, pre-commit can be used by devs directly on the command line or in their .git/hooks without needing to either spin up the docker environment.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mikepapadim commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
mikepapadim commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-904509972


   > Hi Michalis, is this one ready to go in? I think if you push to get past the Jenkins hickup it should proceed through ci as usual.
   
   I am pushing to CI to check


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] areusch commented on a change in pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#discussion_r663125094



##########
File path: .pre-commit-config.yaml
##########
@@ -0,0 +1,40 @@
+repos:
+    - repo: https://github.com/pre-commit/pre-commit-hooks
+      rev: v2.5.0
+      hooks:
+        - id: check-added-large-files
+        - id: check-merge-conflict
+        - id: check-yaml
+        - id: end-of-file-fixer
+    - repo: local
+      hooks:
+        -   id: run-file-types
+            name: Checking File Types...
+            entry: python3 tests/lint/check_file_type.py
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-headers-check
+            name: Checking ASF License Headers...
+            entry: tests/lint/check_asf_header.sh --local
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-black

Review comment:
       i see, but then we'll have two places to change when we list versions, no? also, while the repo pinning will address the version of that particular tool, i don't believe it does anything to pin the dependencies of those tools. in particular, different versions of `astroid` can produce wildly different results when combined with pylint.

##########
File path: .pre-commit-config.yaml
##########
@@ -0,0 +1,40 @@
+repos:
+    - repo: https://github.com/pre-commit/pre-commit-hooks
+      rev: v2.5.0
+      hooks:
+        - id: check-added-large-files
+        - id: check-merge-conflict
+        - id: check-yaml
+        - id: end-of-file-fixer
+    - repo: local
+      hooks:
+        -   id: run-file-types
+            name: Checking File Types...
+            entry: python3 tests/lint/check_file_type.py
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-headers-check
+            name: Checking ASF License Headers...
+            entry: tests/lint/check_asf_header.sh --local
+            language: system
+            always_run: true
+            pass_filenames: false
+        -   id: run-black

Review comment:
       i see, but then we'll have two places to change where we list versions, no? also, while the repo pinning will address the version of that particular tool, i don't believe it does anything to pin the dependencies of those tools. in particular, different versions of `astroid` can produce wildly different results when combined with pylint.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] areusch commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-887125326


   @mikepapadim i think you could get that for all checks if you used `docker/lint.sh` to invoke the tools.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] hogepodge commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
hogepodge commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-887988579


   Regardless, if this is merged it needs to be called out in the developer docs to encourage adoption.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] hogepodge commented on pull request #8382: [Pre-commit] Add pre-commit configuration to perform minimal checks locally

Posted by GitBox <gi...@apache.org>.
hogepodge commented on pull request #8382:
URL: https://github.com/apache/tvm/pull/8382#issuecomment-895676130


   Just to chime in again, bringing a development environment closer to the CI environment is going to help with development velocity, and hopefully reduce time in gate by giving developers an opportunity to test more locally. I'm good for merging once Andrew's concerns are addressed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org