You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/05/12 06:29:34 UTC
[GitHub] [druid] clintropolis opened a new pull request #11238: chill, travis
clintropolis opened a new pull request #11238:
URL: https://github.com/apache/druid/pull/11238
Initial attempt at making travis be a bit more selective about what it runs, since every change takes like.. over a day of compute time.
This PR adds a new script, `check-test-suite.py`, which is used in `.travis.yml` jobs to check whether or not a given job should run. It does this based on an admittedly primitive git diff file path prefix matching against stuff hard coded into the script, initially dividing into 3 categories: 'docs', 'web-console' and like, everything else. We do some similar stuff for code coverage diff checking enforcement for java projects, so a follow-up improvement could provide finer granularity for the 'everything else' bucket.
I'm sure there is a better way to configure this than hard coding stuff into the script as well, but, its a start and should let us at least cut down a tiny amount of the global warming we are surely causing with all of this CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on pull request #11238: chill, travis
Posted by GitBox <gi...@apache.org>.
clintropolis commented on pull request #11238:
URL: https://github.com/apache/druid/pull/11238#issuecomment-840287384
>but I am unable to retrigger any jobs for some reason (amusingly it returns 403 status code), so I'll try to push some more commits to these branches in a bit to try to get things to run through.
oh heh, I wasn't paying close attention, i guess i've used up too many of my "oss credits" on my fork while I was testing stuff before I opened this PR which is I think why it won't let me retry anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vogievetsky commented on pull request #11238: chill, travis
Posted by GitBox <gi...@apache.org>.
vogievetsky commented on pull request #11238:
URL: https://github.com/apache/druid/pull/11238#issuecomment-840149237
I very much approve the spirit of this PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on pull request #11238: chill, travis
Posted by GitBox <gi...@apache.org>.
clintropolis commented on pull request #11238:
URL: https://github.com/apache/druid/pull/11238#issuecomment-840176791
I've opened 3 example PRs to this PR on my fork of Druid to help confirm that this is working as intended:
web-console: https://github.com/clintropolis/druid/pull/2
docs: https://github.com/clintropolis/druid/pull/3
java: https://github.com/clintropolis/druid/pull/4
(they are currently running so I will update this with the outcome once they have finished)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis edited a comment on pull request #11238: chill, travis
Posted by GitBox <gi...@apache.org>.
clintropolis edited a comment on pull request #11238:
URL: https://github.com/apache/druid/pull/11238#issuecomment-840287384
>but I am unable to retrigger any jobs for some reason (amusingly it returns 403 status code), so I'll try to push some more commits to these branches in a bit to try to get things to run through.
oh heh, I wasn't paying close attention, i guess i've used any travis credits i had on my fork while I was testing stuff before I opened this PR which is I think why it won't let me retry anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #11238: chill, travis
Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #11238:
URL: https://github.com/apache/druid/pull/11238#discussion_r631894770
##########
File path: check-test-suite.py
##########
@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import subprocess
+import sys
+
+# do some primitive examination of git diff to determine if a test suite needs to be run or not
+
+always_run_jobs = ['license checks', '(openjdk8) packaging check', '(openjdk11) packaging check']
+
+# ignore changes to these files completely since they don't impact CI, if the changes are only to these files then all
+# of CI can be skipped
+ignore_prefixes = ['.github', '.idea', '.asf.yaml', '.backportrc.json', '.codecov.yml', '.dockerignore', '.gitignore',
+ '.lgtm.yml', 'CONTRIBUTING.md', 'setup-hooks.sh', 'upload.sh', 'dev', 'distribution/docker',
+ 'distribution/asf-release-process-guide.md', '.travis.yml', 'check-test-suite.py']
+# these files are docs changes
+# if changes are limited to this set then we can skip web-console and java
+# if no changes in this set we can skip docs
+docs_prefixes = ['docs/', 'website/']
+# travis docs job name
+docs_jobs = ['docs']
+# these files are web-console changes
+# if changes are limited to this set then we can skip docs and java
+# if no changes in this set we can skip web-console
+web_console_prefixes = ['web-console/']
+# travis web-console job name
+web_console_jobs = ['web console', 'web console end-to-end test']
+
+
+def check_ignore(file):
+ is_always_ignore = True in (file.startswith(prefix) for prefix in ignore_prefixes)
+ if is_always_ignore:
+ print("found ignorable file change: {}".format(file))
+ return is_always_ignore
+
+def check_docs(file):
+ is_docs = True in (file.startswith(prefix) for prefix in docs_prefixes)
+ if is_docs:
+ print("found docs file change: {}".format(file))
+ return is_docs
+
+def check_console(file):
+ is_console = True in (file.startswith(prefix) for prefix in web_console_prefixes)
+ if is_console:
+ print("found web-console file change: {}".format(file))
+ return is_console
+
+def check_should_run_suite(suite, diff_files):
+ if suite in always_run_jobs:
+ # you gotta do what you gotta do
+ return True
+
+ all_ignore = True
+ any_docs = False
+ all_docs = True
+ any_console = False
+ all_console = True
+
+ for f in diff_files:
+ all_ignore = all_ignore and check_ignore(f)
+ is_docs = check_docs(f)
+ any_docs = any_docs or is_docs
+ all_docs = all_docs and is_docs
+ is_console = check_console(f)
+ any_console = any_console or is_console
+ all_console = any_console and is_console
+
+ if all_ignore:
+ return False
+ if suite in docs_jobs:
+ return any_docs
+ if all_docs:
+ return False
+ if suite in web_console_jobs:
+ return any_console
+ if all_console:
+ return False
+
+ return True
+
+def failWithUsage():
+ sys.stderr.write("usage: check-test-suite.py <test-suite-name>\n")
+ sys.stderr.write(" e.g., check-test-suite.py docs")
+ sys.exit(1)
+
+suite_name = ""
+
+if len(sys.argv) == 1:
+ if 'TRAVIS_JOB_NAME' in os.environ:
+ suite_name = os.environ['TRAVIS_JOB_NAME']
+ else:
+ failWithUsage()
+elif len(sys.argv) == 2:
+ suite_name = sys.argv[1]
+else:
+ failWithUsage()
+
+
+all_changed_files_string = subprocess.check_output("git diff --name-only HEAD~1", shell=True).decode('UTF-8')
Review comment:
What happens if there's a PR with multiple commits? I think we need a different git command to find all the commits that are new compared to the base branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #11238: chill, travis
Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #11238:
URL: https://github.com/apache/druid/pull/11238#discussion_r631896897
##########
File path: check-test-suite.py
##########
@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import subprocess
+import sys
+
+# do some primitive examination of git diff to determine if a test suite needs to be run or not
+
+always_run_jobs = ['license checks', '(openjdk8) packaging check', '(openjdk11) packaging check']
+
+# ignore changes to these files completely since they don't impact CI, if the changes are only to these files then all
+# of CI can be skipped
+ignore_prefixes = ['.github', '.idea', '.asf.yaml', '.backportrc.json', '.codecov.yml', '.dockerignore', '.gitignore',
+ '.lgtm.yml', 'CONTRIBUTING.md', 'setup-hooks.sh', 'upload.sh', 'dev', 'distribution/docker',
+ 'distribution/asf-release-process-guide.md', '.travis.yml', 'check-test-suite.py']
+# these files are docs changes
+# if changes are limited to this set then we can skip web-console and java
+# if no changes in this set we can skip docs
+docs_prefixes = ['docs/', 'website/']
+# travis docs job name
+docs_jobs = ['docs']
+# these files are web-console changes
+# if changes are limited to this set then we can skip docs and java
+# if no changes in this set we can skip web-console
+web_console_prefixes = ['web-console/']
+# travis web-console job name
+web_console_jobs = ['web console', 'web console end-to-end test']
+
+
+def check_ignore(file):
+ is_always_ignore = True in (file.startswith(prefix) for prefix in ignore_prefixes)
+ if is_always_ignore:
+ print("found ignorable file change: {}".format(file))
+ return is_always_ignore
+
+def check_docs(file):
+ is_docs = True in (file.startswith(prefix) for prefix in docs_prefixes)
+ if is_docs:
+ print("found docs file change: {}".format(file))
+ return is_docs
+
+def check_console(file):
+ is_console = True in (file.startswith(prefix) for prefix in web_console_prefixes)
+ if is_console:
+ print("found web-console file change: {}".format(file))
+ return is_console
+
+def check_should_run_suite(suite, diff_files):
Review comment:
the logic in this function seems quite involved, could you add a comment explaining what it's aiming to do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on pull request #11238: chill, travis
Posted by GitBox <gi...@apache.org>.
clintropolis commented on pull request #11238:
URL: https://github.com/apache/druid/pull/11238#issuecomment-840254238
The script appears to be making the right things run within the categories, however I'm having a bit of trouble with travis on my fork, which appears to be build cache related with jobs hanging and such, but I am unable to retrigger any jobs for some reason (amusingly it returns 403 status code), so I'll try to push some more commits to these branches in a bit to try to get things to run through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] nishantmonu51 commented on pull request #11238: chill, travis
Posted by GitBox <gi...@apache.org>.
nishantmonu51 commented on pull request #11238:
URL: https://github.com/apache/druid/pull/11238#issuecomment-845546927
looking forward to getting this in!! great initiative @clintropolis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on pull request #11238: chill, travis
Posted by GitBox <gi...@apache.org>.
clintropolis commented on pull request #11238:
URL: https://github.com/apache/druid/pull/11238#issuecomment-845781072
moving this to #11283
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #11238: chill, travis
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #11238:
URL: https://github.com/apache/druid/pull/11238#discussion_r636632021
##########
File path: check-test-suite.py
##########
@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import subprocess
+import sys
+
+# do some primitive examination of git diff to determine if a test suite needs to be run or not
+
+always_run_jobs = ['license checks', '(openjdk8) packaging check', '(openjdk11) packaging check']
+
+# ignore changes to these files completely since they don't impact CI, if the changes are only to these files then all
+# of CI can be skipped
+ignore_prefixes = ['.github', '.idea', '.asf.yaml', '.backportrc.json', '.codecov.yml', '.dockerignore', '.gitignore',
+ '.lgtm.yml', 'CONTRIBUTING.md', 'setup-hooks.sh', 'upload.sh', 'dev', 'distribution/docker',
+ 'distribution/asf-release-process-guide.md', '.travis.yml', 'check-test-suite.py']
+# these files are docs changes
+# if changes are limited to this set then we can skip web-console and java
+# if no changes in this set we can skip docs
+docs_prefixes = ['docs/', 'website/']
+# travis docs job name
+docs_jobs = ['docs']
+# these files are web-console changes
+# if changes are limited to this set then we can skip docs and java
+# if no changes in this set we can skip web-console
+web_console_prefixes = ['web-console/']
+# travis web-console job name
+web_console_jobs = ['web console', 'web console end-to-end test']
+
+
+def check_ignore(file):
+ is_always_ignore = True in (file.startswith(prefix) for prefix in ignore_prefixes)
+ if is_always_ignore:
+ print("found ignorable file change: {}".format(file))
+ return is_always_ignore
+
+def check_docs(file):
+ is_docs = True in (file.startswith(prefix) for prefix in docs_prefixes)
+ if is_docs:
+ print("found docs file change: {}".format(file))
+ return is_docs
+
+def check_console(file):
+ is_console = True in (file.startswith(prefix) for prefix in web_console_prefixes)
+ if is_console:
+ print("found web-console file change: {}".format(file))
+ return is_console
+
+def check_should_run_suite(suite, diff_files):
Review comment:
yeah, i think it might be worth adding a unit test of some sort
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis closed pull request #11238: chill, travis
Posted by GitBox <gi...@apache.org>.
clintropolis closed pull request #11238:
URL: https://github.com/apache/druid/pull/11238
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #11238: chill, travis
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #11238:
URL: https://github.com/apache/druid/pull/11238#discussion_r636631764
##########
File path: check-test-suite.py
##########
@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import subprocess
+import sys
+
+# do some primitive examination of git diff to determine if a test suite needs to be run or not
+
+always_run_jobs = ['license checks', '(openjdk8) packaging check', '(openjdk11) packaging check']
+
+# ignore changes to these files completely since they don't impact CI, if the changes are only to these files then all
+# of CI can be skipped
+ignore_prefixes = ['.github', '.idea', '.asf.yaml', '.backportrc.json', '.codecov.yml', '.dockerignore', '.gitignore',
+ '.lgtm.yml', 'CONTRIBUTING.md', 'setup-hooks.sh', 'upload.sh', 'dev', 'distribution/docker',
+ 'distribution/asf-release-process-guide.md', '.travis.yml', 'check-test-suite.py']
+# these files are docs changes
+# if changes are limited to this set then we can skip web-console and java
+# if no changes in this set we can skip docs
+docs_prefixes = ['docs/', 'website/']
+# travis docs job name
+docs_jobs = ['docs']
+# these files are web-console changes
+# if changes are limited to this set then we can skip docs and java
+# if no changes in this set we can skip web-console
+web_console_prefixes = ['web-console/']
+# travis web-console job name
+web_console_jobs = ['web console', 'web console end-to-end test']
+
+
+def check_ignore(file):
+ is_always_ignore = True in (file.startswith(prefix) for prefix in ignore_prefixes)
+ if is_always_ignore:
+ print("found ignorable file change: {}".format(file))
+ return is_always_ignore
+
+def check_docs(file):
+ is_docs = True in (file.startswith(prefix) for prefix in docs_prefixes)
+ if is_docs:
+ print("found docs file change: {}".format(file))
+ return is_docs
+
+def check_console(file):
+ is_console = True in (file.startswith(prefix) for prefix in web_console_prefixes)
+ if is_console:
+ print("found web-console file change: {}".format(file))
+ return is_console
+
+def check_should_run_suite(suite, diff_files):
+ if suite in always_run_jobs:
+ # you gotta do what you gotta do
+ return True
+
+ all_ignore = True
+ any_docs = False
+ all_docs = True
+ any_console = False
+ all_console = True
+
+ for f in diff_files:
+ all_ignore = all_ignore and check_ignore(f)
+ is_docs = check_docs(f)
+ any_docs = any_docs or is_docs
+ all_docs = all_docs and is_docs
+ is_console = check_console(f)
+ any_console = any_console or is_console
+ all_console = any_console and is_console
+
+ if all_ignore:
+ return False
+ if suite in docs_jobs:
+ return any_docs
+ if all_docs:
+ return False
+ if suite in web_console_jobs:
+ return any_console
+ if all_console:
+ return False
+
+ return True
+
+def failWithUsage():
+ sys.stderr.write("usage: check-test-suite.py <test-suite-name>\n")
+ sys.stderr.write(" e.g., check-test-suite.py docs")
+ sys.exit(1)
+
+suite_name = ""
+
+if len(sys.argv) == 1:
+ if 'TRAVIS_JOB_NAME' in os.environ:
+ suite_name = os.environ['TRAVIS_JOB_NAME']
+ else:
+ failWithUsage()
+elif len(sys.argv) == 2:
+ suite_name = sys.argv[1]
+else:
+ failWithUsage()
+
+
+all_changed_files_string = subprocess.check_output("git diff --name-only HEAD~1", shell=True).decode('UTF-8')
Review comment:
I think per discussion in https://github.com/DefinitelyTyped/DefinitelyTyped/issues/1314 which i came across when searching the interwebs for how to best do this stuff, that it would appear this command works specifically for travis PR builds because of the way it runs them on top of the branch you are merging into.
You're right that this wouldn't be sufficient for branch builds, so this script should either always indicate that non-PR builds should run the entire suite, or, have a different command to use in that case. The way the current diff check is performed could also probably work I think, https://github.com/apache/druid/blob/master/.travis.yml#L176 if we want to enable selectively running tests on branch commits.
Thinking about it though, it might be safest for now to just always run the entire set on branch builds, at least for now, in case this script messes something up and skips too much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #11238: chill, travis
Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #11238:
URL: https://github.com/apache/druid/pull/11238#discussion_r631895643
##########
File path: check-test-suite.py
##########
@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import subprocess
+import sys
+
+# do some primitive examination of git diff to determine if a test suite needs to be run or not
+
+always_run_jobs = ['license checks', '(openjdk8) packaging check', '(openjdk11) packaging check']
+
+# ignore changes to these files completely since they don't impact CI, if the changes are only to these files then all
+# of CI can be skipped
+ignore_prefixes = ['.github', '.idea', '.asf.yaml', '.backportrc.json', '.codecov.yml', '.dockerignore', '.gitignore',
+ '.lgtm.yml', 'CONTRIBUTING.md', 'setup-hooks.sh', 'upload.sh', 'dev', 'distribution/docker',
+ 'distribution/asf-release-process-guide.md', '.travis.yml', 'check-test-suite.py']
+# these files are docs changes
+# if changes are limited to this set then we can skip web-console and java
+# if no changes in this set we can skip docs
+docs_prefixes = ['docs/', 'website/']
+# travis docs job name
+docs_jobs = ['docs']
+# these files are web-console changes
+# if changes are limited to this set then we can skip docs and java
+# if no changes in this set we can skip web-console
+web_console_prefixes = ['web-console/']
+# travis web-console job name
+web_console_jobs = ['web console', 'web console end-to-end test']
+
+
+def check_ignore(file):
+ is_always_ignore = True in (file.startswith(prefix) for prefix in ignore_prefixes)
+ if is_always_ignore:
+ print("found ignorable file change: {}".format(file))
+ return is_always_ignore
+
+def check_docs(file):
+ is_docs = True in (file.startswith(prefix) for prefix in docs_prefixes)
+ if is_docs:
+ print("found docs file change: {}".format(file))
+ return is_docs
+
+def check_console(file):
+ is_console = True in (file.startswith(prefix) for prefix in web_console_prefixes)
+ if is_console:
+ print("found web-console file change: {}".format(file))
+ return is_console
+
+def check_should_run_suite(suite, diff_files):
+ if suite in always_run_jobs:
+ # you gotta do what you gotta do
+ return True
+
+ all_ignore = True
+ any_docs = False
+ all_docs = True
+ any_console = False
+ all_console = True
+
+ for f in diff_files:
+ all_ignore = all_ignore and check_ignore(f)
+ is_docs = check_docs(f)
+ any_docs = any_docs or is_docs
+ all_docs = all_docs and is_docs
+ is_console = check_console(f)
+ any_console = any_console or is_console
+ all_console = any_console and is_console
+
+ if all_ignore:
+ return False
+ if suite in docs_jobs:
+ return any_docs
+ if all_docs:
+ return False
+ if suite in web_console_jobs:
+ return any_console
+ if all_console:
+ return False
+
+ return True
+
+def failWithUsage():
+ sys.stderr.write("usage: check-test-suite.py <test-suite-name>\n")
+ sys.stderr.write(" e.g., check-test-suite.py docs")
+ sys.exit(1)
+
+suite_name = ""
+
+if len(sys.argv) == 1:
+ if 'TRAVIS_JOB_NAME' in os.environ:
+ suite_name = os.environ['TRAVIS_JOB_NAME']
+ else:
+ failWithUsage()
+elif len(sys.argv) == 2:
+ suite_name = sys.argv[1]
+else:
+ failWithUsage()
+
+
+all_changed_files_string = subprocess.check_output("git diff --name-only HEAD~1", shell=True).decode('UTF-8')
Review comment:
Also, can you comment on how this PR will ensure all tests are still run on master or release branches like `0.21.1`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org