You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2025/03/27 00:39:30 UTC
[PR] [SPARK-51618][INFRA] Add a check to reject jars in CI [spark]
HyukjinKwon opened a new pull request, #50416:
URL: https://github.com/apache/spark/pull/50416
### What changes were proposed in this pull request?
This PR adds a check if the commits add/fix jars, and reject.
### Why are the changes needed?
It is discouraged to have jars in the repo.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
l manually tested with the jar removal (https://github.com/apache/spark/commit/82f75fec4bdd58d45f6d69c2c18bbe097590caf4) and changing `dev/test-jars.txt` and:
```
APACHE_SPARK_REF=HEAD~1 dev/is-changed.py
```
It will fails as below:
```
Cannot include jars in source codes. If this has to be added temporarily, please add the file name into dev/test-jars.txt.
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51618][INFRA] Add a check to reject jars in CI [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #50416:
URL: https://github.com/apache/spark/pull/50416#discussion_r2015434202
##########
dev/is-changed.py:
##########
@@ -65,6 +65,20 @@ def main():
changed_files = identify_changed_files_from_git_commits(
os.environ["GITHUB_SHA"], target_ref=os.environ["GITHUB_PREV_SHA"]
)
+
+ if any(f.endswith(".jar") for f in changed_files):
+ with open(
+ os.path.join(os.path.dirname(os.path.realpath(__file__)), "test-jars.txt")
+ ) as jarlist:
+ inter = set((line.strip() for line in jarlist.readlines())).intersection(
+ set(changed_files)
+ )
+ if len(inter) > 0:
+ raise SystemExit(
+ "Cannot include jars in source codes. If this has to be added temporarily, "
+ "please add the file name into dev/test-jars.txt."
Review Comment:
sg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51618][INFRA] Add a check to reject jars in CI [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #50416:
URL: https://github.com/apache/spark/pull/50416#issuecomment-2756482119
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51618][INFRA] Add a check to reject jars in CI [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #50416: [SPARK-51618][INFRA] Add a check to reject jars in CI
URL: https://github.com/apache/spark/pull/50416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51618][INFRA] Add a check to reject jars in CI [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #50416:
URL: https://github.com/apache/spark/pull/50416#discussion_r2015432293
##########
dev/is-changed.py:
##########
@@ -65,6 +65,20 @@ def main():
changed_files = identify_changed_files_from_git_commits(
os.environ["GITHUB_SHA"], target_ref=os.environ["GITHUB_PREV_SHA"]
)
+
+ if any(f.endswith(".jar") for f in changed_files):
+ with open(
+ os.path.join(os.path.dirname(os.path.realpath(__file__)), "test-jars.txt")
+ ) as jarlist:
+ inter = set((line.strip() for line in jarlist.readlines())).intersection(
+ set(changed_files)
+ )
+ if len(inter) > 0:
+ raise SystemExit(
+ "Cannot include jars in source codes. If this has to be added temporarily, "
+ "please add the file name into dev/test-jars.txt."
Review Comment:
Maybe we should mention that the author needs to add an assumption of the artifact's existence for related tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51618][INFRA] Add a check to reject jars in CI [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #50416:
URL: https://github.com/apache/spark/pull/50416#issuecomment-2756073090
cc @yaooqinn @hvanhovell @pan3793 this addresses your comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org