You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "dweiss (via GitHub)" <gi...@apache.org> on 2025/09/18 14:59:10 UTC
[PR] Use git-ls-files as the source of file paths for license checks. [lucene]
dweiss opened a new pull request, #15200:
URL: https://github.com/apache/lucene/pull/15200
Rob's idea. Works quite well, actually. I couldn't pinpoint why the set of paths is slightly different from before... gradle does have a "default exclusion" pattern set so it's possible it excluded something on its own (?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Use git-ls-files as the source of file paths for license checks. [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15200:
URL: https://github.com/apache/lucene/pull/15200#issuecomment-3307983357
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Use git-ls-files as the source of file paths for license checks. [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on code in PR #15200:
URL: https://github.com/apache/lucene/pull/15200#discussion_r2359772528
##########
build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitFileListValueSource.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.gradle.plugins.gitinfo;
+
+import java.io.BufferedOutputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import javax.inject.Inject;
+import org.apache.tools.ant.taskdefs.condition.Os;
+import org.gradle.api.GradleException;
+import org.gradle.api.file.DirectoryProperty;
+import org.gradle.api.provider.ValueSource;
+import org.gradle.api.provider.ValueSourceParameters;
+import org.gradle.api.services.BuildServiceParameters;
+import org.gradle.process.ExecOperations;
+import org.slf4j.LoggerFactory;
+
+/** Returns a list of versioned and non-versioned (but not ignored) git files. */
+public abstract class GitFileListValueSource
+ implements ValueSource<List<String>, GitFileListValueSource.Parameters> {
+ public abstract static class Parameters implements BuildServiceParameters, ValueSourceParameters {
+ public abstract DirectoryProperty getRootProjectDir();
+ }
+
+ @Inject
+ public abstract ExecOperations getExecOps();
+
+ @Override
+ public List<String> obtain() {
+ try (var baos = new ByteArrayOutputStream();
+ var out = new BufferedOutputStream(baos)) {
+ var result =
+ getExecOps()
+ .exec(
+ spec -> {
+ spec.setStandardOutput(out);
+ spec.setErrorOutput(out);
+ spec.setIgnoreExitValue(true);
+
+ spec.setWorkingDir(
+ getParameters().getRootProjectDir().getAsFile().get().getAbsolutePath());
+ spec.setExecutable(Os.isFamily(Os.FAMILY_WINDOWS) ? "git.exe" : "git");
+ spec.args(
+ "ls-files",
+ // show cached
+ "-c",
+ // show others
+ "-o",
+ // apply .gitignore exclusions
+ "--exclude-standard",
+ // don't quote paths, 0-terminate strings.
+ "-z");
+ });
+ out.flush();
+
+ // Assume the output is UTF-8, even if it has 0 eols.
+ String gitOutput = baos.toString(StandardCharsets.UTF_8);
Review Comment:
This seems... correct, right? If there are broken paths (non-utf8 characters) in git then we're screwed but it doesn't seem like the place to do any sophisticated recovery here...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Use git-ls-files as the source of file paths for license checks. [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on code in PR #15200:
URL: https://github.com/apache/lucene/pull/15200#discussion_r2360153454
##########
build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/licenses/CheckLicensesPlugin.java:
##########
@@ -48,79 +53,96 @@ public void apply(Project project) {
private void configureCheckLicenses(CheckLicensesTask task) {
Project project = task.getProject();
- task.getReportFile().set(project.getLayout().getBuildDirectory().file("licenses-report.txt"));
- task.getFiles()
- .from(
- project.fileTree(
- ".",
- tree -> {
- // Exclude build outputs, ide files, .git.
- tree.exclude(".git");
- tree.exclude(".idea");
- tree.exclude(".muse");
- tree.exclude("**/build/**");
- tree.exclude("**/.gradle");
+ assert project.getRootProject() == project;
+ var allNonIgnoredFiles =
+ project.getExtensions().getByType(GitInfoExtension.class).getAllNonIgnoredProjectFiles();
- // Exclude generated stuff.
- tree.exclude("**/src/generated/**");
+ task.getReportFile().set(project.getLayout().getBuildDirectory().file("licenses-report.txt"));
- // Exclude github stuff (templates, workflows).
- tree.exclude(".github");
+ // Build a list of files excluded from the license check. Just reuse ant's glob patterns.
+ var rootDir = getProjectRootPath(project);
+ var excludedPaths =
+ List.of(
+ // Ignore binary files. Previously we used apache rat, which had a 'binary guesser'
+ // but it's faster to just exclude by name (rather than scan the file).
+ "**/*.adoc",
+ "**/*.bin",
+ "**/*.brk",
+ "**/*.bz2",
+ "**/*.dat",
+ "**/*.gif",
+ "**/*.gz",
+ "**/*.png",
+ "**/*.svg",
+ "**/*.xls",
+ "**/*.zip",
Review Comment:
as a followup any of these we have, maybe we can configure in .gitattributes. at least `git grep` will then recognize it as binary, and possibly github. Especially useful for .svg which isnt "really" binary but is annoying AF in search results and diffs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Use git-ls-files as the source of file paths for license checks. [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss merged PR #15200:
URL: https://github.com/apache/lucene/pull/15200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Use git-ls-files as the source of file paths for license checks. [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #15200:
URL: https://github.com/apache/lucene/pull/15200#issuecomment-3309158149
https://jenkins.thetaphi.de/job/Lucene-main-Windows/16867/consoleText
Seems like something is not working on windows after all. I'll revert this and reapply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Use git-ls-files as the source of file paths for license checks. [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15200:
URL: https://github.com/apache/lucene/pull/15200#issuecomment-3307972827
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org