You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "dweiss (via GitHub)" <gi...@apache.org> on 2026/08/21 07:19:16 UTC
[PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
dweiss opened a new pull request, #16543:
URL: https://github.com/apache/lucene/pull/16543
This patch adds a way to specify custom locations for gradle wrapper and gradle distribution. The assumptions were:
1) minimize surface of this change to a single class emulating gradle-wrapper's logic to "install" the gradle distribution specified in `gradle-wrapper.properties`, if it's not installed,
2) custom wrapper and distribution URLs passed via env variables, with `${gradleVersion}` replaceable pattern (works like gradle's official distributionUrl).
3) ensure the downloaded distribution matches the expected checksums; the install path should match the "official" URL and thus be transparent to other scripts and checks in Lucene.
I've tested this locally, for example:
```
$ find ~/tmp/gradle
.
./9.7.0
./9.7.0/gradle-9.7.0-bin.zip
./9.7.0/gradle-wrapper.jar
$ rm -rf ~/.gradle/wrapper
$ LUCENE_GRADLE_WRAPPER_URL='file:///home/dweiss/tmp/gradle/${gradleVersion}/gradle-wrapper.jar' LUCENE_GRADLE_DISTRIBUTION_URL='file:///home/dweiss/tmp/gradle/${gradleVersion}/gradle-${gradleVersion}-bin.zip' ./gradlew tasks
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #16543:
URL: https://github.com/apache/lucene/pull/16543#discussion_r3868305083
##########
.github/workflows/run-checks-gradle-upgrade.yml:
##########
Review Comment:
It's dubious to me if the weight here is worth its value but I leave that to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5414266171
Like I mentioned, if you don't care about it then this should be a one-time operation:
```
LUCENE_GRADLE_DISTRIBUTION_URL=[corporate gradle distro mirror]
LUCENE_GRADLE_WRAPPER_URL=file:///home/myhome/tmp/gradle-wrapper.jar
LUCENE_GRADLE_VERIFY_CHECKSUMS=false
```
Is this something you'd be comfortable with (and would it be convenient in your workflow)? If so, I can review the changes in this patch or commit it back to your original PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5596953316
Merged to main and 10x. I would set up those intranet env variables in a bash/zsh profile. Let me know how this works in practice - I'm curious if it's working for you as intended (that is: you set it up once and pretty much forget about it; apart from setting up your intranet-local gradle release mirrors).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5590641482
Thanks, David. I'll work on it and merge it in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss merged PR #16543:
URL: https://github.com/apache/lucene/pull/16543
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5430974126
I've cleaned up LLM verbosity and added some sanity checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5374277685
In short, it is similar but not the same. It is different enough that I thought I'd file a separate PR instead of pushing to your branch (but I can, it's up to you).
I wanted to keep this thing entirely separate (so that no extra stuff needs to go into gradle-wrapper.properties, for example) and optional (so that unless you specify those env properties, nothing changes for anybody else). The logic we have now works fine for many people and it's a result of multiple iterations of changes, I don't think we should change it, to be honest.
So everything related to this "intranet-mirror" is self-contained in a single, separate class. Also, contrary to your patch, this one simulates gradle wrapper's logic pretty closely - including the same lock files, sha checksums matching the "official" distribution and a separate variable for fetching both the wrapper and the distribution code.
I initially made some shortcuts (launched gradle directly, without gradle-wrapper.jar) but it wasn't good - there are manifest entries in gradle-wrapper.jar that would have to be passed to the bootstrap class (Enable-Native-Access: ALL-UNNAMED)... it was way more complicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5374518780
I guess the big question is if this sort of solution would work for you, David (with two env variables). I think it should be fairly convenient (you can even have those distribution zips mirrored locally, like the above example shows - this way the network is never accessed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5410579584
> It may not change from version to version but each version expects its aligned wrapper.
That's a highly presumptuous claim. And my experience is contrary to your presumption: I've seen no issues locally (misc projects at work) with even Gradle major version divergence between wrapper & distribution.
The wrapper is on their "release train" and gets released without necessarily any changes. I wish they didn't bother doing that given how stable & backwards compatible it is... but whatever.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #16543:
URL: https://github.com/apache/lucene/pull/16543#discussion_r3866684419
##########
build-tools/build-infra/src/main/java/org/apache/lucene/gradle/GradleMirrorSetup.java:
##########
@@ -0,0 +1,531 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.gradle;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Reader;
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+import java.math.BigInteger;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLConnection;
+import java.nio.channels.FileChannel;
+import java.nio.channels.FileLock;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.FileSystem;
+import java.nio.file.FileSystems;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.security.MessageDigest;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Locale;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Standalone class run by {@code gradlew}/{@code gradlew.bat} (before the gradle wrapper) when the
+ * {@value #LUCENE_GRADLE_DISTRIBUTION_URL_ENV} and/or {@value #LUCENE_GRADLE_WRAPPER_URL_ENV}
+ * environment variables are set. This should be used in environments that cannot reach GitHub
+ * and/or services.gradle.org.
+ *
+ * <p>This class downloads (each step only if the corresponding variable is set):
+ *
+ * <ul>
+ * <li>{@code gradle-wrapper.jar} from {@value #LUCENE_GRADLE_WRAPPER_URL_ENV} into {@code
+ * gradle/wrapper/} (otherwise GradleWrapperDownloader would fetch it from GitHub),
+ * <li>the gradle distribution from {@value #LUCENE_GRADLE_DISTRIBUTION_URL_ENV}, installing it in
+ * the gradle user home exactly where the gradle wrapper would install the "official"
+ * distribution so that the wrapper considers it already installed.
+ * </ul>
+ *
+ * <p>Both URLs may contain a {@code ${gradleVersion}} placeholder, replaced with the version parsed
+ * from {@code distributionUrl} in {@code gradle-wrapper.properties}. For example:
+ *
+ * <pre>
+ * LUCENE_GRADLE_DISTRIBUTION_URL=file:///tmp/local-gradle-mirror/${gradleVersion}/gradle-${gradleVersion}-bin.zip
+ * LUCENE_GRADLE_WRAPPER_URL=file:///tmp/local-gradle-mirror/${gradleVersion}/gradle-wrapper.jar
+ * </pre>
+ *
+ * <p>By default, checksums of the distribution and/or the wrapper must match those that are present
+ * in {@code gradle-wrapper.properties}. Setting {@value #LUCENE_GRADLE_VERIFY_CHECKSUMS_ENV} to
+ * {@code false} disables all checksum verification: an existing {@code gradle-wrapper.jar} is used
+ * as-is and downloads are installed unverified.
+ *
+ * <p>This class must not have any dependencies outside of standard java libraries as it's run
+ * directly from source.
+ *
+ * <p>The code in this class is based on automated analysis of gradle wrapper sources from gradle
+ * v9.7.1.
+ */
+public class GradleMirrorSetup {
+ public static final String LUCENE_GRADLE_DISTRIBUTION_URL_ENV = "LUCENE_GRADLE_DISTRIBUTION_URL";
+ public static final String LUCENE_GRADLE_WRAPPER_URL_ENV = "LUCENE_GRADLE_WRAPPER_URL";
+ public static final String LUCENE_GRADLE_VERIFY_CHECKSUMS_ENV = "LUCENE_GRADLE_VERIFY_CHECKSUMS";
+
+ /** Inline this annotation to keep this class self-contained. */
+ @Retention(RetentionPolicy.CLASS)
+ @Target({ElementType.CONSTRUCTOR, ElementType.FIELD, ElementType.METHOD, ElementType.TYPE})
+ private @interface SuppressForbidden {
+ String reason();
+ }
+
+ /**
+ * Checksum verification is on unless {@value #LUCENE_GRADLE_VERIFY_CHECKSUMS_ENV} is set to
+ * "false".
+ */
+ private static final boolean VERIFY_CHECKSUMS =
+ !"false".equalsIgnoreCase(System.getenv(LUCENE_GRADLE_VERIFY_CHECKSUMS_ENV));
+
+ /** Used when {@code networkTimeout} is not set in {@code gradle-wrapper.properties}. */
+ private static final int DEFAULT_NETWORK_TIMEOUT_MILLIS = (int) TimeUnit.SECONDS.toMillis(10);
+
+ private static final Pattern DISTRIBUTION_NAME =
+ Pattern.compile("gradle-(?<version>.+?)-(bin|all)\\.zip");
+ private static final Pattern PLACEHOLDER = Pattern.compile("\\$\\{(?<varname>[^}]*)\\}");
+ private static final boolean IS_WINDOWS =
+ System.getProperty("os.name").toLowerCase(Locale.ROOT).contains("windows");
+
+ private final Path projectDir;
+ private final Path gradleUserHome;
+
+ public static void main(String[] args) {
+ if (args.length < 1) {
+ System.err.println("Usage: java GradleMirrorSetup.java <project dir> [gradle arguments]");
+ System.exit(2);
+ }
+
+ try {
+ checkVersion();
+ new GradleMirrorSetup(Paths.get(args[0]), args).run();
+ } catch (Exception e) {
+ System.err.println("ERROR: " + e.getMessage());
+ System.exit(3);
+ }
+ }
+
+ public static void checkVersion() {
+ int major = Runtime.version().feature();
+ if (major < 17) {
+ throw new IllegalStateException("java version must be 17 or later, your version: " + major);
+ }
+ }
+
+ GradleMirrorSetup(Path projectDir, String[] gradleArgs) {
+ this.projectDir = projectDir.toAbsolutePath().normalize();
+
+ // Gradle user home: -g/--gradle-user-home, GRADLE_USER_HOME or ~/.gradle.
+ String userHome = null;
+ for (int i = 1; i < gradleArgs.length; i++) {
+ String arg = gradleArgs[i];
+ if ((arg.equals("-g") || arg.equals("--gradle-user-home")) && i + 1 < gradleArgs.length) {
+ userHome = gradleArgs[++i];
+ } else if (arg.startsWith("--gradle-user-home=")) {
+ userHome = arg.substring("--gradle-user-home=".length());
+ }
+ }
+ if (userHome == null) {
+ userHome = System.getenv("GRADLE_USER_HOME");
+ }
+ this.gradleUserHome =
+ userHome != null
+ ? Paths.get(userHome)
+ : Paths.get(System.getProperty("user.home"), ".gradle");
+ }
+
+ void run() throws Exception {
+ Path wrapperDir = projectDir.resolve("gradle").resolve("wrapper");
+ Path propertiesFile = wrapperDir.resolve("gradle-wrapper.properties");
+ if (!Files.exists(propertiesFile)) {
+ throw new IOException("Wrapper property file not found: " + propertiesFile);
+ }
+ Properties props = new Properties();
+ try (Reader reader = Files.newBufferedReader(propertiesFile, StandardCharsets.UTF_8)) {
+ props.load(reader);
+ }
+ String distributionUrl = props.getProperty("distributionUrl");
+ if (distributionUrl == null) {
+ throw new IOException("No 'distributionUrl' in " + propertiesFile);
+ }
+ Matcher m = DISTRIBUTION_NAME.matcher(distributionUrl);
+ if (!m.find()) {
+ throw new IOException(
+ "Could not parse the gradle version from distributionUrl in "
+ + propertiesFile
+ + ": "
+ + distributionUrl);
+ }
+ String gradleVersion = m.group("version");
+ int timeout =
+ Integer.parseInt(
+ props
+ .getProperty("networkTimeout", String.valueOf(DEFAULT_NETWORK_TIMEOUT_MILLIS))
+ .trim());
+
+ String wrapperUrl = expand(System.getenv(LUCENE_GRADLE_WRAPPER_URL_ENV), gradleVersion);
+ String mirrorDistributionUrl =
+ expand(System.getenv(LUCENE_GRADLE_DISTRIBUTION_URL_ENV), gradleVersion);
+
+ if (!VERIFY_CHECKSUMS) {
+ log(
+ "NOTE: checksum verification disabled ("
+ + LUCENE_GRADLE_VERIFY_CHECKSUMS_ENV
+ + "=false).");
+ }
+
+ if (wrapperUrl != null) {
+ setupWrapperJar(wrapperDir, wrapperUrl, timeout);
+ }
+ if (mirrorDistributionUrl != null) {
+ setupDistribution(props, distributionUrl, mirrorDistributionUrl, gradleVersion, timeout);
+ }
+ }
+
+ /** Replaces {@code ${gradleVersion}}; null/blank input yields null. */
+ private static String expand(String template, String gradleVersion) throws IOException {
+ if (template == null || template.trim().isEmpty()) {
+ return null;
+ }
+ Matcher m = PLACEHOLDER.matcher(template.trim());
+ StringBuilder sb = new StringBuilder();
+ while (m.find()) {
+ if (!m.group("varname").equals("gradleVersion")) {
+ throw new IOException(
+ "Unknown placeholder "
+ + m.group()
+ + " (only ${gradleVersion} is supported) in: "
+ + template);
+ }
+ m.appendReplacement(sb, Matcher.quoteReplacement(gradleVersion));
+ }
+ return m.appendTail(sb).toString();
+ }
+
+ /**
+ * Makes sure gradle/wrapper/gradle-wrapper.jar is present and matches gradle-wrapper.jar.sha256,
+ * downloading it from the mirror if needed. With checksum verification disabled, any existing jar
+ * is used as-is and downloads are not verified.
+ */
+ private void setupWrapperJar(Path wrapperDir, String wrapperUrl, int timeout) throws Exception {
+ Path jar = wrapperDir.resolve("gradle-wrapper.jar");
+ Path checksumFile = wrapperDir.resolve("gradle-wrapper.jar.sha256");
+ String expectedChecksum = null;
+ if (VERIFY_CHECKSUMS) {
+ for (String line : Files.readAllLines(checksumFile, StandardCharsets.UTF_8)) {
+ // sha256sum format: "<checksum> *gradle-wrapper.jar" ('*' marks binary mode).
+ String[] parts = line.trim().split("\\s+");
+ if (parts.length == 2 && parts[1].replaceFirst("^\\*", "").equals("gradle-wrapper.jar")) {
+ expectedChecksum = parts[0];
+ }
+ }
+ if (expectedChecksum == null) {
+ throw new IOException("No checksum for gradle-wrapper.jar in " + checksumFile);
+ }
+ }
+
+ if (Files.exists(jar)
+ && (!VERIFY_CHECKSUMS || sha256(jar).equalsIgnoreCase(expectedChecksum))) {
+ return;
+ }
+
+ log("Downloading gradle-wrapper.jar from " + wrapperUrl);
+ Path temp = Files.createTempFile(wrapperDir, ".gradle-wrapper", ".tmp");
+ try {
+ download(new URI(wrapperUrl), temp, timeout);
+ if (VERIFY_CHECKSUMS) {
+ String actualChecksum = sha256(temp);
+ if (!actualChecksum.equalsIgnoreCase(expectedChecksum)) {
+ throw new IOException(
+ "The gradle-wrapper.jar downloaded from "
+ + wrapperUrl
+ + " does not match "
+ + checksumFile
+ + " (expected: "
+ + expectedChecksum
+ + ", actual: "
+ + actualChecksum
+ + ").");
+ }
+ }
+ Files.move(temp, jar, StandardCopyOption.REPLACE_EXISTING);
+ } finally {
+ Files.deleteIfExists(temp);
+ }
+ }
+
+ /**
+ * Downloads the distribution from the mirror and installs it where the gradle wrapper would
+ * install the official one: {@code <base>/<path>/<name>/<md5 of the official url as base36>/}.
+ */
+ private void setupDistribution(
+ Properties props, String distributionUrl, String mirrorUrl, String gradleVersion, int timeout)
+ throws Exception {
+ String expectedSha256 = props.getProperty("distributionSha256Sum");
+ if (expectedSha256 == null || expectedSha256.trim().isEmpty()) {
+ if (VERIFY_CHECKSUMS) {
+ throw new IOException(
+ "No 'distributionSha256Sum' in gradle-wrapper.properties; refusing to install a gradle"
+ + " distribution without checksum verification.");
+ }
+ expectedSha256 = null;
+ } else {
+ expectedSha256 = expectedSha256.trim();
+ }
+
+ URI official = new URI(distributionUrl);
+ String zipName = official.getPath().replaceAll(".*/", "");
+ String distName = zipName.replaceAll("\\.[^.]*$", "");
+ String hash =
+ new BigInteger(
+ 1,
+ MessageDigest.getInstance("MD5")
Review Comment:
## CodeQL / Use of a potentially broken or risky cryptographic algorithm
Cryptographic algorithm [MD5](1) may not be secure. Consider using a different algorithm.
[Show more details](https://github.com/apache/lucene/security/code-scanning/269)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5445603901
Did you notice https://github.com/Glavo/gradle-wrapper-neo ? I subscribe to some of those Gradle tickets and saw the post about this. Looks like exactly what we need! Outside of Gradle doing it themselves, of course.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #16543:
URL: https://github.com/apache/lucene/pull/16543#discussion_r3837079324
##########
build-tools/build-infra/src/main/java/org/apache/lucene/gradle/IntranetGradleSetup.java:
##########
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.gradle;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Reader;
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+import java.math.BigInteger;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLConnection;
+import java.nio.channels.FileChannel;
+import java.nio.channels.FileLock;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.FileSystem;
+import java.nio.file.FileSystems;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.security.MessageDigest;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Locale;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Standalone class run by {@code gradlew}/{@code gradlew.bat} (before the gradle wrapper) when the
+ * {@value #WRAPPER_URL_ENV} and/or {@value #DISTRIBUTION_URL_ENV} environment variables are set. It
+ * supports environments that cannot reach GitHub and services.gradle.org by downloading:
+ *
+ * <ul>
+ * <li>{@code gradle-wrapper.jar} from {@value #WRAPPER_URL_ENV} into {@code gradle/wrapper/}
+ * (verified against the checksum in {@code gradle-wrapper.jar.sha256}, so that {@code
+ * WrapperDownloader} doesn't need to fetch it),
+ * <li>the gradle distribution from {@value #DISTRIBUTION_URL_ENV}, installing it in the gradle
+ * user home exactly where the gradle wrapper would install the "official" distribution (under
+ * the hash of the {@code distributionUrl} from {@code gradle-wrapper.properties}), so that
+ * the wrapper considers it already installed.
+ * </ul>
+ *
+ * <p>Both URLs may contain a {@code ${gradleVersion}} placeholder, replaced with the version parsed
+ * from {@code distributionUrl} in {@code gradle-wrapper.properties}.
+ *
+ * <p>Ensure this class has no dependencies outside of standard java libraries as it's run directly
+ * from source. It only uses JDK 11 language features and APIs so that it runs on any reasonably
+ * recent JDK (17+ is required by {@link #checkVersion()}; the actual build requires a newer one).
+ */
+public class IntranetGradleSetup {
+ /**
+ * Copied to keep the class isolated from any other classes.
+ *
+ * @see "https://github.com/apache/lucene/issues/15399"
+ */
+ @Retention(RetentionPolicy.CLASS)
+ @Target({ElementType.CONSTRUCTOR, ElementType.FIELD, ElementType.METHOD, ElementType.TYPE})
+ private @interface SuppressForbidden {
+ /** A reason for suppressing should always be given. */
+ String reason();
+ }
+
+ public static final String DISTRIBUTION_URL_ENV = "LUCENE_GRADLE_DISTRIBUTION_URL";
+ public static final String WRAPPER_URL_ENV = "LUCENE_GRADLE_WRAPPER_URL";
+
+ private static final Pattern DISTRIBUTION_NAME =
+ Pattern.compile("gradle-(?<version>.+?)-(bin|all)\\.zip");
+ private static final Pattern PLACEHOLDER = Pattern.compile("\\$\\{([^}]*)\\}");
+ private static final boolean IS_WINDOWS =
+ System.getProperty("os.name").toLowerCase(Locale.ROOT).contains("windows");
+
+ private final Path projectDir;
+ private final Path gradleUserHome;
+
+ public static void main(String[] args) {
+ if (args.length < 1) {
+ System.err.println("Usage: java IntranetGradleSetup.java <project dir> [gradle arguments]");
+ System.exit(2);
+ }
+
+ try {
+ checkVersion();
+ new IntranetGradleSetup(Paths.get(args[0]), args).run();
+ } catch (Exception e) {
+ System.err.println("ERROR: " + e.getMessage());
+ System.exit(3);
+ }
+ }
+
+ public static void checkVersion() {
+ int major = Runtime.version().feature();
+ if (major < 17) {
+ throw new IllegalStateException("java version must be 17 or later, your version: " + major);
+ }
+ }
+
+ IntranetGradleSetup(Path projectDir, String[] gradleArgs) {
+ this.projectDir = projectDir.toAbsolutePath().normalize();
+
+ // Gradle user home: -g/--gradle-user-home, GRADLE_USER_HOME or ~/.gradle.
+ String userHome = null;
+ for (int i = 1; i < gradleArgs.length; i++) {
+ String arg = gradleArgs[i];
+ if ((arg.equals("-g") || arg.equals("--gradle-user-home")) && i + 1 < gradleArgs.length) {
+ userHome = gradleArgs[++i];
+ } else if (arg.startsWith("--gradle-user-home=")) {
+ userHome = arg.substring("--gradle-user-home=".length());
+ }
+ }
+ if (userHome == null) {
+ userHome = System.getenv("GRADLE_USER_HOME");
+ }
+ this.gradleUserHome =
+ userHome != null
+ ? Paths.get(userHome)
+ : Paths.get(System.getProperty("user.home"), ".gradle");
+ }
+
+ void run() throws Exception {
+ Path wrapperDir = projectDir.resolve("gradle").resolve("wrapper");
+ Path propertiesFile = wrapperDir.resolve("gradle-wrapper.properties");
+ if (!Files.exists(propertiesFile)) {
+ throw new IOException("Wrapper property file not found: " + propertiesFile);
+ }
+ Properties props = new Properties();
+ try (Reader reader = Files.newBufferedReader(propertiesFile, StandardCharsets.UTF_8)) {
+ props.load(reader);
+ }
+ String distributionUrl = props.getProperty("distributionUrl");
+ if (distributionUrl == null) {
+ throw new IOException("No 'distributionUrl' in " + propertiesFile);
+ }
+ Matcher m =
+ DISTRIBUTION_NAME.matcher(distributionUrl.replaceAll("\\?.*", "").replaceAll(".*/", ""));
Review Comment:
should have a comment to clarify what this is doing, maybe with an example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5413609910
> That's a highly presumptuous claim. And my experience is contrary to your presumption: I've seen no issues locally (misc projects at work) with even Gradle major version divergence between wrapper & distribution.
That something works doesn't mean it doesn't have the potential to break. Think of it this way - when you want to install the gradle wrapper, you use a certain gradle version to do so. When you update from a particular distribution, that distribution's wrapper will be written in your project - each one carries its own binary and writes it when needed.
Here's what gradle says about it [1] - you trust that their wording of "usually" means "always", I don't.
<img width="1594" height="642" alt="image" src="https://github.com/user-attachments/assets/1bec9279-cb0e-4295-83c4-93d0f75a4dd2" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5453107405
I've sorta know someone at Gradle who works with the ASF and I crafted an email to make a case that this matter needs attention/priority it's hasn't been getting.
RE neo:
Yes I observed the line count. Neo takes actual gradle wrapper source, which is why. I prefer "lean" but if we don't maintain it then I'm indifferent.
> I like the approach they have with a user-home configuration file though. Maybe it's more convenient than env variables? What do you think?
I like that too... and can be configured overlaying a project's gradle-wrapper.properties (if I recall).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5590090564
Any feedback from that friend of yours, @dsmiley ? I'll rebase this on top of main and commit/ backport to 10x. We can reiterate if there is a better/ standard alternative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on code in PR #16543:
URL: https://github.com/apache/lucene/pull/16543#discussion_r3839543715
##########
build-tools/build-infra/src/main/java/org/apache/lucene/gradle/IntranetGradleSetup.java:
##########
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.gradle;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Reader;
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+import java.math.BigInteger;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLConnection;
+import java.nio.channels.FileChannel;
+import java.nio.channels.FileLock;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.FileSystem;
+import java.nio.file.FileSystems;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.security.MessageDigest;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Locale;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Standalone class run by {@code gradlew}/{@code gradlew.bat} (before the gradle wrapper) when the
+ * {@value #WRAPPER_URL_ENV} and/or {@value #DISTRIBUTION_URL_ENV} environment variables are set. It
+ * supports environments that cannot reach GitHub and services.gradle.org by downloading:
+ *
+ * <ul>
+ * <li>{@code gradle-wrapper.jar} from {@value #WRAPPER_URL_ENV} into {@code gradle/wrapper/}
+ * (verified against the checksum in {@code gradle-wrapper.jar.sha256}, so that {@code
+ * WrapperDownloader} doesn't need to fetch it),
+ * <li>the gradle distribution from {@value #DISTRIBUTION_URL_ENV}, installing it in the gradle
+ * user home exactly where the gradle wrapper would install the "official" distribution (under
+ * the hash of the {@code distributionUrl} from {@code gradle-wrapper.properties}), so that
+ * the wrapper considers it already installed.
+ * </ul>
+ *
+ * <p>Both URLs may contain a {@code ${gradleVersion}} placeholder, replaced with the version parsed
+ * from {@code distributionUrl} in {@code gradle-wrapper.properties}.
+ *
+ * <p>Ensure this class has no dependencies outside of standard java libraries as it's run directly
+ * from source. It only uses JDK 11 language features and APIs so that it runs on any reasonably
+ * recent JDK (17+ is required by {@link #checkVersion()}; the actual build requires a newer one).
+ */
+public class IntranetGradleSetup {
+ /**
+ * Copied to keep the class isolated from any other classes.
+ *
+ * @see "https://github.com/apache/lucene/issues/15399"
+ */
+ @Retention(RetentionPolicy.CLASS)
+ @Target({ElementType.CONSTRUCTOR, ElementType.FIELD, ElementType.METHOD, ElementType.TYPE})
+ private @interface SuppressForbidden {
+ /** A reason for suppressing should always be given. */
+ String reason();
+ }
+
+ public static final String DISTRIBUTION_URL_ENV = "LUCENE_GRADLE_DISTRIBUTION_URL";
+ public static final String WRAPPER_URL_ENV = "LUCENE_GRADLE_WRAPPER_URL";
+
+ private static final Pattern DISTRIBUTION_NAME =
+ Pattern.compile("gradle-(?<version>.+?)-(bin|all)\\.zip");
+ private static final Pattern PLACEHOLDER = Pattern.compile("\\$\\{([^}]*)\\}");
+ private static final boolean IS_WINDOWS =
+ System.getProperty("os.name").toLowerCase(Locale.ROOT).contains("windows");
+
+ private final Path projectDir;
+ private final Path gradleUserHome;
+
+ public static void main(String[] args) {
+ if (args.length < 1) {
+ System.err.println("Usage: java IntranetGradleSetup.java <project dir> [gradle arguments]");
+ System.exit(2);
+ }
+
+ try {
+ checkVersion();
+ new IntranetGradleSetup(Paths.get(args[0]), args).run();
+ } catch (Exception e) {
+ System.err.println("ERROR: " + e.getMessage());
+ System.exit(3);
+ }
+ }
+
+ public static void checkVersion() {
+ int major = Runtime.version().feature();
+ if (major < 17) {
+ throw new IllegalStateException("java version must be 17 or later, your version: " + major);
+ }
+ }
+
+ IntranetGradleSetup(Path projectDir, String[] gradleArgs) {
+ this.projectDir = projectDir.toAbsolutePath().normalize();
+
+ // Gradle user home: -g/--gradle-user-home, GRADLE_USER_HOME or ~/.gradle.
+ String userHome = null;
+ for (int i = 1; i < gradleArgs.length; i++) {
+ String arg = gradleArgs[i];
+ if ((arg.equals("-g") || arg.equals("--gradle-user-home")) && i + 1 < gradleArgs.length) {
+ userHome = gradleArgs[++i];
+ } else if (arg.startsWith("--gradle-user-home=")) {
+ userHome = arg.substring("--gradle-user-home=".length());
+ }
+ }
+ if (userHome == null) {
+ userHome = System.getenv("GRADLE_USER_HOME");
+ }
+ this.gradleUserHome =
+ userHome != null
+ ? Paths.get(userHome)
+ : Paths.get(System.getProperty("user.home"), ".gradle");
+ }
+
+ void run() throws Exception {
+ Path wrapperDir = projectDir.resolve("gradle").resolve("wrapper");
+ Path propertiesFile = wrapperDir.resolve("gradle-wrapper.properties");
+ if (!Files.exists(propertiesFile)) {
+ throw new IOException("Wrapper property file not found: " + propertiesFile);
+ }
+ Properties props = new Properties();
+ try (Reader reader = Files.newBufferedReader(propertiesFile, StandardCharsets.UTF_8)) {
+ props.load(reader);
+ }
+ String distributionUrl = props.getProperty("distributionUrl");
+ if (distributionUrl == null) {
+ throw new IOException("No 'distributionUrl' in " + propertiesFile);
+ }
+ Matcher m =
+ DISTRIBUTION_NAME.matcher(distributionUrl.replaceAll("\\?.*", "").replaceAll(".*/", ""));
Review Comment:
I'll go through this code again if we decide to commit this approach - I just wanted to open a separate PR so that we can discuss this alternative take on the problem. It is machine generated code; I'll go through the patch and remove odd comments etc. if we decide this is the way to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5388337068
> I hoped we would either replace or maybe add a little to our WrapperDownloader. As an ASF project + gradle build, we need _something_ -- just trying to keep that _something_ narrow. I don't see why this new IntranetGradle does _both_ wrapper downloading and gradle distribution downloading.
I wanted to keep it separate from WrapperDownloader because... well, it is hacky. WrapperDownloader is simple, that other class is hairy because gradle is dumb not to offer this by default... This is the only reason - to keep this separate just to signal that it's something we do that we shouldn't really be doing (gradle should be allowing it).
> Maybe you disagree but if I'm going to set env vars to choose my gradle wrapper in some way, I'd rather no sha256 be verified, and/or allow me to have yet another env var to specify whatever the sha is for it.
Everyone is so obsessed about security these days that I thought this wouldn't hurt. It ensures gradle distribution you point at (even if it's hosted somewhere else) is identical to that expected from the official server. I personally don't care that much about it but it seems like a reasonable thing to do (?).
> I think most of all I want to make the wrapper distribution something in my control/env. I can have it on my machine somewhere but the build here needs to know where it is. _If_ I specify my gradle wrapper, Lucene shouldn't further insist what version and/or sha256 it is. I'm elecrting to choose the wrapper.
Get it. I think it's going to hurt you if they change something in an incompatible way - the wrapper jar is part of each distribution (major or minor) and it does change from time to time (what they change - I've no idea but the checksum changes [1]). Typically, this jar would be versioned alongside the rest of the sources - the decision to download it dynamically was driven by ASF requirements not to ship binaries with the source code, so the downloading/ sha check is a workaround but it still works as if the wrapper were versioned - for each commit it is fixed to a particular binary.
The simplest way to dodge checksum checking would be a small adjustment to gradlew scripts in which - if the env URLs are provided, we use them, download and install the jar/gradle distro and skip the original WrapperDownloader (which does checksum checking)... This said, I think it's better to keep those checksums - this will avoid potential odd surprises in the future. If you live behind a corporate firewall then it's fairly easy to create a directory structure for each gradle version needed + the corresponding wrapper (what I tried to show in the description of this patch). This is a one-time effort, the directory structure stays the same on disk and you can keep the env pointer URLs constant (they can use the replaceable gradleVersion part).
[1] https://gradle.org/release-checksums/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5452492151
The one thing I don't like about neo is that it's a one-way option for everyone. I'd rather use the "official" wrapper for everyone and have a loophole for those folks who can't access official services (for various reasons).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5410163330
Also - perhaps at the risk of stating the obvious - you don't need to run `gradlew` scripts at all. If you'd rather use your own gradle distribution(s) then they will work too - the version must match but otherwise you can run gradle directly (without any gradle-wrapper fiddling):
```
~/.gradle/wrapper/dists/gradle-9.7.0-bin/d4tj7w02tcgubx9zk9hbippn6/gradle-9.7.0/bin/gradle check -x test
```
We are strict about the gradle version used... maybe it'd be simpler to add an option to relax this check and allow compatible major versions too. Knowing how much changes from version to version I would never use it myself but it's also something to consider if you have such strong feelings about gradle-wrapper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5408158546
> Keeping an edited gradle-wrapper.properties with a corporate distributionUrl hasn't been too annoying to maintain.
I'm sorry but I disagree with you here. This file is outside of Lucene's control, it shouldn't be modified. When upstream changes gradle version you'll get conflicts on any pull/merge. It seems weird to accept it.
> The wrapper is not paired with the distribution requiring version alignment.
Of course it is - that's why they publish SHA checksums for each and every release:
https://gradle.org/release-checksums/
It may not *change* from version to version but each version expects its aligned wrapper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5452479314
No, I didn't know about it - good find. It's only a month old though... and looking at the code I think I like the much leaner version here (~500 lines) better - take a look at it [1] (3500 lines). I like the approach they have with a user-home configuration file though. Maybe it's more convenient than env variables? What do you think?
[1] https://github.com/Glavo/gradle-wrapper-neo/blob/main/gradle/wrapper/GradleWrapperNeo.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on code in PR #16543:
URL: https://github.com/apache/lucene/pull/16543#discussion_r3869493904
##########
.github/workflows/run-checks-gradle-upgrade.yml:
##########
Review Comment:
I use this workflow when upgrading gradle. I'd like to have this class covered so I leave it in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5435615957
> Boy I wish we could convince the Gradle folks to prioritize an ASF compatible solution.
They don't seem to care.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5414180863
Shrug... fair but the consequence is low. If it works, it works... (no need to upgrade) and if it doesn't then upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #16543:
URL: https://github.com/apache/lucene/pull/16543#discussion_r3828240538
##########
build-tools/build-infra/src/main/java/org/apache/lucene/gradle/IntranetGradleSetup.java:
##########
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.gradle;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Reader;
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+import java.math.BigInteger;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLConnection;
+import java.nio.channels.FileChannel;
+import java.nio.channels.FileLock;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.FileSystem;
+import java.nio.file.FileSystems;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.security.MessageDigest;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Locale;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Standalone class run by {@code gradlew}/{@code gradlew.bat} (before the gradle wrapper) when the
+ * {@value #WRAPPER_URL_ENV} and/or {@value #DISTRIBUTION_URL_ENV} environment variables are set. It
+ * supports environments that cannot reach GitHub and services.gradle.org by downloading:
+ *
+ * <ul>
+ * <li>{@code gradle-wrapper.jar} from {@value #WRAPPER_URL_ENV} into {@code gradle/wrapper/}
+ * (verified against the checksum in {@code gradle-wrapper.jar.sha256}, so that {@code
+ * WrapperDownloader} doesn't need to fetch it),
+ * <li>the gradle distribution from {@value #DISTRIBUTION_URL_ENV}, installing it in the gradle
+ * user home exactly where the gradle wrapper would install the "official" distribution (under
+ * the hash of the {@code distributionUrl} from {@code gradle-wrapper.properties}), so that
+ * the wrapper considers it already installed.
+ * </ul>
+ *
+ * <p>Both URLs may contain a {@code ${gradleVersion}} placeholder, replaced with the version parsed
+ * from {@code distributionUrl} in {@code gradle-wrapper.properties}.
+ *
+ * <p>Ensure this class has no dependencies outside of standard java libraries as it's run directly
+ * from source. It only uses JDK 11 language features and APIs so that it runs on any reasonably
+ * recent JDK (17+ is required by {@link #checkVersion()}; the actual build requires a newer one).
+ */
+public class IntranetGradleSetup {
+ /**
+ * Copied to keep the class isolated from any other classes.
+ *
+ * @see "https://github.com/apache/lucene/issues/15399"
+ */
+ @Retention(RetentionPolicy.CLASS)
+ @Target({ElementType.CONSTRUCTOR, ElementType.FIELD, ElementType.METHOD, ElementType.TYPE})
+ private @interface SuppressForbidden {
+ /** A reason for suppressing should always be given. */
+ String reason();
+ }
+
+ public static final String DISTRIBUTION_URL_ENV = "LUCENE_GRADLE_DISTRIBUTION_URL";
+ public static final String WRAPPER_URL_ENV = "LUCENE_GRADLE_WRAPPER_URL";
+
+ private static final Pattern DISTRIBUTION_NAME =
+ Pattern.compile("gradle-(?<version>.+?)-(bin|all)\\.zip");
+ private static final Pattern PLACEHOLDER = Pattern.compile("\\$\\{([^}]*)\\}");
+ private static final boolean IS_WINDOWS =
+ System.getProperty("os.name").toLowerCase(Locale.ROOT).contains("windows");
+
+ private final Path projectDir;
+ private final Path gradleUserHome;
+
+ public static void main(String[] args) {
+ if (args.length < 1) {
+ System.err.println("Usage: java IntranetGradleSetup.java <project dir> [gradle arguments]");
+ System.exit(2);
+ }
+
+ try {
+ checkVersion();
+ new IntranetGradleSetup(Paths.get(args[0]), args).run();
+ } catch (Exception e) {
+ System.err.println("ERROR: " + e.getMessage());
+ System.exit(3);
+ }
+ }
+
+ public static void checkVersion() {
+ int major = Runtime.version().feature();
+ if (major < 17) {
+ throw new IllegalStateException("java version must be 17 or later, your version: " + major);
+ }
+ }
+
+ IntranetGradleSetup(Path projectDir, String[] gradleArgs) {
+ this.projectDir = projectDir.toAbsolutePath().normalize();
+
+ // Gradle user home: -g/--gradle-user-home, GRADLE_USER_HOME or ~/.gradle.
+ String userHome = null;
+ for (int i = 1; i < gradleArgs.length; i++) {
+ String arg = gradleArgs[i];
+ if ((arg.equals("-g") || arg.equals("--gradle-user-home")) && i + 1 < gradleArgs.length) {
+ userHome = gradleArgs[++i];
+ } else if (arg.startsWith("--gradle-user-home=")) {
+ userHome = arg.substring("--gradle-user-home=".length());
+ }
+ }
+ if (userHome == null) {
+ userHome = System.getenv("GRADLE_USER_HOME");
+ }
+ this.gradleUserHome =
+ userHome != null
+ ? Paths.get(userHome)
+ : Paths.get(System.getProperty("user.home"), ".gradle");
+ }
+
+ void run() throws Exception {
+ Path wrapperDir = projectDir.resolve("gradle").resolve("wrapper");
+ Path propertiesFile = wrapperDir.resolve("gradle-wrapper.properties");
+ if (!Files.exists(propertiesFile)) {
+ throw new IOException("Wrapper property file not found: " + propertiesFile);
+ }
+ Properties props = new Properties();
+ try (Reader reader = Files.newBufferedReader(propertiesFile, StandardCharsets.UTF_8)) {
+ props.load(reader);
+ }
+ String distributionUrl = props.getProperty("distributionUrl");
+ if (distributionUrl == null) {
+ throw new IOException("No 'distributionUrl' in " + propertiesFile);
+ }
+ Matcher m =
+ DISTRIBUTION_NAME.matcher(distributionUrl.replaceAll("\\?.*", "").replaceAll(".*/", ""));
+ if (!m.matches()) {
+ throw new IOException(
+ "Could not parse the gradle version from distributionUrl in "
+ + propertiesFile
+ + ": "
+ + distributionUrl);
+ }
+ String gradleVersion = m.group("version");
+ int timeout = Integer.parseInt(props.getProperty("networkTimeout", "10000").trim());
+
+ String wrapperUrl = expand(System.getenv(WRAPPER_URL_ENV), gradleVersion);
+ String mirrorDistributionUrl = expand(System.getenv(DISTRIBUTION_URL_ENV), gradleVersion);
+
+ setupWrapperJar(wrapperDir, wrapperUrl, timeout);
+ if (mirrorDistributionUrl != null) {
+ setupDistribution(props, distributionUrl, mirrorDistributionUrl, gradleVersion, timeout);
+ }
+ }
+
+ /** Replaces {@code ${gradleVersion}}; null/blank input yields null. */
+ private static String expand(String template, String gradleVersion) throws IOException {
+ if (template == null || template.trim().isEmpty()) {
+ return null;
+ }
+ Matcher m = PLACEHOLDER.matcher(template.trim());
+ StringBuilder sb = new StringBuilder();
+ while (m.find()) {
+ if (!m.group(1).equals("gradleVersion")) {
+ throw new IOException(
+ "Unknown placeholder "
+ + m.group()
+ + " (only ${gradleVersion} is supported) in: "
+ + template);
+ }
+ m.appendReplacement(sb, Matcher.quoteReplacement(gradleVersion));
+ }
+ return m.appendTail(sb).toString();
+ }
+
+ /**
+ * Makes sure gradle/wrapper/gradle-wrapper.jar is present and matches gradle-wrapper.jar.sha256,
+ * downloading it from the mirror if needed (otherwise WrapperDownloader would try GitHub).
+ */
+ private void setupWrapperJar(Path wrapperDir, String wrapperUrl, int timeout) throws Exception {
+ Path jar = wrapperDir.resolve("gradle-wrapper.jar");
+ Path checksumFile = wrapperDir.resolve("gradle-wrapper.jar.sha256");
+ String expected = null;
+ for (String line : Files.readAllLines(checksumFile, StandardCharsets.UTF_8)) {
+ // sha256sum format: "<checksum> *gradle-wrapper.jar" ('*' marks binary mode).
+ String[] parts = line.trim().split("\\s+");
+ if (parts.length == 2 && parts[1].replaceFirst("^\\*", "").equals("gradle-wrapper.jar")) {
+ expected = parts[0];
+ }
+ }
+ if (expected == null) {
+ throw new IOException("No checksum for gradle-wrapper.jar in " + checksumFile);
+ }
+
+ if (Files.exists(jar) && sha256(jar).equalsIgnoreCase(expected)) {
+ return;
+ }
+ if (wrapperUrl == null) {
+ log(
+ "WARNING: "
+ + jar
+ + " is missing or does not match "
+ + checksumFile
+ + " and "
+ + WRAPPER_URL_ENV
+ + " is not set; WrapperDownloader will try to fetch it from GitHub.");
+ return;
+ }
+
+ log("Downloading gradle-wrapper.jar from " + wrapperUrl);
+ Path temp = Files.createTempFile(wrapperDir, ".gradle-wrapper", ".tmp");
+ try {
+ download(new URI(wrapperUrl), temp, timeout);
+ String actual = sha256(temp);
+ if (!actual.equalsIgnoreCase(expected)) {
+ throw new IOException(
+ "The gradle-wrapper.jar downloaded from "
+ + wrapperUrl
+ + " does not match "
+ + checksumFile
+ + " (expected: "
+ + expected
+ + ", actual: "
+ + actual
+ + ").");
+ }
+ Files.move(temp, jar, StandardCopyOption.REPLACE_EXISTING);
+ } finally {
+ Files.deleteIfExists(temp);
+ }
+ }
+
+ /**
+ * Downloads the distribution from the mirror and installs it where the gradle wrapper would
+ * install the official one: {@code <base>/<path>/<name>/<md5 of the official url as base36>/}.
+ */
+ private void setupDistribution(
+ Properties props, String distributionUrl, String mirrorUrl, String gradleVersion, int timeout)
+ throws Exception {
+ String expectedSha256 = props.getProperty("distributionSha256Sum");
+ if (expectedSha256 == null || expectedSha256.trim().isEmpty()) {
+ throw new IOException(
+ "No 'distributionSha256Sum' in gradle-wrapper.properties; refusing to install a gradle"
+ + " distribution without checksum verification.");
+ }
+ expectedSha256 = expectedSha256.trim();
+
+ URI official = new URI(distributionUrl);
+ String zipName = official.getPath().replaceAll(".*/", "");
+ String distName = zipName.replaceAll("\\.[^.]*$", "");
+ String hash =
+ new BigInteger(
+ 1,
+ MessageDigest.getInstance("MD5")
Review Comment:
## CodeQL / Use of a potentially broken or risky cryptographic algorithm
Cryptographic algorithm [MD5](1) may not be secure. Consider using a different algorithm.
[Show more details](https://github.com/apache/lucene/security/code-scanning/268)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5455452575
> I've sorta know someone at Gradle who works with the ASF and I crafted an email to make a case that this matter needs attention/priority it's hasn't been getting.
Thanks! I definitely think this is something of value.
> Neo takes actual gradle wrapper source, which is why. I prefer "lean" but if we don't maintain it then I'm indifferent.
I prefer the burden of maintaining that 500 lines of code I've reviewed than reviewing updates to neo, to be honest... I'm not against it but let's wait and see what evolves - maybe gradle folks will add something of their own, maybe neo get more traction in the community.
For now, I suggest to either accept this patch as is or reshape the code to use a (property) file in the user's home directory instead of env variables. I don't quite see the value of their regexp replacements - I think an URL with those replaceable gradleVersion is more consistent with what gradle wrapper does and more intuitive and I'd stick to it. The remaining logic seems very similar and there's not much to borrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5404662185
> If you live behind a corporate firewall then it's fairly easy to create a directory structure for each gradle version needed + the corresponding wrapper (what I tried to show in the description of this patch). This is a one-time effort, the directory structure stays the same on disk and you can keep the env pointer URLs constant (they can use the replaceable gradleVersion part).
I'd rather not do obscure local machine stuff (that I may forget how to do) _every time Lucene/Solr bumps the Gradle version_. It's annoying. Keeping an edited gradle-wrapper.properties with a corporate distributionUrl hasn't been too annoying to maintain. It's at least a known/documented thing. But _also_ aligning the gradle wrapper has felt like a needless chore. The wrapper is not paired with the distribution requiring version alignment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5388442768
I've added a separate env variable - LUCENE_GRADLE_VERIFY_CHECKSUMS=false to ignore all checksums... Take a look at commit 47ac309. What I stated in my previous comment still holds though - I don't think it's a good thing to ignore them. I think they should be verified and the correct (expected) gradle-wrapper.jar should be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add custom URL patterns for downloading gradle distribution and gradle wrapper [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #16543:
URL: https://github.com/apache/lucene/pull/16543#issuecomment-5591099237
I've subscribed to the gradle issue as well. If they accept it or do something about it, we'll move on to their standard solution and revert this hack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org