You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "rmannibucau (via GitHub)" <gi...@apache.org> on 2026/09/25 12:05:37 UTC
[I] [Feature Request]: publish hop distro on sdkman (hop)
rmannibucau opened a new issue, #8608:
URL: https://github.com/apache/hop/issues/8608
### What would you like to happen?
I'd like to get hop using `sdk install hop 2.19.0` and manage version with sdk man but today it is not published.
I did the PR (https://github.com/sdkman/sdkman-db-migrations/pull/799) to register HOP but I can't get credentials to do it since I'm not commiter on the project.
Can you allow me there to do it or take over this responsability please?
(cc @marc0der)
### Issue Priority
Priority: 3
### Issue Component
Component: Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: publish hop distro on sdkman (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #8608:
URL: https://github.com/apache/hop/issues/8608#issuecomment-5854478857
Thanks you for the reply @marc0der, currently releases are done by the release manager which is an individual and it is not always the same individual. They sign the release with a personal key. This then gets vetted. Pushing to sdkman does not work in this process.
I know the infra team is working on a central trusted building system I will reach out to them to see how this could become part of that. As far as I understood all releases will be build using keys of the foundation, if this means that you can have a single key from the foundation that can be used to add all of its projects that sounds like a win-win.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: publish hop distro on sdkman (hop)
Posted by "rmannibucau (via GitHub)" <gi...@apache.org>.
rmannibucau commented on issue #8608:
URL: https://github.com/apache/hop/issues/8608#issuecomment-5865092889
@jdaugherty 👍 , this is more or less the feedback I got from infra in the night. So I guess it can be as simple as having a hook on a tag on github and ensuring the tag is pushed once the binaries are published on archive or alike (and tag must be pushed after the vote anyway on the apache repo so not crazy).
@hansva does it sound doable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: publish hop distro on sdkman (hop)
Posted by "rmannibucau (via GitHub)" <gi...@apache.org>.
rmannibucau commented on issue #8608:
URL: https://github.com/apache/hop/issues/8608#issuecomment-5832626222
@hansva maven does it "outside" the asf (but I got a mail that I was not a committer so not possible even if it is the case for apache maven), not sure if anything is ASF owned. Basically it is a curl after the release or a cronjob based on archive to sync it automatically (GH action?. Technically it is just https://sdkman.io/vendors/ .
typically something like that in a scheduled action can be sufficient (not tested but you get the idea):
```
import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.security.GeneralSecurityException;
import java.security.MessageDigest;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Base64;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.regex.Pattern;
import javax.crypto.Cipher;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import javax.xml.parsers.DocumentBuilderFactory;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.NodeList;
/**
* Publishes Apache Hop releases from the Apache archive to SDKMAN! (vendor API
* documented at https://sdkman.io/vendors/).
*
* <p>Each version directory of the archive (https://archive.apache.org/dist/hop/)
* contains the archive to publish {@code apache-hop-client-<version>.zip}; the SDKMAN
* version is derived from the <b>zip name</b> (e.g. {@code apache-hop-client-1.0.0-incubating.zip}
* -> {@code 1.0.0-incubating}) and the official {@code .sha512} sidecar is forwarded as
* the SHA-512 checksum of the release. Versions are processed in natural ascending
* order ({@code 2.9.0} before {@code 2.18.0}).
*
* <p>The tool is <b>resumable</b>: each successfully published version is added to the
* local state file (--state, one version per line, written after each 2xx),
* and versions already visible on SDKMAN
* ({@code candidates/<candidate>/linux/versions/all}) are ignored. A failure in
* the middle of the run can therefore simply be re-run: only the missing versions
* are republished (the vendor POST is itself idempotent).
*
* <p>Credentials are read from {@code ~/.m2/settings.xml} (a {@code <server>} entry
* whose id equals --server-id; {@code <username>} = Consumer-Key, {@code <password>} =
* Consumer-Token). A password encrypted by Maven ({@code {...}}) is decrypted
* with the master of {@code ~/.m2/settings-security.xml}, exactly as Maven does
* (port of the plexus-cipher format: EVP-style SHA-256 derivation, AES/CBC,
* master passphrase = {@code "settings.security"}). As a fallback, the environment
* variables {@code SDKMAN_CONSUMER_KEY} / {@code SDKMAN_CONSUMER_TOKEN}.
*
* <p>Usage: {@code java SdkManHopSync.java [options]} (source-file mode, Java 25+),
* see {@code --help}. Exit code: 0 = ok, 1 = failure (can be re-run),
* 2 = usage or configuration error.
*/
public final class SdkManHopSync {
private static final String DEFAULT_ARCHIVE_URL = "https://archive.apache.org/dist/hop/";
private static final String DEFAULT_CANDIDATE = "hop";
private static final String VENDOR_API = "https://vendors.sdkman.io";
private static final String CANDIDATES_API = "https://api.sdkman.io/2";
private static final String RELEASE_PATH = "/release";
private static final String DEFAULT_PATH = "/default";
private static final String STATE_PLATFORM = "linux";
private static final String MASTER_PASSPHRASE = "settings.security";
private static final String USER_AGENT = "SdkManHopSync/1.0";
private static final String ZIP_PREFIX = "apache-hop-client-";
private static final String ZIP_SUFFIX = ".zip";
private static final int MAX_ATTEMPTS = 3;
private static final Pattern HREF = Pattern.compile("<a\\s+href=\"([^\"]+)\"", Pattern.CASE_INSENSITIVE);
private static final Pattern VERSION = Pattern.compile("[0-9][A-Za-z0-9._-]*");
private static final HttpClient HTTP = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(30))
.followRedirects(HttpClient.Redirect.NORMAL)
.build();
private SdkManHopSync() {
// utility class, no instance
}
public static void main(String[] args) {
int code = run(args);
if (code != 0) {
System.exit(code);
}
}
private static int run(String[] args) {
Config config;
try {
config = Config.parse(args);
} catch (UsageException e) {
System.err.println("error: " + e.getMessage());
System.err.println("run with --help for usage");
return 2;
}
if (config.help()) {
printHelp();
return 0;
}
try {
if (config.verifyOnly()) {
Credentials credentials = credentials(config);
System.out.println("credentials OK: consumer key " + mask(credentials.key())
+ ", consumer token " + mask(credentials.token())
+ " (from " + credentials.source() + ")");
return 0;
}
if (config.debug()) {
System.out.println("archive: " + config.archiveUrl());
System.out.println("candidate: " + config.candidate());
System.out.println("vendor API: " + config.vendorApiUrl());
System.out.println("candidates API: " + config.candidatesApiUrl());
System.out.println("settings: " + config.settings());
System.out.println("security settings: " + config.securitySettings());
System.out.println("state file: " + config.stateFile());
System.out.println("platform: " + STATE_PLATFORM);
}
if (!candidateRegistered(config)) {
System.out.println("warn: candidate '" + config.candidate() + "' is not registered on SDKMAN"
+ " (missing from " + config.candidatesApiUrl() + "/candidates/all).");
System.out.println(" Complete the vendor onboarding first:"
+ " https://github.com/sdkman/sdkman-cli/wiki/Vendor-onboarding-process");
if (!config.dryRun()) {
throw new SyncException("candidate '" + config.candidate() + "' is not registered on SDKMAN;"
+ " contact info@sdkman.io to onboard the candidate and get your"
+ " Consumer-Key/Consumer-Token, then re-run");
}
}
List<HopRelease> releases = discover(config);
if (releases.isEmpty()) {
System.out.println("no releases found under " + config.archiveUrl());
return 0;
}
Set<String> local = loadPublished(config);
Set<String> remote = remotePublished(config);
List<HopRelease> pending = new ArrayList<>();
for (HopRelease release : releases) {
if (!local.contains(release.version()) && !remote.contains(release.version())) {
pending.add(release);
}
}
if (config.dryRun()) {
System.out.println();
System.out.println("DRY-RUN - " + pending.size() + " release(s) would be published:");
for (HopRelease release : pending) {
System.out.println(" " + release.version() + " " + release.zipUrl()
+ (release.sha512() == null ? " (no sha512 checksum)" : ""));
}
System.out.println("skipped: " + (releases.size() - pending.size()) + " (already published or recorded)");
if (config.setDefault() != null) {
if (releases.stream().noneMatch(r -> r.version().equals(config.setDefault()))) {
System.out.println("warn: --set-default version '" + config.setDefault()
+ "' is not among the discovered releases");
} else {
System.out.println("would set default for " + config.candidate() + " " + config.setDefault());
}
}
return 0;
}
Credentials credentials = credentials(config);
int published = 0;
int skipped = 0;
for (HopRelease release : releases) {
if (local.contains(release.version())) {
skipped++;
continue;
}
if (remote.contains(release.version())) {
skipped++;
System.out.println("skip: " + release.version() + " already published on SDKMAN (recording locally)");
recordPublished(config, release.version());
continue;
}
publish(config, credentials, release);
recordPublished(config, release.version());
published++;
}
if (config.setDefault() != null) {
if (releases.stream().noneMatch(r -> r.version().equals(config.setDefault()))) {
throw new SyncException("--set-default version '" + config.setDefault()
+ "' is not among the discovered releases");
}
setDefault(config, credentials, config.setDefault());
System.out.println("default set: " + config.candidate() + " " + config.setDefault());
}
System.out.println();
System.out.println("done: " + published + " published, " + skipped + " skipped,"
+ " " + releases.size() + " release(s) in the archive");
return 0;
} catch (SyncException e) {
System.err.println();
System.err.println("error: " + e.getMessage());
System.err.println("the run is resumable: fix the cause if any and re-run;"
+ " versions already published are skipped");
return 1;
}
}
private static void printHelp() {
System.out.println("""
Usage: java SdkManHopSync.java [options]
Publishes Apache Hop releases from the Apache distribution archive to SDKMAN!.
Each version directory of the archive must contain an
apache-hop-client-<version>.zip (the SDKMAN version is derived from the zip
filename, e.g. apache-hop-client-1.0.0-incubating.zip -> 1.0.0-incubating);
the matching .sha512 sidecar is sent as the SHA-512 checksum of the release.
Options:
--archive-url <url> archive base URL (default: https://archive.apache.org/dist/hop/)
--candidate <name> SDKMAN candidate (default: hop)
--server-id <id> id of the <server> holding the SDKMAN credentials in the
Maven settings (default: sdkman); <username> is the
Consumer-Key and <password> the Consumer-Token. A Maven
encrypted <password> ({...}) is decrypted with the master
of --security-settings, exactly like Maven does.
--settings <path> Maven settings file (default: ~/.m2/settings.xml)
--security-settings <path> Maven settings-security file (default: ~/.m2/settings-security.xml)
--state <path> local file recording published versions, one per line
(default: ~/.cache/sdkman-hop-sync/published.txt)
--set-default <version> after the release, set this version as the SDKMAN default
--master-passphrase <pwd> passphrase to decrypt the <master> of settings-security
when it is itself Maven-encrypted (only needed if the
master was created with -Dsettings.security; Maven would
prompt for it interactively)
--vendor-api-url <url> SDKMAN vendor API base URL (default: https://vendors.sdkman.io)
--candidates-api-url <url> SDKMAN public API base URL (default: https://api.sdkman.io/2)
--dry-run discover releases and print what would be published; no
call to the vendor API, no credentials needed
--verify-settings resolve and check the credentials, print a masked summary
--debug print the resolved configuration and vendor API responses
--help show this help
Environment fallback: when no <server> matches --server-id, the
SDKMAN_CONSUMER_KEY and SDKMAN_CONSUMER_TOKEN environment variables are used.
Example settings.xml server entry:
<server>
<id>sdkman</id>
<username>CONSUMER_KEY</username>
<password>{ENCRYPTED_CONSUMER_TOKEN}</password>
</server>
Behaviour:
- versions are processed in natural ascending order (2.9.0 before 2.18.0)
- a version already recorded in --state or already visible on SDKMAN
(candidates/<candidate>/linux/versions/all) is skipped; the state file is
updated right after each successful release, so a failed run can simply
be re-run and only the missing versions are published (the vendor POST
itself is idempotent)
- any rejected request (4xx) or persistent failure aborts the run with a
clear message; retries happen on 429/5xx and network errors
Exit codes: 0 = ok, 1 = failure (safe to re-run), 2 = usage error.
""");
}
// ---------------------------------------------------------------------
// Configuration
// ---------------------------------------------------------------------
private record Config(
String archiveUrl,
String candidate,
String serverId,
String vendorApiUrl,
String candidatesApiUrl,
Path settings,
Path securitySettings,
Path stateFile,
String setDefault,
String masterPassphrase,
boolean dryRun,
boolean verifyOnly,
boolean debug,
boolean help) {
static Config parse(String... args) throws UsageException {
String archiveUrl = DEFAULT_ARCHIVE_URL;
String candidate = DEFAULT_CANDIDATE;
String serverId = "sdkman";
String vendorApiUrl = VENDOR_API;
String candidatesApiUrl = CANDIDATES_API;
Path settings = Path.of(resolveHome("~/.m2/settings.xml"));
Path securitySettings = Path.of(resolveHome("~/.m2/settings-security.xml"));
Path stateFile = Path.of(resolveHome("~/.cache/sdkman-hop-sync/published.txt"));
String setDefault = null;
String masterPassphrase = null;
boolean dryRun = false;
boolean verifyOnly = false;
boolean debug = false;
boolean help = false;
for (int i = 0; i < args.length; i++) {
String arg = args[i];
switch (arg) {
case "--help" -> help = true;
case "--dry-run" -> dryRun = true;
case "--verify-settings" -> verifyOnly = true;
case "--debug" -> debug = true;
case "--archive-url" -> archiveUrl = value(args, ++i, arg);
case "--candidate" -> candidate = value(args, ++i, arg);
case "--server-id" -> serverId = value(args, ++i, arg);
case "--vendor-api-url" -> vendorApiUrl = stripTrailingSlash(value(args, ++i, arg));
case "--candidates-api-url" -> candidatesApiUrl = stripTrailingSlash(value(args, ++i, arg));
case "--settings" -> settings = Path.of(resolveHome(value(args, ++i, arg)));
case "--security-settings" -> securitySettings = Path.of(resolveHome(value(args, ++i, arg)));
case "--state" -> stateFile = Path.of(resolveHome(value(args, ++i, arg)));
case "--set-default" -> setDefault = value(args, ++i, arg);
case "--master-passphrase" -> masterPassphrase = value(args, ++i, arg);
default -> throw new UsageException("unknown option: " + arg);
}
}
return new Config(archiveUrl, candidate, serverId, vendorApiUrl, candidatesApiUrl,
settings, securitySettings, stateFile, setDefault, masterPassphrase,
dryRun, verifyOnly, debug, help);
}
private static String stripTrailingSlash(String url) {
return url.endsWith("/") ? url.substring(0, url.length() - 1) : url;
}
private static String value(String[] args, int index, String option) throws UsageException {
if (index >= args.length) {
throw new UsageException("missing value for " + option);
}
return args[index];
}
}
private record Credentials(String key, String token, String source) {
}
private record HopRelease(String version, String zipUrl, String sha512) {
}
private static final class UsageException extends Exception {
private UsageException(String message) {
super(message);
}
}
private static final class SyncException extends Exception {
private SyncException(String message) {
super(message);
}
}
// ---------------------------------------------------------------------
// Credentials (settings.xml + settings-security.xml, decryption
// identical to Maven / plexus-cipher)
// ---------------------------------------------------------------------
private static Credentials credentials(Config config) throws SyncException {
String envKey = System.getenv("SDKMAN_CONSUMER_KEY");
String envToken = System.getenv("SDKMAN_CONSUMER_TOKEN");
if (envKey != null || envToken != null) {
if (envKey == null || envToken == null || envKey.isBlank() || envToken.isBlank()) {
throw new SyncException("SDKMAN_CONSUMER_KEY and SDKMAN_CONSUMER_TOKEN must both be set");
}
return new Credentials(envKey, envToken, "environment");
}
if (!Files.isRegularFile(config.settings())) {
throw new SyncException("Maven settings file not found: " + config.settings()
+ " (use --settings, or set SDKMAN_CONSUMER_KEY/SDKMAN_CONSUMER_TOKEN)");
}
Document settingsDoc = parseXml(config.settings(), "settings");
NodeList servers = settingsDoc.getElementsByTagName("server");
for (int i = 0; i < servers.getLength(); i++) {
Element server = (Element) servers.item(i);
if (config.serverId().equals(childText(server, "id"))) {
String key = childText(server, "username");
String token = childText(server, "password");
if (key == null || token == null || key.isBlank() || token.isBlank()) {
throw new SyncException("server '" + config.serverId() + "' in " + config.settings()
+ " must define <username> (consumer key) and <password> (consumer token)");
}
return new Credentials(key, decryptIfNeeded(token, config),
"server '" + config.serverId() + "' (" + config.settings() + ")");
}
}
throw new SyncException("no server with id '" + config.serverId() + "' in " + config.settings()
+ " (add one, or set SDKMAN_CONSUMER_KEY/SDKMAN_CONSUMER_TOKEN)");
}
private static String decryptIfNeeded(String value, Config config) throws SyncException {
if (!looksEncrypted(value)) {
return value;
}
String master = loadMaster(config);
try {
return decryptPlexus(unDecorate(value), master);
} catch (GeneralSecurityException e) {
throw new SyncException("cannot decrypt the password of server '" + config.serverId()
+ "' with the master of " + config.securitySettings() + ": " + e.getMessage());
}
}
private static String loadMaster(Config config) throws SyncException {
if (!Files.isRegularFile(config.securitySettings())) {
throw new SyncException("the password of server '" + config.serverId() + "' is Maven-encrypted but "
+ config.securitySettings() + " is missing (create it with: mvn --encrypt-master-password)");
}
Document doc = parseXml(config.securitySettings(), "settings-security");
String master = childText(doc.getDocumentElement(), "master");
if (master == null || master.isBlank()) {
throw new SyncException("no <master> found in " + config.securitySettings());
}
if (!looksEncrypted(master)) {
return master;
}
if (config.masterPassphrase() != null) {
try {
return decryptPlexus(unDecorate(master), config.masterPassphrase());
} catch (GeneralSecurityException e) {
throw new SyncException("cannot decrypt <master> of " + config.securitySettings()
+ " with --master-passphrase: " + e.getMessage());
}
}
try {
return decryptPlexus(unDecorate(master), MASTER_PASSPHRASE);
} catch (GeneralSecurityException e) {
throw new SyncException("cannot decrypt <master> of " + config.securitySettings() + ": " + e.getMessage()
+ " (Maven would prompt for the master passphrase interactively; either pass it with"
+ " --master-passphrase or re-create it with: mvn --encrypt-master-password)");
}
}
private static boolean looksEncrypted(String value) {
return value.length() > 2 && value.startsWith("{") && value.endsWith("}");
}
private static String unDecorate(String value) {
return value.substring(1, value.length() - 1);
}
/**
* Faithful port of the plexus-cipher format (PBECipher from plexus-cipher 2.0,
* used by Maven for settings.xml):
* {@code base64(salt(8) || padLen(1) || ciphertext)}, AES-128 key + IV derived
* from SHA-256(password || salt[0..8)) the EVP_BytesToKey way.
*/
static String decryptPlexus(String base64Value, String passphrase) throws GeneralSecurityException {
byte[] all = Base64.getDecoder().decode(base64Value.trim());
if (all.length < 16) {
throw new GeneralSecurityException("encrypted payload too short");
}
byte[] salt = Arrays.copyOfRange(all, 0, 8);
int padLen = all[8] & 0xFF;
int payloadLength = all.length - 9 - padLen;
if (payloadLength <= 0) {
throw new GeneralSecurityException("invalid encrypted payload");
}
byte[] encrypted = Arrays.copyOfRange(all, 9, 9 + payloadLength);
byte[] keyAndIv = deriveKeyAndIv(passphrase.getBytes(StandardCharsets.UTF_8), salt);
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
cipher.init(Cipher.DECRYPT_MODE, new SecretKeySpec(keyAndIv, 0, 16, "AES"),
new IvParameterSpec(keyAndIv, 16, 16));
return new String(cipher.doFinal(encrypted), StandardCharsets.UTF_8);
}
private static byte[] deriveKeyAndIv(byte[] password, byte[] salt) throws GeneralSecurityException {
byte[] keyAndIv = new byte[32];
int position = 0;
byte[] previous = null;
while (position < keyAndIv.length) {
MessageDigest digester = MessageDigest.getInstance("SHA-256");
if (previous != null) {
digester.update(previous);
}
digester.update(password);
digester.update(salt, 0, 8);
previous = digester.digest();
int copied = Math.min(previous.length, keyAndIv.length - position);
System.arraycopy(previous, 0, keyAndIv, position, copied);
position += copied;
}
return keyAndIv;
}
private static Document parseXml(Path file, String label) throws SyncException {
try {
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
factory.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true);
factory.setFeature("http://xml.org/sax/features/external-general-entities", false);
factory.setFeature("http://xml.org/sax/features/external-parameter-entities", false);
factory.setXIncludeAware(false);
factory.setExpandEntityReferences(false);
return factory.newDocumentBuilder().parse(file.toFile());
} catch (Exception e) {
throw new SyncException("cannot parse " + label + " file " + file + ": " + e.getMessage());
}
}
private static String childText(Element parent, String name) {
NodeList nodes = parent.getElementsByTagName(name);
if (nodes.getLength() == 0) {
return null;
}
String text = nodes.item(0).getTextContent();
return text == null ? null : text.trim();
}
// ---------------------------------------------------------------------
// Published state (local file + public SDKMAN API)
// ---------------------------------------------------------------------
private static Set<String> loadPublished(Config config) throws SyncException {
if (!Files.isRegularFile(config.stateFile())) {
return Set.of();
}
try {
Set<String> versions = new HashSet<>();
for (String line : Files.readAllLines(config.stateFile(), StandardCharsets.UTF_8)) {
String version = line.trim();
if (!version.isEmpty() && !version.startsWith("#")) {
versions.add(version);
}
}
return versions;
} catch (IOException e) {
throw new SyncException("cannot read state file " + config.stateFile() + ": " + e.getMessage());
}
}
private static void recordPublished(Config config, String version) throws SyncException {
try {
Path parent = config.stateFile().getParent();
if (parent != null) {
Files.createDirectories(parent);
}
Files.writeString(config.stateFile(), version + System.lineSeparator(),
StandardCharsets.UTF_8, StandardOpenOption.CREATE, StandardOpenOption.APPEND);
} catch (IOException e) {
throw new SyncException("cannot write state file " + config.stateFile() + ": " + e.getMessage());
}
}
private static Set<String> remotePublished(Config config) {
Set<String> versions = new HashSet<>();
try {
String csv = getBody(config.candidatesApiUrl() + "/candidates/" + config.candidate() + "/" + STATE_PLATFORM
+ "/versions/all");
for (String version : csv.split(",")) {
if (!version.isBlank()) {
versions.add(version.trim());
}
}
} catch (SyncException e) {
System.out.println("warn: cannot query already published versions from SDKMAN (" + e.getMessage()
+ "), relying on the local state file only");
}
return versions;
}
private static boolean candidateRegistered(Config config) {
try {
String list = getBody(config.candidatesApiUrl() + "/candidates/all");
return Arrays.stream(list.split(",")).map(String::trim).anyMatch(config.candidate()::equals);
} catch (SyncException e) {
System.out.println("warn: cannot check candidate registration on SDKMAN (" + e.getMessage()
+ "), assuming the candidate is registered");
return true;
}
}
// ---------------------------------------------------------------------
// Archive discovery
// ---------------------------------------------------------------------
private static List<HopRelease> discover(Config config) throws SyncException {
String base = config.archiveUrl().endsWith("/") ? config.archiveUrl() : config.archiveUrl() + "/";
String index = getBody(base);
Set<String> versionsSeen = new HashSet<>();
List<HopRelease> releases = new ArrayList<>();
for (String dir : hrefs(index).stream()
.filter(h -> h.endsWith("/"))
.filter(h -> !h.startsWith("/") && !h.startsWith("?") && !h.startsWith("http"))
.map(h -> h.substring(0, h.length() - 1))
.filter(h -> !h.isEmpty())
.sorted(SdkManHopSync::compareNatural)
.toList()) {
String dirUrl = base + dir + "/";
HttpResponse<String> page = get(dirUrl);
if (page.statusCode() != 200) {
System.out.println("warn: cannot list " + dirUrl + " (HTTP " + page.statusCode() + "), skipping");
continue;
}
String zip = hrefs(page.body()).stream()
.filter(f -> f.startsWith(ZIP_PREFIX) && f.endsWith(ZIP_SUFFIX))
.findFirst()
.orElse(null);
if (zip == null) {
System.out.println("warn: no " + ZIP_PREFIX + "<version>" + ZIP_SUFFIX + " in " + dirUrl + ", skipping");
continue;
}
String version = zip.substring(ZIP_PREFIX.length(), zip.length() - ZIP_SUFFIX.length());
if (!VERSION.matcher(version).matches()) {
System.out.println("warn: invalid version derived from " + zip + ", skipping");
continue;
}
if (!versionsSeen.add(version)) {
System.out.println("warn: duplicate version " + version + " (from " + dirUrl + "), keeping the first one");
continue;
}
String sha512 = null;
HttpResponse<String> shaResponse = get(dirUrl + zip + ".sha512");
if (shaResponse.statusCode() == 200) {
sha512 = parseSha512(shaResponse.body());
if (sha512 == null) {
System.out.println("warn: cannot parse the sha512 sidecar of " + zip
+ ", publishing without checksum");
}
} else {
System.out.println("warn: no sha512 sidecar for " + zip + " (HTTP " + shaResponse.statusCode()
+ "), publishing without checksum");
}
releases.add(new HopRelease(version, dirUrl + zip, sha512));
}
return releases;
}
private static List<String> hrefs(String html) {
List<String> out = new ArrayList<>();
var matcher = HREF.matcher(html);
while (matcher.find()) {
out.add(matcher.group(1));
}
return out;
}
private static String parseSha512(String content) {
String first = content.trim().split("\\s+")[0];
boolean hex = first.length() == 128
&& first.chars().allMatch(c -> (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F'));
return hex ? first.toLowerCase() : null;
}
/** Natural sort: numeric segments compare numerically (2.9.0 < 2.18.0). */
private static int compareNatural(String left, String right) {
int i = 0;
int j = 0;
while (i < left.length() && j < right.length()) {
char lc = left.charAt(i);
char rc = right.charAt(j);
if (Character.isDigit(lc) && Character.isDigit(rc)) {
int startLeft = i;
int startRight = j;
while (i < left.length() && Character.isDigit(left.charAt(i))) {
i++;
}
while (j < right.length() && Character.isDigit(right.charAt(j))) {
j++;
}
String numberLeft = stripZeros(left.substring(startLeft, i));
String numberRight = stripZeros(right.substring(startRight, j));
int compared = numberLeft.length() != numberRight.length()
? Integer.compare(numberLeft.length(), numberRight.length())
: numberLeft.compareTo(numberRight);
if (compared != 0) {
return compared;
}
} else {
int compared = Character.compare(lc, rc);
if (compared != 0) {
return compared;
}
i++;
j++;
}
}
return Integer.compare(left.length(), right.length());
}
private static String stripZeros(String number) {
int firstNonZero = 0;
while (firstNonZero < number.length() - 1 && number.charAt(firstNonZero) == '0') {
firstNonZero++;
}
return number.substring(firstNonZero);
}
// ---------------------------------------------------------------------
// Publication sur l'API vendor SDKMAN
// ---------------------------------------------------------------------
private static void publish(Config config, Credentials credentials, HopRelease release) throws SyncException {
System.out.println("publishing " + config.candidate() + " " + release.version() + " from " + release.zipUrl());
String response = sendJson("POST", config.vendorApiUrl() + RELEASE_PATH, credentials,
json(config.candidate(), release.version(), release.zipUrl(), release.sha512()));
if (config.debug()) {
System.out.println(" vendor API response: " + trimTo(response, 500));
}
}
private static void setDefault(Config config, Credentials credentials, String version) throws SyncException {
String response = sendJson("PUT", config.vendorApiUrl() + DEFAULT_PATH, credentials,
json(config.candidate(), version, null, null));
if (config.debug()) {
System.out.println(" vendor API response: " + trimTo(response, 500));
}
}
private static String json(String candidate, String version, String url, String sha512) {
StringBuilder builder = new StringBuilder();
builder.append("{\"candidate\":\"").append(jsonEscape(candidate))
.append("\",\"version\":\"").append(jsonEscape(version)).append('"');
if (url != null) {
builder.append(",\"url\":\"").append(jsonEscape(url)).append('"');
}
if (sha512 != null) {
builder.append(",\"checksums\":{\"SHA-512\":\"").append(sha512).append("\"}");
}
return builder.append('}').toString();
}
private static String jsonEscape(String value) {
return value.replace("\\", "\\\\").replace("\"", "\\\"");
}
private static String sendJson(String method, String url, Credentials credentials, String json)
throws SyncException {
HttpRequest.Builder builder = HttpRequest.newBuilder(URI.create(url))
.timeout(Duration.ofMinutes(5))
.header("Consumer-Key", credentials.key())
.header("Consumer-Token", credentials.token())
.header("Content-Type", "application/json")
.header("Accept", "application/json")
.header("User-Agent", USER_AGENT);
HttpRequest request = switch (method) {
case "POST" -> builder.POST(HttpRequest.BodyPublishers.ofString(json)).build();
case "PUT" -> builder.PUT(HttpRequest.BodyPublishers.ofString(json)).build();
default -> throw new SyncException("unsupported method " + method);
};
SyncException last = null;
for (int attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
try {
HttpResponse<String> response = HTTP.send(request, HttpResponse.BodyHandlers.ofString());
int status = response.statusCode();
if (status >= 200 && status < 300) {
return response.body();
}
String detail = trimTo(response.body(), 400);
if (status == 429 || status >= 500) {
if (attempt < MAX_ATTEMPTS) {
System.out.println(" got HTTP " + status + " (attempt " + attempt + "/" + MAX_ATTEMPTS
+ "), retrying in " + backoffSeconds(attempt) + "s");
sleepMillis(backoffSeconds(attempt) * 1000L);
continue;
}
throw new SyncException("SDKMAN vendor API error after " + MAX_ATTEMPTS
+ " attempts (HTTP " + status + "): " + detail);
}
throw new SyncException("SDKMAN vendor API rejected the request (HTTP " + status + "): " + detail);
} catch (SyncException e) {
throw e;
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new SyncException("interrupted");
} catch (IOException e) {
last = new SyncException("HTTP " + method + " " + url + " failed: " + e.getMessage());
if (attempt < MAX_ATTEMPTS) {
sleepMillis(backoffSeconds(attempt) * 1000L);
}
}
}
throw last;
}
private static HttpResponse<String> get(String url) throws SyncException {
HttpRequest request = HttpRequest.newBuilder(URI.create(url))
.timeout(Duration.ofMinutes(2))
.header("User-Agent", USER_AGENT)
.GET()
.build();
SyncException last = null;
for (int attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
try {
HttpResponse<String> response = HTTP.send(request, HttpResponse.BodyHandlers.ofString());
int status = response.statusCode();
if (status == 429 || status >= 500) {
if (attempt < MAX_ATTEMPTS) {
sleepMillis(backoffSeconds(attempt) * 1000L);
continue;
}
}
return response;
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new SyncException("interrupted");
} catch (IOException e) {
last = new SyncException("GET " + url + " failed: " + e.getMessage());
if (attempt < MAX_ATTEMPTS) {
sleepMillis(backoffSeconds(attempt) * 1000L);
}
}
}
throw last;
}
private static String getBody(String url) throws SyncException {
HttpResponse<String> response = get(url);
if (response.statusCode() != 200) {
throw new SyncException("GET " + url + " returned HTTP " + response.statusCode());
}
return response.body();
}
// ---------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------
private static String mask(String value) {
if (value == null || value.isEmpty()) {
return "(empty)";
}
if (value.length() <= 6) {
return "****";
}
return value.substring(0, 3) + "..." + value.substring(value.length() - 3);
}
private static String trimTo(String value, int max) {
if (value == null) {
return "";
}
String trimmed = value.trim();
if (trimmed.length() <= max) {
return trimmed;
}
return trimmed.substring(0, max) + "...";
}
private static long backoffSeconds(int attempt) {
return attempt == 1 ? 5 : 15;
}
private static void sleepMillis(long milliseconds) throws SyncException {
try {
Thread.sleep(milliseconds);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new SyncException("interrupted");
}
}
private static String resolveHome(String path) {
return path.startsWith("~/") ? System.getProperty("user.home") + path.substring(1) : path;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: publish hop distro on sdkman (hop)
Posted by "marc0der (via GitHub)" <gi...@apache.org>.
marc0der commented on issue #8608:
URL: https://github.com/apache/hop/issues/8608#issuecomment-6047967135
We will be releasing an updated version of our GitHub release action shortly that will make releasing from GitHub workflows even simpler. All you need is a set of credentials that we will happily issue to the project.
Thanks @jdaugherty, and I think Apache Groovy is another project that releases via our API cc: @paulk-asert
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: publish hop distro on sdkman (hop)
Posted by "marc0der (via GitHub)" <gi...@apache.org>.
marc0der commented on issue #8608:
URL: https://github.com/apache/hop/issues/8608#issuecomment-5850489860
@hansva I'd prefer if the credentials were issued to an official email address associated with hop, preferably to an email associated with a build account.
Ideally it should be triggered as part of your build pipeline. I would not recommend using code like the above, but rather that you use our GitHub action. Feel free to contact me directly if you would like to be onboarded or require assistance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: publish hop distro on sdkman (hop)
Posted by "rmannibucau (via GitHub)" <gi...@apache.org>.
rmannibucau commented on issue #8608:
URL: https://github.com/apache/hop/issues/8608#issuecomment-6035385522
Hey guys
sdkman process changed (they migrated away from mongo to PG) so now to register the candidate for hop we must send a mail to [info@sdkman.io](mailto:info@sdkman.io)
data from https://github.com/sdkman/sdkman-db-migrations/pull/799#issue-5579916098 can be reused, can you do it?
also any news on the process? do we get infra setting a secret on GH actions and a cron synchronization job or a post release step?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: publish hop distro on sdkman (hop)
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on issue #8608:
URL: https://github.com/apache/hop/issues/8608#issuecomment-5862595441
I noticed this mentioned in the ASF slack and thought I'd mention that Apache Grails is also on SDKman.
If there's interest, here's how our process works:
Our release process is performed via GitHub actions (which requires a reproducible build per ASF Security). SDKman is published via our post release steps (example [here](https://github.com/apache/grails-core/actions/runs/35689679691/job/107969029202)). As part of our release process, we duplicate our convenience binary to the GitHub release (we could easily point it at the ASF distro site now instead) and define the sdkman config as part of our [cli project](https://github.com/apache/grails-core/blob/b607f3deaca525ef41864a77c5076cc4159a0ad5/grails-forge/grails-cli/build.gradle#L245). The associated SDKman keys are stored in Github Secrets that are managed by our PMC. We gate the run of the SDKman publish on a Github environment - which can be restricted to run by approval of only certain users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: publish hop distro on sdkman (hop)
Posted by "marc0der (via GitHub)" <gi...@apache.org>.
marc0der commented on issue #8608:
URL: https://github.com/apache/hop/issues/8608#issuecomment-5856259391
Yep, that sounds excellent. Let me know how it goes and we'll get you sorted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: publish hop distro on sdkman (hop)
Posted by "rmannibucau (via GitHub)" <gi...@apache.org>.
rmannibucau commented on issue #8608:
URL: https://github.com/apache/hop/issues/8608#issuecomment-5857996428
@hansva I'm not sure it would be intended nor desired, at Apache the ownership is really by PMC so there should be a set of key per PMC (not strictly per project if you look at incubator or commons for ex) even if owned by Apache infra. Ultimately PMC must have access to a set of keys too since they are legal owners of the content.
Short term I guess the refresh job on github action with an infra secret can be a compromise.
But it is not very different than docker images - generally speaking, as soon as it is no more sources it is no more 100% required to be done at apache - where some people handle it outside apache.
@marc0der how is it managed by all projects on sdkman today - I know some are not aligned with your security rules for example and several are done by individual?
```
Apache ActiveMQ (Classic) (5.17.1) https://activemq.apache.org/
Ant (1.10.17) https://ant.apache.org/
CXF (3.2.5) https://cxf.apache.org/
Flink (1.19.0) https://flink.apache.org/
hadoop (3.4.1) https://hadoop.apache.org/
Apache JMeter (5.6.3) https://jmeter.apache.org/
Karaf (4.2.8) https://karaf.apache.org/
Maven (3.9.16) https://maven.apache.org/
Spark (4.1.1) https://spark.apache.org/
Apache Tomcat (11.0.22) https://tomcat.apache.org/
```
Since Apache are standardized (either using central or https://archive.apache.org/dist/ - and guess we can ask all projects to converge to central if it helps - including Apache Hop) any way there is a scheduled job synchronizing releases in sdkman instead of awaiting for a push? Can also make it smoother for everyone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: publish hop distro on sdkman (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #8608:
URL: https://github.com/apache/hop/issues/8608#issuecomment-5832155238
Is there an ASF precedent and procedure to do this?
It seems only one set of keys can get access and publish this, how would this impact our release process?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org