You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by GitBox <gi...@apache.org> on 2022/10/01 09:53:43 UTC
[GitHub] [maven-resolver] michael-o commented on a diff in pull request #199: [MRESOLVER-269] [MRESOLVER-275] Trusted checksums source and more compact format backed source
michael-o commented on code in PR #199:
URL: https://github.com/apache/maven-resolver/pull/199#discussion_r985076081
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/checksum/SparseFileTrustedChecksumsSource.java:
##########
@@ -0,0 +1,125 @@
+package org.eclipse.aether.internal.impl.checksum;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import javax.inject.Inject;
+import javax.inject.Named;
+import javax.inject.Singleton;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.eclipse.aether.RepositorySystemSession;
+import org.eclipse.aether.artifact.Artifact;
+import org.eclipse.aether.internal.impl.LocalPathComposer;
+import org.eclipse.aether.repository.ArtifactRepository;
+import org.eclipse.aether.spi.connector.checksum.ChecksumAlgorithmFactory;
+import org.eclipse.aether.spi.io.FileProcessor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Sparse file {@link FileTrustedChecksumsSourceSupport} implementation that use specified directory as base
+ * directory, where it expects artifacts checksums on standard Maven2 "local" layout. This implementation uses Artifact
+ * coordinates solely to form path from basedir, pretty much as Maven local repository does.
+ * <p>
+ * The source may be configured to be "origin aware", in that case it will factor in origin repository ID as well into
+ * base directory name (for example ".checksums/central/...").
Review Comment:
Call me stupid, but why don't we use the `LocalPathPrefixComposer` for this. It will behave identically to `EnhancedLocalRepositoryManager`?
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/checksum/FileTrustedChecksumsSourceSupport.java:
##########
@@ -0,0 +1,154 @@
+package org.eclipse.aether.internal.impl.checksum;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+
+import org.eclipse.aether.RepositorySystemSession;
+import org.eclipse.aether.artifact.Artifact;
+import org.eclipse.aether.repository.ArtifactRepository;
+import org.eclipse.aether.spi.checksums.TrustedChecksumsSource;
+import org.eclipse.aether.spi.connector.checksum.ChecksumAlgorithmFactory;
+import org.eclipse.aether.util.ConfigUtils;
+import org.eclipse.aether.util.DirectoryUtils;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Support class for implementing {@link TrustedChecksumsSource} backed by local filesystem. It implements basic support
+ * like basedir calculation, "enabled" flag and "originAware" flag.
+ * <p>
+ * The configuration keys supported:
+ * <ul>
+ * <li><pre>aether.trustedChecksumsSource.${name}.enabled</pre> (boolean) must be explicitly set to "true"
+ * to become enabled</li>
+ * <li><pre>aether.trustedChecksumsSource.${name}.basedir</pre> (string, path) directory from where implementation
+ * can use files. May be relative path (the is resolved from local repository basedir) or absolute. If unset,
+ * default value is ".checksums" and is resolved from local repository basedir.</li>
+ * <li><pre>aether.trustedChecksumsSource.${name}.originAware</pre> (boolean) whether to make implementation
+ * "originAware", to factor in origin repository ID as well or not.</li>
+ * </ul>
+ * <p>
+ * This implementation ensures that implementations have "name" property, used in configuration properties above.
+ *
+ * @since TBD
+ */
+abstract class FileTrustedChecksumsSourceSupport
+ implements TrustedChecksumsSource
+{
+ private static final String CONFIG_PROP_PREFIX = "aether.trustedChecksumsSource.";
+
+ private static final String CONF_NAME_ENABLED = "enabled";
+
+ private static final String CONF_NAME_BASEDIR = "basedir";
+
+ private static final String CONF_NAME_ORIGIN_AWARE = "originAware";
+
+ /**
+ * Visible for testing.
+ */
+ static final String LOCAL_REPO_PREFIX = ".checksums";
Review Comment:
This should be rather `LOCAL_REPO_PREFIX_DIR` since it stands as a dir name .
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/checksum/CompactFileTrustedChecksumsSource.java:
##########
@@ -0,0 +1,162 @@
+package org.eclipse.aether.internal.impl.checksum;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import javax.inject.Named;
+import javax.inject.Singleton;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.eclipse.aether.RepositorySystemSession;
+import org.eclipse.aether.artifact.Artifact;
+import org.eclipse.aether.repository.ArtifactRepository;
+import org.eclipse.aether.spi.connector.checksum.ChecksumAlgorithmFactory;
+import org.eclipse.aether.util.artifact.ArtifactIdUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Compact file {@link FileTrustedChecksumsSourceSupport} implementation that use specified directory as base
+ * directory, where it expects a "summary" file named as "checksums.${checksumExt}" for each checksum algorithm, and
+ * file format is artifact ID and checksum separated by space per line. The format supports comments "#" (hash) and
+ * empty lines (both are ignored).
+ * <p>
+ * The source may be configured to be "origin aware", in that case it will factor in origin repository ID as well into
+ * file name (for example "checksums-central.sha1").
+ * <p>
+ * The checksums file once loaded are cached in session, so in-flight file changes during lifecycle of session are NOT
+ * noticed.
+ * <p>
+ * The name of this implementation is "file-compact".
+ *
+ * @see ArtifactIdUtils#toId(Artifact)
+ * @since TBD
+ */
+@Singleton
+@Named( CompactFileTrustedChecksumsSource.NAME )
+public final class CompactFileTrustedChecksumsSource
+ extends FileTrustedChecksumsSourceSupport
+{
+ public static final String NAME = "file-compact";
+
+ private static final String CHECKSUMS_FILE_PREFIX = "checksums";
+
+ private static final String CHECKSUMS_CACHE_KEY = CompactFileTrustedChecksumsSource.class.getName() + ".checksums";
+
+ private static final Logger LOGGER = LoggerFactory.getLogger( CompactFileTrustedChecksumsSource.class );
+
+ public CompactFileTrustedChecksumsSource()
+ {
+ super( NAME );
+ }
+
+ @SuppressWarnings( "unchecked" )
+ @Override
+ protected Map<String, String> performLookup( RepositorySystemSession session,
+ Path basedir,
+ Artifact artifact,
+ ArtifactRepository artifactRepository,
+ List<ChecksumAlgorithmFactory> checksumAlgorithmFactories )
+ {
+ final String prefix;
+ if ( isOriginAware( session ) )
+ {
+ prefix = CHECKSUMS_FILE_PREFIX + "-" + artifactRepository.getId() + ".";
+ }
+ else
+ {
+ prefix = CHECKSUMS_FILE_PREFIX + ".";
+ }
+
+ final ConcurrentHashMap<String, ConcurrentHashMap<String, String>> basedirProvidedChecksums =
+ (ConcurrentHashMap<String, ConcurrentHashMap<String, String>>) session.getData()
+ .computeIfAbsent( CHECKSUMS_CACHE_KEY, ConcurrentHashMap::new );
+
+ final HashMap<String, String> checksums = new HashMap<>();
+ for ( ChecksumAlgorithmFactory checksumAlgorithmFactory : checksumAlgorithmFactories )
+ {
+ ConcurrentHashMap<String, String> algorithmChecksums = basedirProvidedChecksums.computeIfAbsent(
+ checksumAlgorithmFactory.getName(),
+ algName -> loadProvidedChecksums(
+ basedir.resolve( prefix + checksumAlgorithmFactory.getFileExtension() )
+ )
+ );
+ String checksum = algorithmChecksums.get( ArtifactIdUtils.toId( artifact ) );
+ if ( checksum != null )
+ {
+ checksums.put( checksumAlgorithmFactory.getName(), checksum );
+ }
+ }
+ return checksums;
+ }
+
+ private ConcurrentHashMap<String, String> loadProvidedChecksums( Path checksumFile )
+ {
+ ConcurrentHashMap<String, String> result = new ConcurrentHashMap<>();
+ try
+ {
+ try ( BufferedReader reader = Files.newBufferedReader( checksumFile, StandardCharsets.UTF_8 ) )
+ {
+ LOGGER.debug( "Loading provided checksums file '{}'", checksumFile );
+ String line;
+ while ( ( line = reader.readLine() ) != null )
+ {
+ if ( !line.startsWith( "#" ) && !line.isEmpty() )
+ {
+ String[] parts = line.split( " ", 2 );
+ if ( parts.length == 2 )
+ {
+ String old = result.put( parts[0], parts[1] );
+ if ( old != null )
+ {
+ LOGGER.warn( "Checksums file '{}' contains duplicate checksums for artifact {}: "
+ + "old '{}' replaced by new '{}'", checksumFile, parts[0], old, parts[1] );
+ }
+ }
+ else
+ {
+ LOGGER.warn( "Checksums file '{}' ignored malformed line '{}'", checksumFile, line );
+ }
+ }
+ }
+ }
+ }
+ catch ( NoSuchFileException e )
+ {
+ // ignore, will return empty result
+ LOGGER.debug( "Checksums file '{}' not found", checksumFile );
Review Comment:
This is an antipattern according to Joshua Bloch. Test for existance instead of reyling on such exceptions.
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/checksum/SparseFileTrustedChecksumsSource.java:
##########
@@ -0,0 +1,125 @@
+package org.eclipse.aether.internal.impl.checksum;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import javax.inject.Inject;
+import javax.inject.Named;
+import javax.inject.Singleton;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.eclipse.aether.RepositorySystemSession;
+import org.eclipse.aether.artifact.Artifact;
+import org.eclipse.aether.internal.impl.LocalPathComposer;
+import org.eclipse.aether.repository.ArtifactRepository;
+import org.eclipse.aether.spi.connector.checksum.ChecksumAlgorithmFactory;
+import org.eclipse.aether.spi.io.FileProcessor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Sparse file {@link FileTrustedChecksumsSourceSupport} implementation that use specified directory as base
+ * directory, where it expects artifacts checksums on standard Maven2 "local" layout. This implementation uses Artifact
+ * coordinates solely to form path from basedir, pretty much as Maven local repository does.
+ * <p>
+ * The source may be configured to be "origin aware", in that case it will factor in origin repository ID as well into
+ * base directory name (for example ".checksums/central/...").
+ * <p>
+ * The name of this implementation is "file-sparse".
+ *
+ * @see LocalPathComposer
+ * @since TBD
+ */
+@Singleton
+@Named( SparseFileTrustedChecksumsSource.NAME )
+public final class SparseFileTrustedChecksumsSource
Review Comment:
I know what sparse means here and I think it is used incorrectly. Your understanding: A spare directory tree where only a limited amount of files exist for artifacts required. Meaning of sparse file: https://en.wikipedia.org/wiki/Sparse_file
I think the term is abused.
It should be `SparseDirectoryFileTrustedChecksumsSource` or `SparseDirectoryTrustedChecksumsSource`.
See impl of a sparse dir tree: https://svnbook.red-bean.com/en/1.7/svn.advanced.sparsedirs.html
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/checksum/CompactFileTrustedChecksumsSource.java:
##########
@@ -0,0 +1,162 @@
+package org.eclipse.aether.internal.impl.checksum;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import javax.inject.Named;
+import javax.inject.Singleton;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.eclipse.aether.RepositorySystemSession;
+import org.eclipse.aether.artifact.Artifact;
+import org.eclipse.aether.repository.ArtifactRepository;
+import org.eclipse.aether.spi.connector.checksum.ChecksumAlgorithmFactory;
+import org.eclipse.aether.util.artifact.ArtifactIdUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Compact file {@link FileTrustedChecksumsSourceSupport} implementation that use specified directory as base
+ * directory, where it expects a "summary" file named as "checksums.${checksumExt}" for each checksum algorithm, and
+ * file format is artifact ID and checksum separated by space per line. The format supports comments "#" (hash) and
Review Comment:
Stupid question: If you say yourself that it uses a summary file, why just not call it so? `SummaryFileTrustedChecksumsSource`, DI name `summary-file`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org