You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by GitBox <gi...@apache.org> on 2022/11/06 13:25:41 UTC
[GitHub] [maven-resolver] michael-o commented on a diff in pull request #213: [MRESOLVER-283] Shared executor service
michael-o commented on code in PR #213:
URL: https://github.com/apache/maven-resolver/pull/213#discussion_r1014828015
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/DefaultMetadataResolver.java:
##########
@@ -374,41 +391,36 @@ else if ( exception == null )
if ( !tasks.isEmpty() )
{
- int threads = ConfigUtils.getInteger( session, 4, CONFIG_PROP_THREADS );
- Executor executor = getExecutor( Math.min( tasks.size(), threads ) );
- try
+ RunnableErrorForwarder errorForwarder = new RunnableErrorForwarder();
+ ArrayList<Runnable> runnable = new ArrayList<>( tasks.size() );
Review Comment:
runnables
##########
maven-resolver-connector-basic/src/main/java/org/eclipse/aether/connector/basic/BasicRepositoryConnector.java:
##########
@@ -146,8 +150,10 @@
this.repository = repository;
this.fileProcessor = fileProcessor;
this.providedChecksumsSources = providedChecksumsSources;
+ this.resolverExecutor = resolverExecutorService.getResolverExecutor( session, RepositoryConnector.class,
+ ConfigUtils.getInteger( session, CONFIG_PROP_THREADS_DEFAULT, CONFIG_PROP_THREADS,
+ "maven.artifact.threads" ) );
Review Comment:
We should also deprecate this property since it does not reflect reality. MD is not artifacts, but this connector does both.
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/concurrency/DefaultResolverExecutorService.java:
##########
@@ -0,0 +1,92 @@
+package org.eclipse.aether.internal.impl.concurrency;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import javax.inject.Named;
+import javax.inject.Singleton;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.eclipse.aether.RepositorySystemSession;
+import org.eclipse.aether.spi.concurrency.ResolverExecutor;
+import org.eclipse.aether.spi.concurrency.ResolverExecutorService;
+import org.eclipse.aether.util.concurrency.WorkerThreadFactory;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Default implementation of {@link ResolverExecutor}.
+ * <p>
+ * This implementation uses {@link RepositorySystemSession#getData()} to store created {@link ExecutorService}
+ * instances. It creates instances that may be eventually garbage collected, so no explicit shutdown happens on
+ * them. When {@code maxThreads} parameter is 1 (accepted values are greater than zero), this implementation assumes
+ * caller wants "direct execution" (on caller thread) and creates {@link ResolverExecutor} instances accordingly.
+ */
+@Singleton
+@Named
+public final class DefaultResolverExecutorService implements ResolverExecutorService
+{
+ @Override
+ public ResolverExecutor getResolverExecutor( RepositorySystemSession session,
+ Class<?> service,
+ int maxThreads )
+ {
+ requireNonNull( session );
+ requireNonNull( service );
+ if ( maxThreads < 1 )
+ {
+ throw new IllegalArgumentException( "threads must be greater than zero" );
+ }
+
+ final ExecutorService executorService;
+ if ( maxThreads == 1 ) // direct
+ {
+ executorService = null;
+ }
+ else // shared && pooled
+ {
+ String key = DefaultResolverExecutorService.class.getName() + "." + service.getSimpleName();
+ executorService = (ExecutorService) session.getData()
+ .computeIfAbsent( key, () -> createExecutorService( service, maxThreads ) );
+ }
+ return new DefaultResolverExecutor( executorService );
+ }
+
+ /**
+ * Creates am {@link ExecutorService} that allows its core threads to die off in case of inactivity, and allows
+ * for proper garbage collection. This is important detail, as these instances are kept within session data, and
+ * currently there is no way to shut down them.
+ */
+ private ExecutorService createExecutorService( Class<?> service, int maxThreads )
+ {
+ ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(
+ maxThreads,
+ maxThreads,
+ 3L, TimeUnit.SECONDS,
+ new LinkedBlockingQueue<>(),
+ new WorkerThreadFactory( getClass().getSimpleName() + "-" + service.getSimpleName() + "-" )
+ );
+ threadPoolExecutor.allowCoreThreadTimeOut( true );
Review Comment:
Why is this necessary?
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/concurrency/DefaultResolverExecutorService.java:
##########
@@ -0,0 +1,92 @@
+package org.eclipse.aether.internal.impl.concurrency;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import javax.inject.Named;
+import javax.inject.Singleton;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.eclipse.aether.RepositorySystemSession;
+import org.eclipse.aether.spi.concurrency.ResolverExecutor;
+import org.eclipse.aether.spi.concurrency.ResolverExecutorService;
+import org.eclipse.aether.util.concurrency.WorkerThreadFactory;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Default implementation of {@link ResolverExecutor}.
+ * <p>
+ * This implementation uses {@link RepositorySystemSession#getData()} to store created {@link ExecutorService}
+ * instances. It creates instances that may be eventually garbage collected, so no explicit shutdown happens on
+ * them. When {@code maxThreads} parameter is 1 (accepted values are greater than zero), this implementation assumes
+ * caller wants "direct execution" (on caller thread) and creates {@link ResolverExecutor} instances accordingly.
+ */
+@Singleton
+@Named
+public final class DefaultResolverExecutorService implements ResolverExecutorService
+{
+ @Override
+ public ResolverExecutor getResolverExecutor( RepositorySystemSession session,
+ Class<?> service,
+ int maxThreads )
+ {
+ requireNonNull( session );
+ requireNonNull( service );
+ if ( maxThreads < 1 )
+ {
+ throw new IllegalArgumentException( "threads must be greater than zero" );
+ }
+
+ final ExecutorService executorService;
+ if ( maxThreads == 1 ) // direct
+ {
+ executorService = null;
+ }
+ else // shared && pooled
+ {
+ String key = DefaultResolverExecutorService.class.getName() + "." + service.getSimpleName();
+ executorService = (ExecutorService) session.getData()
+ .computeIfAbsent( key, () -> createExecutorService( service, maxThreads ) );
+ }
+ return new DefaultResolverExecutor( executorService );
+ }
+
+ /**
+ * Creates am {@link ExecutorService} that allows its core threads to die off in case of inactivity, and allows
+ * for proper garbage collection. This is important detail, as these instances are kept within session data, and
+ * currently there is no way to shut down them.
+ */
+ private ExecutorService createExecutorService( Class<?> service, int maxThreads )
+ {
+ ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(
+ maxThreads,
+ maxThreads,
+ 3L, TimeUnit.SECONDS,
+ new LinkedBlockingQueue<>(),
+ new WorkerThreadFactory( getClass().getSimpleName() + "-" + service.getSimpleName() + "-" )
Review Comment:
So we have lost the per repo thread pool?
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/DefaultMetadataResolver.java:
##########
@@ -83,8 +78,16 @@
implements MetadataResolver, Service
{
+ /**
+ * The count of threads to be used when resolving metadata in parallel, default value 4.
+ */
private static final String CONFIG_PROP_THREADS = "aether.metadataResolver.threads";
+ /**
+ * The default value for {@link #CONFIG_PROP_THREADS}.
+ */
+ private static final int CONFIG_PROP_THREADS_DEFAULT = 4;
Review Comment:
Why not 5 like in the other one?
##########
maven-resolver-connector-basic/src/main/java/org/eclipse/aether/connector/basic/BasicRepositoryConnector.java:
##########
@@ -287,9 +269,10 @@ public void get( Collection<? extends ArtifactDownload> artifactDownloads,
task = new GetTaskRunner( location, transfer.getFile(), checksumPolicy,
checksumAlgorithmFactories, checksumLocations, providedChecksums, listener );
}
- executor.execute( errorForwarder.wrap( task ) );
+ runnable.add( errorForwarder.wrap( task ) );
}
+ resolverExecutor.submitOrDirect( runnable );
Review Comment:
Why wait until they are all collected instead separate MD and A like before?
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/concurrency/DefaultResolverExecutorService.java:
##########
@@ -0,0 +1,92 @@
+package org.eclipse.aether.internal.impl.concurrency;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import javax.inject.Named;
+import javax.inject.Singleton;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.eclipse.aether.RepositorySystemSession;
+import org.eclipse.aether.spi.concurrency.ResolverExecutor;
+import org.eclipse.aether.spi.concurrency.ResolverExecutorService;
+import org.eclipse.aether.util.concurrency.WorkerThreadFactory;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Default implementation of {@link ResolverExecutor}.
+ * <p>
+ * This implementation uses {@link RepositorySystemSession#getData()} to store created {@link ExecutorService}
+ * instances. It creates instances that may be eventually garbage collected, so no explicit shutdown happens on
+ * them. When {@code maxThreads} parameter is 1 (accepted values are greater than zero), this implementation assumes
+ * caller wants "direct execution" (on caller thread) and creates {@link ResolverExecutor} instances accordingly.
+ */
+@Singleton
+@Named
+public final class DefaultResolverExecutorService implements ResolverExecutorService
+{
+ @Override
+ public ResolverExecutor getResolverExecutor( RepositorySystemSession session,
+ Class<?> service,
+ int maxThreads )
+ {
+ requireNonNull( session );
+ requireNonNull( service );
+ if ( maxThreads < 1 )
+ {
+ throw new IllegalArgumentException( "threads must be greater than zero" );
Review Comment:
maxThreads must be...
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/concurrency/DefaultResolverExecutor.java:
##########
@@ -0,0 +1,93 @@
+package org.eclipse.aether.internal.impl.concurrency;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Collection;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import org.eclipse.aether.spi.concurrency.ResolverExecutor;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Default implementation of {@link ResolverExecutor}.
+ * <p>
+ * It relies on ctor passed {@link ExecutorService} that may be {@code null}, in which case "direct invocation" (on
+ * caller thread) happens, otherwise the non-null executor service is used.
+ */
+final class DefaultResolverExecutor implements ResolverExecutor
+{
+ private final ExecutorService executorService;
+
+ DefaultResolverExecutor( final ExecutorService executorService )
+ {
+ this.executorService = executorService;
+ }
+
+ @Override
+ public void submitOrDirect( Collection<Runnable> tasks )
Review Comment:
I think from an abstract PoV just `submit` is enough. It is an implementation detail whether it should be executed directly or scheduled for execution. The behavior should be described in the Javadoc of the class.
##########
maven-resolver-connector-basic/src/main/java/org/eclipse/aether/connector/basic/BasicRepositoryConnector.java:
##########
@@ -225,11 +205,13 @@ public void get( Collection<? extends ArtifactDownload> artifactDownloads,
throw new IllegalStateException( "connector closed" );
}
- Executor executor = getExecutor( artifactDownloads, metadataDownloads );
RunnableErrorForwarder errorForwarder = new RunnableErrorForwarder();
List<ChecksumAlgorithmFactory> checksumAlgorithmFactories = layout.getChecksumAlgorithmFactories();
+ Collection<? extends MetadataDownload> mds = safe( metadataDownloads );
+ Collection<? extends ArtifactDownload> ads = safe( artifactDownloads );
+ ArrayList<Runnable> runnable = new ArrayList<>( mds.size() + ads.size() );
Review Comment:
runnables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org