You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by ct...@apache.org on 2026/02/06 19:38:12 UTC
(accumulo-classloaders) branch main updated: Rebrand to CachingClassLoaderFactory (#68)
This is an automated email from the ASF dual-hosted git repository.
ctubbsii pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-classloaders.git
The following commit(s) were added to refs/heads/main by this push:
new d20da30 Rebrand to CachingClassLoaderFactory (#68)
d20da30 is described below
commit d20da30a091b63cd0142ad16299ec660336729a6
Author: Christopher Tubbs <ct...@apache.org>
AuthorDate: Fri Feb 6 14:38:06 2026 -0500
Rebrand to CachingClassLoaderFactory (#68)
* Use shorter name "Caching" (dropping "Local" and "Context") for improved branding
* Shorten "ContextDefinition" to "Manifest"
* Improve javadoc and README
* Reduce class visiblity (reduce the number that are public) and
consolidate into fewer packages (also helps with Logger names, since
the parent package will be "cargo" for the Logger instances)
* Make optional grace period property re-configurable on a running
system
---
.../.gitignore | 0
modules/caching-class-loader/README.md | 287 +++++++++++++++
.../pom.xml | 4 +-
.../classloader/ccl/CachingClassLoaderFactory.java | 404 +++++++++++++++++++++
.../classloader/ccl}/DeduplicationCache.java | 4 +-
.../classloader/ccl/DeduplicationCacheKey.java} | 26 +-
.../accumulo/classloader/ccl}/LocalStore.java | 111 +++---
.../classloader/ccl/manifest/Manifest.java} | 43 ++-
.../classloader/ccl/manifest}/Resource.java | 2 +-
.../ccl/CachingClassLoaderFactoryTest.java} | 391 ++++++++++----------
.../classloader/ccl}/DeduplicationCacheTest.java | 2 +-
.../accumulo/classloader/ccl}/LocalStoreTest.java | 113 +++---
.../MiniAccumuloClusterClassLoaderFactoryTest.java | 72 ++--
.../accumulo/classloader/ccl}/TestUtils.java | 38 +-
.../accumulo/classloader/ccl}/URLTypesTest.java | 2 +-
.../src/test/java/test/HelloWorldTemplate | 0
.../src/test/java/test/Test.java | 0
.../src/test/java/test/TestTemplate | 0
.../src/test/resources/log4j2-test.properties | 4 +-
.../src/test/shell/makeHelloWorldJars.sh | 0
.../src/test/shell/makeTestJars.sh | 0
modules/local-caching-classloader/README.md | 290 ---------------
.../lcc/LocalCachingContextClassLoaderFactory.java | 321 ----------------
.../accumulo/classloader/lcc/util/LccUtils.java | 133 -------
pom.xml | 4 +-
25 files changed, 1099 insertions(+), 1152 deletions(-)
diff --git a/modules/local-caching-classloader/.gitignore b/modules/caching-class-loader/.gitignore
similarity index 100%
rename from modules/local-caching-classloader/.gitignore
rename to modules/caching-class-loader/.gitignore
diff --git a/modules/caching-class-loader/README.md b/modules/caching-class-loader/README.md
new file mode 100644
index 0000000..25f33b7
--- /dev/null
+++ b/modules/caching-class-loader/README.md
@@ -0,0 +1,287 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# Caching ClassLoader Factory
+
+`CachingClassLoaderFactory` implements Accumulo's `ContextClassLoaderFactory`
+SPI. This implementation is designed around a set of remote resources
+(typically, `.jar` files) listed in a JSON-formatted manifest for each context,
+and a local cache directory that is intended to be shared across multiple
+processes.
+
+The URL to the manifest, as a String, is used for the context parameter. In
+return, this factory downloads the resource from the remote locations specified
+in the manifest to a local storage cache, where they will be used to produce a
+corresponding `ClassLoader` instance. The specified URL will then be monitored
+for any changes to the manifest at that location, at the monitoring interval
+specified in the manifest.
+
+## Introduction
+
+This factory creates `ClassLoader` instances that point to locally cached
+copies of remote resource files. In this way, this factory allows placing
+common resources in a remote location for use across many hosts.
+
+This factory uses a storage cache in the local file system for any files it
+downloads from a remote URL.
+
+To use this factory, one must store resource files in a location that can be
+specified by a supported URL, and then must create a JSON-formatted manifest
+file that contains a monitoring interval (in seconds, greater than 0), and a
+list of resource URLs along with a checksum for each resource file. This
+manifest file must then be stored somewhere where this factory can download it,
+and use the URL to that file as the `context` parameter for this factory's
+`getClassLoader(String context)` method.
+
+This factory can handle manifest and resource URLs of any type that are
+supported by your application via a registered [URLStreamHandlerProvider][1],
+such as the built-in `file:` and `http:` types. A provider that handles `hdfs:`
+URL types must be provided by the user. This may be provided by the Apache
+Hadoop project, or by another library. A reference implementation is available
+[elsewhere in this project][2].
+
+Here is an example manifest:
+
+```json
+{
+ "monitorIntervalSeconds": 5,
+ "resources": [
+ {
+ "location": "file:/home/user/ClassLoaderTestA/TestA.jar",
+ "algorithm": "MD5",
+ "checksum": "ae5e8248a9243751d60dbcaaedeb93ba"
+ },
+ {
+ "location": "hdfs://localhost:8020/contextB/TestB.jar",
+ "algorithm": "SHA-256",
+ "checksum": "ed95fe130090fd64c2caddc3e37555ada8ba49a91bfd8ec1fd5c989d340ad0e0"
+ },
+ {
+ "location": "http://localhost:80/TestC.jar",
+ "algorithm": "SHA3-224",
+ "checksum": "958f12ddc5acf87c2fe0ceed645327bb0c92e268acf915c4a374c14b"
+ },
+ {
+ "location": "https://localhost:80/TestD.jar",
+ "algorithm": "SHA-512/224",
+ "checksum": "f7f982521ceb8ca97662973ada9b92b86de6bbaf233f14fd47efd792"
+ }
+ ]
+}
+```
+
+## How it Works
+
+When this factory receives a request for a `ClassLoader` for a given URL, it
+downloads a copy of the manifest and parses it. If it has recently acquired
+that manifest, based on the monitoring interval from a previous retrieval, it
+can skip this step and use the manifest from the earlier retrieval, which is
+kept up-to-date by the background monitoring thread that started when it was
+previously retrieved. Once it has the manifest, it then returns a `ClassLoader`
+instance containing the resources defined in that manifest, first downloading
+any missing resources and verifying them using the checksums in manifest.
+
+`ClassLoader` instances are stored in a de-deduplicating cache in memory with a
+minimum lifetime of 24 hours. So, no two instances will ever exist in a process
+for the same manifest.
+
+If this manifest had not previously been downloaded, a background monitoring
+task is set up to ensure the URL is watched for any changes to the manifest.
+This monitoring continues for as long as there exists `ClassLoader` instances
+in the system that were constructed from the manifest at that URL (at least 24
+hours, since that is the minimum time they will exist in the de-duplicating
+cache).
+
+## Local Storage Cache
+
+The local storage cache location is configured by the user by setting the
+required Accumulo property named `general.custom.classloader.ccl.cache.dir`
+to a directory on the local file system. This location may be specified as an
+absolute path or as a URL representing an absolute path with the `file` scheme.
+The location, and its directory structure will be created on first use, if it
+does not already exist. This will cause an error if the application does not
+have permission to create the directories.
+
+The selected location should be a persistent location with plenty of space to
+store downloaded resources (usually jar files), and should be writable by all
+the processes which use this factory to share the same resources. You may wish
+to pre-create the base directory specified by the property, and the three
+sub-directories, `manifests`, `resources`, and `working`, to set the
+appropriate permissions and ACLs.
+
+Resources downloaded to this cache may be used by multiple manifests, threads,
+and processes, so be very careful when removing old contents to ensure that
+they are no longer needed. If a resource file is deleted from the local storage
+cache while a `ClassLoader` exists that references it, that `ClassLoader` may,
+and probably will, stop working correctly. Similarly, files that have been
+downloaded should not be modified, because any modification will likely cause
+unexpected behavior to classloaders still using the file.
+
+* Do **NOT** use a temporary directory for the local storage cache location.
+* The local storage cache location **MUST** use a file system that supports
+ atomic moves and hard links.
+
+## Security
+
+The Accumulo property `general.custom.classloader.ccl.allowed.urls.pattern`
+is another required parameter. It is used to limit the allowed URLs that can be
+fetched when downloading manifests or resources. Since the process using this
+factory will be using its own permissions to fetch resources, and placing a
+copy of those resources in a local directory where others may access them, that
+presents presents a potential file disclosure security risk. This property
+allows a system administrator to mitigate that risk by restricting access to
+only approved URLs. (e.g. to exclude non-approved locations like
+`file:/path/to/accumulo.properties` or
+`hdfs://host/path/to/accumulo/rfile.rf`).
+
+An example value for this property might look like:
+`https://example.com/path/to/manifests/.*` or
+`(file:/etc|hdfs://example[.]com:9000)/path/to/manifests/.*`
+
+Note: this property affects all URLs fetched by this factory, including
+manifest URLs and any resource URLs defined inside any fetched manifest. It
+should be updated by a system maintainer if any new manifests have need to use
+new locations. It may be updated on a running system, and will take effect
+after approximately a minute.
+
+## Creating a Manifest
+
+Users may take advantage of the `Manifest.create(int,String,URL[])` method to
+construct a `Manifest` object, programmatically. This will calculate the
+checksums of the classpath elements. `Manifest.toJson()` can be used to
+serialize the `Manifest` to a `String` to store in a file.
+
+Alternatively, if this library's jar is built and placed onto Accumulo's
+`CLASSPATH`, then one can run `bin/accumulo create-classloader-manifest` to
+create the manifest using the command-line. The resulting json is printed to
+stdout and can be redirected to a file. The command takes two arguments:
+
+1. the monitor interval, in seconds (e.g. `-i 300`),
+2. an optional checksum algorithm to use (e.g. `-a 'SHA3-512'`), and
+3. a list of file URLs (e.g. `hdfs://host:port/path/to/one.jar
+ file://host/path/to/two.jar`)
+
+## Updating a Manifest
+
+This factory uses a background thread to fetch the manifest at its initial URL
+using the interval specified in the manifest the last time it was retrieved.
+The manifest at a watched URL can be replaced at any time with any changes,
+including to changes to the monitor interval and resources. When the manifest
+is next retrieved, the new manifest will be used as though it were an entirely
+new manifest at that URL. The next retrieval will occur after the monitor
+interval read from the most recent retrieval elapses. Changes to the context's
+resources in any way will result in those new resources being downloaded,
+verified, and a new `ClassLoader` instance created and ready to be returned the
+next time `getClassLoader(String context)` is called with that URL.
+
+Note: if the contents of a manifest change in only inconsequential ways, such
+as JSON formatting changes, then those changes will not trigger any new
+downloads or `ClassLoader` staging. The is because the manifest JSON is
+normalized prior to computing its checksum to determine if any changes have
+occurred.
+
+## Error Handling
+
+If there is an exception in creating the initial `ClassLoader`, such as being
+unable to retrieve the manifest at the specified URL, or if a resource does not
+match its checksum in the manifest, then a `ContextClassLoaderException` is
+thrown. If there is an exception when updating the classloader, then the
+exception is logged and the classloader is not updated. Calls to
+`getClassLoader(String context)` will return the most recent classloader with
+valid contents.
+
+The property `general.custom.classloader.ccl.update.grace.minutes` determines
+how long the update process continues to return the most recent valid
+classloader when an exception occurs in the background update thread. A zero
+value (default) will cause the most recent valid classloader to be returned.
+Otherwise, if a non-zero number is configured, then monitoring will stop after
+the update has failed for that number of minutes. Once monitoring has stopped,
+any subsequent calls to `getClassLoader(String context)` will behave as it
+would during an initial request, throwing a `ContextClassLoaderException` if
+the manifest cannot be retrieved or a `ClassLoader` cannot be constructed at
+that time.
+
+## Cleanup
+
+Because the cache directory is shared among multiple processes, and one process
+can't know what the other processes are doing, this class cannot always clean
+up the shared cache directory of unused resources automatically. It is left to
+the user to remove unused files from the cache. The local storage is organized
+into several directories, which are explained here to aid in understanding when
+unused files can be safely removed.
+
+### Manifests
+
+The `manifests` directory contents are always safe to delete, but doing so may
+impair troubleshooting. This directory contains copies of any downloaded
+manifests, but the behavior of the factory does not depend on the existence of
+these files.
+
+### Resources
+
+The `resources` directory contains a shared pool of remote resource files that
+have been fetched for all manifests (typically, `.jar` files). The files in
+this directory are safe to delete any time. However, some considerations should
+be made:
+
+1. Deleting resources that are still needed will cause them to be downloaded
+ again the next time they are needed, which may cause an increase in network
+ activity.
+2. If any of the removed files had hard-linked "copies" in the `working`
+ directory, the newly downloaded copy will increase the total amount of
+ storage (whereas the original would have shared storage space with the
+ hard-linked "copies").
+
+### Working
+
+The `working` directory contains temporary files for files currently being
+downloaded, and temporary directories containing hard-linked "copies" of files
+from the `resources` directory. These files and directories contain the process
+ID (PID) for the process that created them. Normally, these files are
+automatically cleaned up, but if a process is killed before that can happen,
+they may be left behind. The files with the PID in them can safely be removed,
+so long as the process that created them has been terminated.
+
+This directory also contains files that do not contain a PID. These files end
+with the `.downloading` suffix and exist to signal across processes that a
+resource file is currently being downloaded by a process. These files are very
+small, containing only the PID of the most recent process to attempt
+downloading the file. They are removed when a download completes, or whenever
+the next time the corresponding resource file is used, if it has already been
+successfully downloaded by a previously failed process. Removing them won't
+break the application in any way, but doing so may result in a redundant
+download, which can result in increased network activity or storage space (see
+the previous section for considerations regarding the `resources` directory).
+
+## Accumulo Configuration
+
+To use this with Accumulo:
+
+1. Set the following Accumulo properties:
+ * `general.context.class.loader.factory=org.apache.accumulo.classloader.ccl.CachingClassLoaderFactory`
+ * `general.custom.classloader.ccl.cache.dir=file://path/to/some/directory`
+ * `general.custom.classloader.ccl.allowed.urls.pattern=someRegexPatternForAllowedUrls`
+ * (optional) `general.custom.classloader.ccl.update.grace.minutes=30`
+2. Set the following table property to link to a manifest. For example:
+ * `table.class.loader.context=(file|hdfs|http|https)://path/to/context/manifest.json`
+
+
+[1]: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/spi/URLStreamHandlerProvider.html
+[2]: https://github.com/apache/accumulo-classloaders/tree/main/modules/hdfs-urlstreamhandler-provider
diff --git a/modules/local-caching-classloader/pom.xml b/modules/caching-class-loader/pom.xml
similarity index 98%
rename from modules/local-caching-classloader/pom.xml
rename to modules/caching-class-loader/pom.xml
index bd73a78..297f69e 100644
--- a/modules/local-caching-classloader/pom.xml
+++ b/modules/caching-class-loader/pom.xml
@@ -27,8 +27,8 @@
<version>1.0.0-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
- <artifactId>local-caching-classloader</artifactId>
- <name>classloader-extras-local-caching</name>
+ <artifactId>caching-classloader</artifactId>
+ <name>classloader-extras-caching-classloader</name>
<dependencies>
<dependency>
<!-- needed for build checks, but not for runtime -->
diff --git a/modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/CachingClassLoaderFactory.java b/modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/CachingClassLoaderFactory.java
new file mode 100644
index 0000000..d5b199b
--- /dev/null
+++ b/modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/CachingClassLoaderFactory.java
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.ccl;
+
+import static java.util.Objects.requireNonNull;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.lang.ref.Cleaner;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BiConsumer;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+
+import org.apache.accumulo.classloader.ccl.LocalStore.HardLinkFailedException;
+import org.apache.accumulo.classloader.ccl.manifest.Manifest;
+import org.apache.accumulo.classloader.ccl.manifest.Resource;
+import org.apache.accumulo.core.spi.common.ContextClassLoaderEnvironment;
+import org.apache.accumulo.core.spi.common.ContextClassLoaderFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Stopwatch;
+import com.google.common.base.Suppliers;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+/**
+ * A ContextClassLoaderFactory implementation where the context is based on a JSON-formatted
+ * {@link Manifest} file at a URL specified in the parameter for the {@link #getClassLoader(String)}
+ * method. The manifest contains a monitor interval and a set of {@link Resource} URLs to transport
+ * from their remote storage location to the local disk, where it will be used to construct and
+ * return a {@link ClassLoader}.
+ *
+ * <p>
+ * The manifest URL will be monitored for updates at the monitor interval specified in the manifest,
+ * and the class loader instance will be updated if any changes to the manifest occur. If the
+ * monitoring fails for a period configurable with the {@value #PROP_GRACE_PERIOD} property, then
+ * monitoring will discontinue until the next time the {@link #getClassLoader(String)} is called for
+ * that manifest URL.
+ *
+ * <p>
+ * Each resource in the manifest is defined by a URL to the file, a checksum algorithm, and a
+ * checksum.
+ *
+ * <p>
+ * The URLs supplied for the manifest and for the resources may use any URL type with a registered
+ * provider in your application, such as: file, http, https, or hdfs.
+ *
+ * <p>
+ * Properties to control this factory's behavior can be passed via
+ * {@link #init(ContextClassLoaderEnvironment)} and include the following:
+ *
+ * <ul>
+ * <li>{@value #PROP_CACHE_DIR}: (required) the location of the local storage for holding
+ * resources</li>
+ * <li>{@value #PROP_ALLOWED_URLS}: (required) a Java regular expression of allowable URLs to access
+ * for both manifests and resources</li>
+ * <li>{@value #PROP_GRACE_PERIOD}: (optional; default: 0) the number of minutes to tolerate
+ * failures while attempting to monitor a manifest URL before causing an error in the application; a
+ * value of 0 will tolerate failures indefinitely</li>
+ * </ul>
+ *
+ * <p>
+ * The storage directory may be shared by multiple processes using this factory. Temporary files,
+ * download-in-progress signal files, hard-links, and atomic file system moves are used to ensure
+ * that processes do not conflict with one another, and to ensure network efficiency by avoiding
+ * concurrent downloads for the same resource files.
+ */
+public class CachingClassLoaderFactory implements ContextClassLoaderFactory {
+
+ public static final String PROP_CACHE_DIR = "general.custom.classloader.ccl.cache.dir";
+
+ public static final String PROP_GRACE_PERIOD =
+ "general.custom.classloader.ccl.update.grace.minutes";
+
+ public static final String PROP_ALLOWED_URLS =
+ "general.custom.classloader.ccl.allowed.urls.pattern";
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingClassLoaderFactory.class);
+ private static final Cleaner CLEANER = Cleaner.create();
+
+ // executor for the monitor tasks
+ private final ScheduledExecutorService executor = Executors.newScheduledThreadPool(0);
+
+ // stores the latest seen manifest for a remote URL; String types are used here for the key
+ // instead of URL because URL.hashCode could trigger network activity for hostname lookups
+ private final ConcurrentHashMap<String,Manifest> manifests = new ConcurrentHashMap<>();
+
+ // to keep this coherent with the manifests, updates to this should be done in manifests.compute()
+ private final DeduplicationCache<DeduplicationCacheKey,LocalStore,URLClassLoader> classloaders =
+ new DeduplicationCache<>(CachingClassLoaderFactory::createClassLoader, Duration.ofHours(24),
+ null);
+
+ private final AtomicReference<LocalStore> localStore = new AtomicReference<>();
+
+ private final Map<String,Stopwatch> classloaderFailures = new HashMap<>();
+ private volatile Supplier<Duration> updateFailureGracePeriodMins;
+
+ // this is a BiConsumer so we can pass a type in the String
+ private volatile BiConsumer<String,URL> allowedUrlChecker;
+
+ /**
+ * Schedule a task to execute at {@code interval} seconds to update the cached resources if the
+ * manifest has changed. The task schedules a follow-on task at the update interval value (if it
+ * changed) and the task is successful or throws a handled exception. When an unhandled exception
+ * is thrown, then the corresponding entry in the manifests map is cleared. The next call to
+ * {@code #getClassLoader(String)} will recreate the manifests map entry and schedule the monitor
+ * task.
+ */
+ private void monitor(final String url, long interval) {
+ LOG.trace("Monitoring manifest {} for changes at {} second intervals", url, interval);
+ executor.schedule(() -> {
+ try {
+ checkMonitoredUrl(url, interval);
+ } catch (Throwable t) {
+ LOG.error("Unhandled exception occurred in manifest monitor thread. Removing manifest {}.",
+ url, t);
+ manifests.remove(url);
+ throw t;
+ }
+ }, interval, TimeUnit.SECONDS);
+ }
+
+ @Override
+ public void init(ContextClassLoaderEnvironment env) {
+ String value = requireNonNull(env.getConfiguration().get(PROP_CACHE_DIR),
+ "Property " + PROP_CACHE_DIR + " not set, cannot create cache directory.");
+
+ // these suppliers are used so we can update these config properties without restarting,
+ // but limit the frequency at which we check the config for the grace period and the url pattern
+ var maxFrequency = Duration.ofMinutes(1);
+ updateFailureGracePeriodMins = Suppliers.memoizeWithExpiration(() -> {
+ String graceProp = env.getConfiguration().get(PROP_GRACE_PERIOD);
+ long graceMins = graceProp == null ? 0 : Long.parseLong(graceProp);
+ return Duration.ofMinutes(graceMins);
+ }, maxFrequency);
+ Supplier<
+ Pattern> allowedUrlsPattern =
+ Suppliers.memoizeWithExpiration(
+ () -> Pattern.compile(requireNonNull(env.getConfiguration().get(PROP_ALLOWED_URLS),
+ "Property " + PROP_ALLOWED_URLS + " not set, no URLs are allowed")),
+ maxFrequency);
+
+ allowedUrlChecker = (locationType, url) -> {
+ var p = allowedUrlsPattern.get();
+ Preconditions.checkArgument(p.matcher(url.toExternalForm()).matches(),
+ "Context %s URL (%s) not allowed by pattern (%s)", locationType, url.toExternalForm(),
+ p.pattern());
+ };
+ try {
+ // check the allowed URLs pattern, getting it ready for first use, and warning if it is bad
+ allowedUrlsPattern.get();
+ } catch (RuntimeException npe) {
+ LOG.warn(
+ "Property {} is not set or contains an invalid pattern ()."
+ + " No ClassLoader instances will be created until it is set.",
+ PROP_ALLOWED_URLS, env.getConfiguration().get(PROP_ALLOWED_URLS), npe);
+ }
+ final Path baseCacheDir;
+ if (value.startsWith("file:")) {
+ try {
+ baseCacheDir = Path.of(new URL(value).toURI());
+ } catch (IOException | URISyntaxException e) {
+ throw new IllegalArgumentException(
+ "Malformed file: URL specified for base directory: " + value, e);
+ }
+ } else if (value.startsWith("/")) {
+ baseCacheDir = Path.of(value);
+ } else {
+ throw new IllegalArgumentException(
+ "Base directory is neither a file URL nor an absolute file path: " + value);
+ }
+ try {
+ localStore.set(new LocalStore(baseCacheDir, allowedUrlChecker));
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to create the local storage area at " + baseCacheDir,
+ e);
+ }
+ }
+
+ @Override
+ public ClassLoader getClassLoader(final String manifestUrl) throws ContextClassLoaderException {
+ Preconditions.checkState(localStore.get() != null,
+ "init not called before calling getClassLoader");
+ requireNonNull(manifestUrl, "manifest URL must be supplied");
+ final var classloader = new AtomicReference<URLClassLoader>();
+ try {
+ // get the current manifest, or create it from the URL if absent; this has the side effect of
+ // creating and caching a class loader instance if it doesn't exist for the computed manifest
+ manifests.compute(manifestUrl,
+ (key, previous) -> computeManifestAndClassLoader(classloader, key, previous));
+ } catch (RuntimeException e) {
+ throw new ContextClassLoaderException(e.getMessage(), e);
+ }
+ return classloader.get();
+ }
+
+ private Manifest computeManifestAndClassLoader(AtomicReference<URLClassLoader> resultHolder,
+ String url, Manifest previous) {
+ Manifest computed;
+ if (previous == null) {
+ try {
+ computed = downloadManifest(url);
+ // we can set up monitoring now, but it will be blocked from doing anything yet, until this
+ // finishes, since this code and the monitoring code both use manifests.compute(), which
+ // is atomic/blocking for the same key
+ monitor(url, computed.getMonitorIntervalSeconds());
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ } else {
+ computed = previous;
+ }
+ resultHolder.set(
+ classloaders.computeIfAbsent(new DeduplicationCacheKey(url, computed), localStore::get));
+ return computed;
+ }
+
+ private Manifest downloadManifest(String url) throws IOException {
+ LOG.trace("Retrieving manifest from {}", url);
+ URL urlUrl = new URL(url);
+ allowedUrlChecker.accept("manifest", urlUrl);
+ return Manifest.download(urlUrl);
+ }
+
+ private void checkMonitoredUrl(String url, long interval) {
+ Manifest current = manifests.compute(url, (key, previous) -> {
+ if (previous == null) {
+ // manifest has been removed from the map, no need to check for update
+ LOG.debug("Manifest for {} not present, no longer monitoring for changes", url);
+ return null;
+ }
+ // check for any classloaders still in the cache that were created for a manifest
+ // found at this URL
+ if (!classloaders.anyMatch(cacheKey -> cacheKey.getLocation().equals(url))) {
+ LOG.debug("ClassLoader for {} not present, no longer monitoring for changes", url);
+ return null;
+ }
+ return previous;
+ });
+ if (current == null) {
+ return;
+ }
+ long nextInterval = interval;
+ try {
+ final Manifest update = downloadManifest(url);
+ if (!current.getChecksum().equals(update.getChecksum())) {
+ LOG.debug("Context manifest for {} has changed", url);
+ localStore.get().storeContext(update);
+ manifests.put(url, update);
+ nextInterval = update.getMonitorIntervalSeconds();
+ classloaderFailures.remove(url);
+ } else {
+ LOG.trace("Context manifest for {} has not changed", url);
+ }
+ // reschedule this task to run if the manifest exists.
+ // Atomically lock on the key and only reschedule if the value is present.
+ final long finalMonitorInterval = nextInterval;
+ manifests.compute(url, (k, v) -> {
+ if (v != null) {
+ monitor(url, finalMonitorInterval);
+ }
+ return v;
+ });
+ } catch (IOException | RuntimeException e) {
+ LOG.error("Error parsing updated manifest at {}. Classloader NOT updated!", url, e);
+ final Stopwatch failureTimer = classloaderFailures.get(url);
+ var gracePeriod = updateFailureGracePeriodMins.get();
+ if (gracePeriod.isZero()) {
+ // failure monitoring is disabled
+ LOG.debug("Property {} not set, not tracking classloader failures for {}",
+ PROP_GRACE_PERIOD, url);
+ } else if (failureTimer == null) {
+ // first failure, start the timer
+ classloaderFailures.put(url, Stopwatch.createStarted());
+ LOG.debug(
+ "Tracking classloader failures for {}, will NOT return working classloader if failures continue for {} minutes",
+ url, gracePeriod.toMinutes());
+ } else if (failureTimer.elapsed().compareTo(gracePeriod) > 0) {
+ // has been failing for the grace period
+ // unset the classloader reference so that the failure
+ // will return from getClassLoader in the calling thread
+ LOG.info("Grace period for failing classloader has elapsed for {}", url);
+ manifests.remove(url);
+ classloaderFailures.remove(url);
+ } else {
+ LOG.trace("Failing to update classloader for {} within the grace period", url, e);
+ }
+ // reschedule this task to run if the manifest exists.
+ // Don't put this in finally block as we only want to reschedule
+ // on success or handled exception
+ // Atomically lock on the key and only reschedule if the value is present.
+ final long finalMonitorInterval = nextInterval;
+ manifests.compute(url, (k, v) -> {
+ if (v != null) {
+ monitor(url, finalMonitorInterval);
+ }
+ return v;
+ });
+ }
+ }
+
+ @SuppressFBWarnings(value = "DP_CREATE_CLASSLOADER_INSIDE_DO_PRIVILEGED",
+ justification = "doPrivileged is deprecated without replacement and removed in newer Java")
+ static URLClassLoader createClassLoader(DeduplicationCacheKey cacheKey, LocalStore localStore) {
+ // use a LinkedHashSet to preserve the order of the resources
+ final var hardLinks = new LinkedHashSet<Path>();
+ Path hardLinksDir = null;
+
+ var manifest = cacheKey.getManifest();
+
+ // stage the downloads before attempting hard link creation
+ localStore.storeContext(manifest);
+
+ // keep trying to hard-link all the resources if the hard-linking fails
+ while (hardLinksDir == null) {
+ hardLinks.clear();
+ try {
+ hardLinksDir = localStore.createWorkingHardLinks(manifest, hardLinks::add);
+ LOG.trace("Created hard links at {} for {}", hardLinksDir, cacheKey);
+ } catch (HardLinkFailedException e) {
+ var failedHardLinksDir = e.getDestinationDirectory();
+ LOG.warn(
+ "Exception creating a hard link in {} due to missing resource {}; attempting re-download of context resources",
+ failedHardLinksDir, e.getMissingResource(), e);
+ try {
+ recursiveDelete(failedHardLinksDir);
+ } catch (IOException ioe) {
+ LOG.warn(
+ "Saw exception removing directory {} after hard link creation failure; this should be cleaned up manually",
+ failedHardLinksDir, ioe);
+ }
+ localStore.storeContext(manifest);
+ }
+ }
+
+ URL[] urls = hardLinks.stream().map(p -> {
+ try {
+ LOG.trace("Added resource {} to classpath", p);
+ return p.toUri().toURL();
+ } catch (MalformedURLException e) {
+ // shouldn't be possible, since these are file-based URLs
+ throw new UncheckedIOException(e);
+ }
+ }).toArray(URL[]::new);
+
+ final var cl = new URLClassLoader(cacheKey.toString(), urls,
+ CachingClassLoaderFactory.class.getClassLoader());
+ LOG.info("New classloader created for {}", cacheKey);
+
+ final var cleanDir = hardLinksDir;
+ CLEANER.register(cl, () -> {
+ try {
+ recursiveDelete(cleanDir);
+ } catch (IOException e) {
+ LOG.warn("Saw exception when executing cleaner on directory {}", cleanDir, e);
+ }
+ });
+ return cl;
+ }
+
+ private static void recursiveDelete(Path directory) throws IOException {
+ if (Files.exists(directory)) {
+ try (var walker = Files.walk(directory)) {
+ walker.map(Path::toFile).sorted(Comparator.reverseOrder()).forEach(File::delete);
+ }
+ }
+ }
+
+}
diff --git a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/DeduplicationCache.java b/modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/DeduplicationCache.java
similarity index 97%
rename from modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/DeduplicationCache.java
rename to modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/DeduplicationCache.java
index 4edaa46..7e8660f 100644
--- a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/DeduplicationCache.java
+++ b/modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/DeduplicationCache.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.accumulo.classloader.lcc.util;
+package org.apache.accumulo.classloader.ccl;
import static java.util.Objects.requireNonNull;
@@ -39,7 +39,7 @@ import com.github.benmanes.caffeine.cache.Scheduler;
* minimum amount of time specified, to prevent garbage collection too frequently for objects that
* may be used again.
*/
-public class DeduplicationCache<KEY,PARAMS,VALUE> {
+class DeduplicationCache<KEY,PARAMS,VALUE> {
private static final Logger LOG = LoggerFactory.getLogger(DeduplicationCache.class);
diff --git a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/ContextCacheKey.java b/modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/DeduplicationCacheKey.java
similarity index 67%
rename from modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/ContextCacheKey.java
rename to modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/DeduplicationCacheKey.java
index f9f402e..1169368 100644
--- a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/ContextCacheKey.java
+++ b/modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/DeduplicationCacheKey.java
@@ -16,35 +16,35 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.accumulo.classloader.lcc.util;
+package org.apache.accumulo.classloader.ccl;
import static java.util.Objects.hash;
import static java.util.Objects.requireNonNull;
import java.util.Objects;
-import org.apache.accumulo.classloader.lcc.definition.ContextDefinition;
+import org.apache.accumulo.classloader.ccl.manifest.Manifest;
-public class ContextCacheKey {
+class DeduplicationCacheKey {
private final String location;
- private final ContextDefinition definition;
+ private final Manifest manifest;
- public ContextCacheKey(String location, ContextDefinition definition) {
+ public DeduplicationCacheKey(String location, Manifest manifest) {
this.location = requireNonNull(location);
- this.definition = requireNonNull(definition);
+ this.manifest = requireNonNull(manifest);
}
public String getLocation() {
return location;
}
- public ContextDefinition getContextDefinition() {
- return definition;
+ public Manifest getManifest() {
+ return manifest;
}
@Override
public int hashCode() {
- return hash(location, definition);
+ return hash(location, manifest);
}
@Override
@@ -52,16 +52,16 @@ public class ContextCacheKey {
if (this == obj) {
return true;
}
- if (obj instanceof ContextCacheKey) {
- var o = (ContextCacheKey) obj;
- return Objects.equals(location, o.location) && Objects.equals(definition, o.definition);
+ if (obj instanceof DeduplicationCacheKey) {
+ var o = (DeduplicationCacheKey) obj;
+ return Objects.equals(location, o.location) && Objects.equals(manifest, o.manifest);
}
return false;
}
@Override
public String toString() {
- return definition.getChecksumAlgorithm() + " (" + location + ") = " + definition.getChecksum();
+ return manifest.getChecksumAlgorithm() + " (" + location + ") = " + manifest.getChecksum();
}
}
diff --git a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/LocalStore.java b/modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/LocalStore.java
similarity index 79%
rename from modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/LocalStore.java
rename to modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/LocalStore.java
index ecf7f7e..be65060 100644
--- a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/LocalStore.java
+++ b/modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/LocalStore.java
@@ -16,15 +16,13 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.accumulo.classloader.lcc.util;
+package org.apache.accumulo.classloader.ccl;
import static java.nio.charset.StandardCharsets.UTF_8;
import static java.nio.file.StandardCopyOption.ATOMIC_MOVE;
import static java.nio.file.StandardOpenOption.CREATE_NEW;
import static java.nio.file.StandardOpenOption.SYNC;
import static java.util.Objects.requireNonNull;
-import static org.apache.accumulo.classloader.lcc.util.LccUtils.checksumForFileName;
-import static org.apache.accumulo.classloader.lcc.util.LccUtils.getDigester;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
@@ -45,22 +43,28 @@ import java.util.function.BiConsumer;
import java.util.function.Consumer;
import java.util.regex.Pattern;
-import org.apache.accumulo.classloader.lcc.definition.ContextDefinition;
-import org.apache.accumulo.classloader.lcc.definition.Resource;
+import org.apache.accumulo.classloader.ccl.manifest.Manifest;
+import org.apache.accumulo.classloader.ccl.manifest.Resource;
+import org.apache.commons.codec.digest.DigestUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
/**
- * A simple storage service backed by a local file system for storing downloaded
- * {@link ContextDefinition} files and the {@link Resource} objects it references.
+ * A simple storage service backed by a local file system for storing downloaded {@link Manifest}
+ * files and the {@link Resource} objects it references.
+ *
* <p>
- * The layout of the storage area consists of two directories:
+ * The layout of the storage area consists of three directories. You may create this directory
+ * structure in advance, but they must be on the same file system, one which supports atomic moves
+ * and hard-links. The directories are:
* <ul>
- * <li><b>contexts</b> stores a copy of the {@link ContextDefinition} JSON files for each context,
- * and exist primarily for user convenience (they aren't used again by this factory)
- * <li><b>resources</b> stores a copy of all the {@link Resource} files for all contexts
+ * <li><b>manifests</b>: stores a copy of the {@link Manifest} JSON files for each context, and
+ * exist primarily for user convenience (they aren't used again by this factory)
+ * <li><b>resources</b>: stores a copy of all the {@link Resource} files for all contexts
+ * <li><b>working</b>: files in use, such as temporary files while downloading, and directories of
+ * hard-linked resource files for the lifetime of a classloader
* </ul>
*
* <p>
@@ -71,27 +75,26 @@ import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
* <p>
* Downloads make a best effort attempt to avoid multiple processes or threads from downloading the
* same file at the same time, using a temporary signal file with the suffix ".downloading" to avoid
- * duplicate effort. Duplicate effort and race conditions can occur if one download attempt stalls.
- * This is intentional, so that others will not be blocked indefinitely. When multiple attempts to
- * download a file do occur concurrently, they use unique temporary file names and atomic file
- * system moves to ensure correctness, even under these circumstances.
+ * duplicate effort. Concurrent downloads can occur if one download attempt stalls. This is
+ * intentional, so that others will not be blocked indefinitely. When multiple attempts to download
+ * a file do occur concurrently, they use unique temporary file names and atomic file system moves
+ * to ensure correctness, even under these circumstances.
*
* <p>
- * Once all files for a context are downloaded, an array of {@link URL}s will be returned, which
- * point to the local files that have been downloaded for the context, in the same order as
- * specified in the {@link ContextDefinition} file, to be used for constructing a
- * {@link URLClassLoader}.
+ * Once all files for a context are downloaded, they are hard-linked to a temporary directory where
+ * their checksums are verified before use, so that a {@link URLClassLoader} can be constructed
+ * using the same resource ordering as in the {@link Manifest} file.
*/
-public final class LocalStore {
+final class LocalStore {
private static final Logger LOG = LoggerFactory.getLogger(LocalStore.class);
private static final String PID = Long.toString(ProcessHandle.current().pid());
- private final Path contextsDir;
+ private final Path manifestsDir;
private final Path resourcesDir;
private final Path workingDir;
private final BiConsumer<String,URL> allowedUrlChecker;
- public static final String CONTEXTS_DIR = "contexts";
+ public static final String MANIFESTS_DIR = "manifests";
public static final String RESOURCES_DIR = "resources";
public static final String WORKING_DIR = "working";
@@ -99,13 +102,13 @@ public final class LocalStore {
throws IOException {
requireNonNull(baseDir);
this.allowedUrlChecker = requireNonNull(allowedUrlChecker);
- this.contextsDir = Files.createDirectories(baseDir.resolve(CONTEXTS_DIR));
+ this.manifestsDir = Files.createDirectories(baseDir.resolve(MANIFESTS_DIR));
this.resourcesDir = Files.createDirectories(baseDir.resolve(RESOURCES_DIR));
this.workingDir = Files.createDirectories(baseDir.resolve(WORKING_DIR));
}
- Path contextsDir() {
- return contextsDir;
+ Path manifestsDir() {
+ return manifestsDir;
}
Path resourcesDir() {
@@ -133,6 +136,18 @@ public final class LocalStore {
return String.format("%s-%s", remoteFileName, checksum);
}
+ private static String checksumForFileName(String algorithm, String checksum) {
+ return algorithm.replace('/', '_') + "-" + checksum;
+ }
+
+ static String checksumForFileName(Manifest manifest) {
+ return checksumForFileName(manifest.getChecksumAlgorithm(), manifest.getChecksum());
+ }
+
+ static String checksumForFileName(Resource resource) {
+ return checksumForFileName(resource.getAlgorithm(), resource.getChecksum());
+ }
+
// creates a new empty file with a unique name, for use as a temporary file
@SuppressFBWarnings(value = "PATH_TRAVERSAL_IN",
justification = "the working directory is intentionally controlled by the user config")
@@ -156,18 +171,18 @@ public final class LocalStore {
}
/**
- * Save the {@link ContextDefinition} to the contexts directory, and all of its resources to the
- * resources directory.
+ * Save the {@link Manifest} to the manifests directory, and all of its resources to the resources
+ * directory.
*/
- public void storeContextResources(final ContextDefinition contextDefinition) {
- requireNonNull(contextDefinition, "definition must be supplied");
- final String destinationName = checksumForFileName(contextDefinition) + ".json";
+ void storeContext(final Manifest manifest) {
+ requireNonNull(manifest, "manifest must be supplied");
+ final String destinationName = checksumForFileName(manifest) + ".json";
try {
- storeContextDefinition(contextDefinition, destinationName);
+ storeManifest(manifest, destinationName);
int waitingOnOtherDownloadsCount;
do {
waitingOnOtherDownloadsCount = 0;
- for (Resource resource : contextDefinition.getResources()) {
+ for (Resource resource : manifest.getResources()) {
Path path = storeResource(resource);
if (path == null) {
LOG.trace("Skipped resource {} while another process or thread is downloading it",
@@ -181,10 +196,10 @@ public final class LocalStore {
Thread.sleep(Math.min(waitingOnOtherDownloadsCount * 100, 1_000));
} while (waitingOnOtherDownloadsCount > 0);
} catch (IOException e) {
- LOG.error("Error storing resources for context {}", destinationName, e);
+ LOG.error("Error storing resources for {}", destinationName, e);
throw new UncheckedIOException(e);
} catch (RuntimeException e) {
- LOG.error("Error storing resources for context {}", destinationName, e);
+ LOG.error("Error storing resources for {}", destinationName, e);
throw e;
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
@@ -192,15 +207,15 @@ public final class LocalStore {
}
}
- private void storeContextDefinition(final ContextDefinition contextDefinition,
- final String destinationName) throws IOException {
- Path destinationPath = contextsDir.resolve(destinationName);
+ private void storeManifest(final Manifest manifest, final String destinationName)
+ throws IOException {
+ Path destinationPath = manifestsDir.resolve(destinationName);
if (Files.exists(destinationPath)) {
return;
}
// Avoid colliding with other processes by saving to a unique temp name first
Path tempPath = createTempFile(destinationName);
- Files.write(tempPath, contextDefinition.toJson().getBytes(UTF_8));
+ Files.write(tempPath, manifest.toJson().getBytes(UTF_8));
Files.move(tempPath, destinationPath, ATOMIC_MOVE);
}
@@ -317,7 +332,7 @@ public final class LocalStore {
justification = "user-supplied URL is the intended functionality")
private void downloadFile(Path tempPath, Resource resource) {
URL url = resource.getLocation();
- allowedUrlChecker.accept("Resource", url);
+ allowedUrlChecker.accept("resource", url);
// SYNC ensures file integrity on each write, in case of system failure. Buffering minimizes
// system calls te read/write data which minimizes the number of syncs.
@@ -334,7 +349,7 @@ public final class LocalStore {
final String algorithm = resource.getAlgorithm();
final String checksum;
try {
- checksum = getDigester(algorithm).digestAsHex(downloadPath);
+ checksum = new DigestUtils(algorithm).digestAsHex(downloadPath);
} catch (IOException e) {
throw new UncheckedIOException("Unable to perform checksum verification on " + downloadPath
+ " for resource " + resource.getLocation(), e);
@@ -342,7 +357,7 @@ public final class LocalStore {
if (!resource.getChecksum().equals(checksum)) {
var ise = new IllegalStateException(
"Checksum " + checksum + " for resource " + resource.getLocation()
- + " does not match checksum in context definition " + resource.getChecksum());
+ + " does not match checksum in the manifest " + resource.getChecksum());
if (cleanUpAction != null) {
try {
cleanUpAction.close();
@@ -354,11 +369,11 @@ public final class LocalStore {
}
}
- Path createWorkingHardLinks(final ContextDefinition contextDefinition, Consumer<Path> forEachLink)
+ Path createWorkingHardLinks(final Manifest manifest, Consumer<Path> forEachLink)
throws HardLinkFailedException {
- Path hardLinkDir = createTempDirectory("context-" + checksumForFileName(contextDefinition));
+ Path hardLinkDir = createTempDirectory("context-" + checksumForFileName(manifest));
// create all hard links first
- for (Resource r : contextDefinition.getResources()) {
+ for (Resource r : manifest.getResources()) {
String fileName = localResourceName(r);
Path p = resourcesDir.resolve(fileName);
try {
@@ -372,7 +387,7 @@ public final class LocalStore {
}
}
// verify checksums
- for (Resource r : contextDefinition.getResources()) {
+ for (Resource r : manifest.getResources()) {
String fileName = localResourceName(r);
Path hardLink = hardLinkDir.resolve(fileName);
LOG.trace("Verifying checksum of hard link {} for resource {}", hardLink, r.getLocation());
@@ -388,17 +403,17 @@ public final class LocalStore {
private final Path destDir;
private Path missingResource;
- public HardLinkFailedException(Path destDir, Path missingResource, NoSuchFileException cause) {
+ HardLinkFailedException(Path destDir, Path missingResource, NoSuchFileException cause) {
super("Creating hard link in directory " + destDir + " failed", cause);
this.destDir = destDir;
this.missingResource = missingResource;
}
- public Path getDestinationDirectory() {
+ Path getDestinationDirectory() {
return destDir;
}
- public Path getMissingResource() {
+ Path getMissingResource() {
return missingResource;
}
diff --git a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/ContextDefinition.java b/modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/manifest/Manifest.java
similarity index 82%
rename from modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/ContextDefinition.java
rename to modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/manifest/Manifest.java
index 962abd2..e3d3672 100644
--- a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/ContextDefinition.java
+++ b/modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/manifest/Manifest.java
@@ -16,12 +16,11 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.accumulo.classloader.lcc.definition;
+package org.apache.accumulo.classloader.ccl.manifest;
import static java.nio.charset.StandardCharsets.UTF_8;
import static java.util.Objects.hash;
import static java.util.Objects.requireNonNull;
-import static org.apache.accumulo.classloader.lcc.util.LccUtils.getDigester;
import java.io.BufferedInputStream;
import java.io.EOFException;
@@ -39,6 +38,7 @@ import java.util.function.Consumer;
import java.util.function.Supplier;
import org.apache.accumulo.start.spi.KeywordExecutable;
+import org.apache.commons.codec.digest.DigestUtils;
import com.beust.jcommander.JCommander;
import com.beust.jcommander.Parameter;
@@ -52,7 +52,7 @@ import com.google.gson.GsonBuilder;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
@AutoService(KeywordExecutable.class)
-public class ContextDefinition implements KeywordExecutable {
+public class Manifest implements KeywordExecutable {
static class Opts {
@Parameter(names = {"-i", "--interval"}, required = true,
@@ -111,28 +111,28 @@ public class ContextDefinition implements KeywordExecutable {
@SuppressFBWarnings(value = "URLCONNECTION_SSRF_FD",
justification = "user-supplied URL is the intended functionality")
- public static ContextDefinition create(int monitorIntervalSecs, String algorithm, URL... sources)
+ public static Manifest create(int monitorIntervalSecs, String algorithm, URL... sources)
throws IOException {
- // use a LinkedHashSet to preserve the order of the context resources
+ // use a LinkedHashSet to preserve the order of the resources
LinkedHashSet<Resource> resources = new LinkedHashSet<>();
for (URL u : sources) {
try (InputStream is = new BufferedInputStream(u.openStream())) {
- String checksum = getDigester(algorithm).digestAsHex(is);
+ String checksum = new DigestUtils(algorithm).digestAsHex(is);
resources.add(new Resource(u, algorithm, checksum));
}
}
- return new ContextDefinition(monitorIntervalSecs, resources);
+ return new Manifest(monitorIntervalSecs, resources);
}
@SuppressFBWarnings(value = "URLCONNECTION_SSRF_FD",
justification = "user-supplied URL is the intended functionality")
- public static ContextDefinition fromRemoteURL(final URL url) throws IOException {
+ public static Manifest download(final URL url) throws IOException {
try (InputStream is = url.openStream()) {
- var def = GSON.fromJson(new InputStreamReader(is, UTF_8), ContextDefinition.class);
- if (def == null) {
- throw new EOFException("InputStream does not contain a valid ContextDefinition at " + url);
+ var manifest = GSON.fromJson(new InputStreamReader(is, UTF_8), Manifest.class);
+ if (manifest == null) {
+ throw new EOFException("InputStream does not contain a valid manifest at " + url);
}
- return def;
+ return manifest;
}
}
@@ -140,16 +140,16 @@ public class ContextDefinition implements KeywordExecutable {
// transient fields that don't go in the json
private final transient Supplier<String> checksum =
- Suppliers.memoize(() -> getDigester(getChecksumAlgorithm()).digestAsHex(toJson()));
+ Suppliers.memoize(() -> new DigestUtils(getChecksumAlgorithm()).digestAsHex(toJson()));
// serialized fields for json
- // use a LinkedHashSet to preserve the order specified in the context file
+ // use a LinkedHashSet to preserve the order specified in the file
private int monitorIntervalSeconds;
private LinkedHashSet<Resource> resources;
- public ContextDefinition() {}
+ public Manifest() {}
- public ContextDefinition(int monitorIntervalSeconds, LinkedHashSet<Resource> resources) {
+ public Manifest(int monitorIntervalSeconds, LinkedHashSet<Resource> resources) {
Preconditions.checkArgument(monitorIntervalSeconds > 0,
"monitor interval must be greater than zero");
this.monitorIntervalSeconds = monitorIntervalSeconds;
@@ -180,7 +180,7 @@ public class ContextDefinition implements KeywordExecutable {
if (getClass() != obj.getClass()) {
return false;
}
- ContextDefinition other = (ContextDefinition) obj;
+ Manifest other = (Manifest) obj;
return monitorIntervalSeconds == other.monitorIntervalSeconds
&& Objects.equals(resources, other.resources);
}
@@ -202,24 +202,23 @@ public class ContextDefinition implements KeywordExecutable {
@Override
public String keyword() {
- return "create-context-definition";
+ return "create-classloader-manifest";
}
@Override
public String description() {
- return "Creates and prints a Context Definition";
+ return "Creates and prints a class loader context manifest for the CachingClassLoaderFactory";
}
@Override
public void execute(String[] args) throws Exception {
Opts opts = new Opts();
- opts.parseArgs(ContextDefinition.class.getName(), args);
+ opts.parseArgs(Manifest.class.getName(), args);
URL[] urls = new URL[opts.files.size()];
int count = 0;
for (String f : opts.files) {
urls[count++] = new URL(f);
}
- ContextDefinition def = create(opts.monitorInterval, opts.algorithm, urls);
- System.out.print(def.toJson());
+ System.out.print(create(opts.monitorInterval, opts.algorithm, urls).toJson());
}
}
diff --git a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/Resource.java b/modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/manifest/Resource.java
similarity index 97%
rename from modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/Resource.java
rename to modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/manifest/Resource.java
index 9aef499..56153e0 100644
--- a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/Resource.java
+++ b/modules/caching-class-loader/src/main/java/org/apache/accumulo/classloader/ccl/manifest/Resource.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.accumulo.classloader.lcc.definition;
+package org.apache.accumulo.classloader.ccl.manifest;
import java.net.URL;
import java.nio.file.Path;
diff --git a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactoryTest.java b/modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/CachingClassLoaderFactoryTest.java
similarity index 61%
rename from modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactoryTest.java
rename to modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/CachingClassLoaderFactoryTest.java
index caf5aaf..25ef891 100644
--- a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactoryTest.java
+++ b/modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/CachingClassLoaderFactoryTest.java
@@ -16,18 +16,18 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.accumulo.classloader.lcc;
+package org.apache.accumulo.classloader.ccl;
import static java.nio.file.StandardCopyOption.REPLACE_EXISTING;
-import static org.apache.accumulo.classloader.lcc.LocalCachingContextClassLoaderFactory.ALLOWED_URLS_PATTERN;
-import static org.apache.accumulo.classloader.lcc.LocalCachingContextClassLoaderFactory.CACHE_DIR_PROPERTY;
-import static org.apache.accumulo.classloader.lcc.LocalCachingContextClassLoaderFactory.UPDATE_FAILURE_GRACE_PERIOD_MINS;
-import static org.apache.accumulo.classloader.lcc.TestUtils.createContextDefinitionFile;
-import static org.apache.accumulo.classloader.lcc.TestUtils.testClassFailsToLoad;
-import static org.apache.accumulo.classloader.lcc.TestUtils.testClassLoads;
-import static org.apache.accumulo.classloader.lcc.TestUtils.updateContextDefinitionFile;
-import static org.apache.accumulo.classloader.lcc.util.LocalStore.RESOURCES_DIR;
-import static org.apache.accumulo.classloader.lcc.util.LocalStore.WORKING_DIR;
+import static org.apache.accumulo.classloader.ccl.CachingClassLoaderFactory.PROP_ALLOWED_URLS;
+import static org.apache.accumulo.classloader.ccl.CachingClassLoaderFactory.PROP_CACHE_DIR;
+import static org.apache.accumulo.classloader.ccl.CachingClassLoaderFactory.PROP_GRACE_PERIOD;
+import static org.apache.accumulo.classloader.ccl.LocalStore.RESOURCES_DIR;
+import static org.apache.accumulo.classloader.ccl.LocalStore.WORKING_DIR;
+import static org.apache.accumulo.classloader.ccl.TestUtils.createManifestFile;
+import static org.apache.accumulo.classloader.ccl.TestUtils.testClassFailsToLoad;
+import static org.apache.accumulo.classloader.ccl.TestUtils.testClassLoads;
+import static org.apache.accumulo.classloader.ccl.TestUtils.updateManifestFile;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
@@ -57,10 +57,9 @@ import java.util.concurrent.atomic.AtomicBoolean;
import java.util.regex.PatternSyntaxException;
import java.util.stream.Collectors;
-import org.apache.accumulo.classloader.lcc.TestUtils.TestClassInfo;
-import org.apache.accumulo.classloader.lcc.definition.ContextDefinition;
-import org.apache.accumulo.classloader.lcc.definition.Resource;
-import org.apache.accumulo.classloader.lcc.util.LocalStore;
+import org.apache.accumulo.classloader.ccl.TestUtils.TestClassInfo;
+import org.apache.accumulo.classloader.ccl.manifest.Manifest;
+import org.apache.accumulo.classloader.ccl.manifest.Resource;
import org.apache.accumulo.core.conf.ConfigurationCopy;
import org.apache.accumulo.core.spi.common.ContextClassLoaderFactory.ContextClassLoaderException;
import org.apache.accumulo.core.util.ConfigurationImpl;
@@ -78,7 +77,7 @@ import org.junit.jupiter.api.io.TempDir;
import com.google.gson.JsonSyntaxException;
-public class LocalCachingContextClassLoaderFactoryTest {
+class CachingClassLoaderFactoryTest {
protected static final int MONITOR_INTERVAL_SECS = 5;
// MD5 sum for "bad"
@@ -91,15 +90,15 @@ public class LocalCachingContextClassLoaderFactoryTest {
private static URL jarCOrigLocation;
private static URL jarDOrigLocation;
private static URL jarEOrigLocation;
- private static URL localAllContext;
- private static URL hdfsAllContext;
- private static URL jettyAllContext;
+ private static URL localAllUrl;
+ private static URL hdfsAllUrl;
+ private static URL jettyAllUrl;
private static TestClassInfo classA;
private static TestClassInfo classB;
private static TestClassInfo classC;
private static TestClassInfo classD;
- private LocalCachingContextClassLoaderFactory FACTORY;
+ private CachingClassLoaderFactory FACTORY;
private Path baseCacheDir;
@TempDir(cleanup = CleanupMode.ON_SUCCESS)
@@ -109,19 +108,19 @@ public class LocalCachingContextClassLoaderFactoryTest {
public static void beforeAll() throws Exception {
// Find the Test jar files
jarAOrigLocation =
- LocalCachingContextClassLoaderFactoryTest.class.getResource("/ClassLoaderTestA/TestA.jar");
+ CachingClassLoaderFactoryTest.class.getResource("/ClassLoaderTestA/TestA.jar");
assertNotNull(jarAOrigLocation);
jarBOrigLocation =
- LocalCachingContextClassLoaderFactoryTest.class.getResource("/ClassLoaderTestB/TestB.jar");
+ CachingClassLoaderFactoryTest.class.getResource("/ClassLoaderTestB/TestB.jar");
assertNotNull(jarBOrigLocation);
jarCOrigLocation =
- LocalCachingContextClassLoaderFactoryTest.class.getResource("/ClassLoaderTestC/TestC.jar");
+ CachingClassLoaderFactoryTest.class.getResource("/ClassLoaderTestC/TestC.jar");
assertNotNull(jarCOrigLocation);
jarDOrigLocation =
- LocalCachingContextClassLoaderFactoryTest.class.getResource("/ClassLoaderTestD/TestD.jar");
+ CachingClassLoaderFactoryTest.class.getResource("/ClassLoaderTestD/TestD.jar");
assertNotNull(jarDOrigLocation);
jarEOrigLocation =
- LocalCachingContextClassLoaderFactoryTest.class.getResource("/ClassLoaderTestE/TestE.jar");
+ CachingClassLoaderFactoryTest.class.getResource("/ClassLoaderTestE/TestE.jar");
assertNotNull(jarEOrigLocation);
// Put B into HDFS
@@ -140,23 +139,23 @@ public class LocalCachingContextClassLoaderFactoryTest {
jetty = TestUtils.getJetty(jarCParentDirectory);
final URL jarCJettyLocation = jetty.getURI().resolve("TestC.jar").toURL();
- // ContextDefinition with all jars
- var allJarsDef = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation,
+ // manifest with all jars
+ var allJars = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation,
jarBHdfsLocation, jarCJettyLocation, jarDOrigLocation);
- String allJarsDefJson = allJarsDef.toJson();
+ String allJarsJson = allJars.toJson();
- // Create local context definition in jar C directory
- File localDefFile = jarCParentDirectory.resolve("allContextDefinition.json").toFile();
- Files.writeString(localDefFile.toPath(), allJarsDefJson);
- assertTrue(Files.exists(localDefFile.toPath()));
+ // Create local manifest in jar C directory
+ File localFile = jarCParentDirectory.resolve("all.json").toFile();
+ Files.writeString(localFile.toPath(), allJarsJson);
+ assertTrue(Files.exists(localFile.toPath()));
- var hdfsDefFile = new org.apache.hadoop.fs.Path("/allContextDefinition.json");
- fs.copyFromLocalFile(new org.apache.hadoop.fs.Path(localDefFile.toURI()), hdfsDefFile);
- assertTrue(fs.exists(hdfsDefFile));
+ var hdfsFile = new org.apache.hadoop.fs.Path("/all.json");
+ fs.copyFromLocalFile(new org.apache.hadoop.fs.Path(localFile.toURI()), hdfsFile);
+ assertTrue(fs.exists(hdfsFile));
- localAllContext = localDefFile.toURI().toURL();
- hdfsAllContext = new URL(fs.getUri().toString() + hdfsDefFile.toUri().toString());
- jettyAllContext = jetty.getURI().resolve("allContextDefinition.json").toURL();
+ localAllUrl = localFile.toURI().toURL();
+ hdfsAllUrl = new URL(fs.getUri().toString() + hdfsFile.toUri().toString());
+ jettyAllUrl = jetty.getURI().resolve("all.json").toURL();
classA = new TestClassInfo("test.TestObjectA", "Hello from A");
classB = new TestClassInfo("test.TestObjectB", "Hello from B");
@@ -179,9 +178,9 @@ public class LocalCachingContextClassLoaderFactoryTest {
public void beforeEach(TestInfo info) throws Exception {
baseCacheDir = tempDir.resolve(info.getTestMethod().orElseThrow().getName());
ConfigurationCopy acuConf = new ConfigurationCopy(
- Map.of(CACHE_DIR_PROPERTY, baseCacheDir.toAbsolutePath().toUri().toURL().toExternalForm(),
- UPDATE_FAILURE_GRACE_PERIOD_MINS, "1", ALLOWED_URLS_PATTERN, ".*"));
- FACTORY = new LocalCachingContextClassLoaderFactory();
+ Map.of(PROP_CACHE_DIR, baseCacheDir.toAbsolutePath().toUri().toURL().toExternalForm(),
+ PROP_GRACE_PERIOD, "1", PROP_ALLOWED_URLS, ".*"));
+ FACTORY = new CachingClassLoaderFactory();
FACTORY.init(() -> new ConfigurationImpl(acuConf));
}
@@ -189,31 +188,31 @@ public class LocalCachingContextClassLoaderFactoryTest {
public void testAllowedUrls() throws Exception {
// use a different factory than other tests; only allow file: URLs
ConfigurationCopy acuConf = new ConfigurationCopy(
- Map.of(CACHE_DIR_PROPERTY, baseCacheDir.toAbsolutePath().toUri().toURL().toExternalForm(),
- ALLOWED_URLS_PATTERN, "file:.*"));
- var factory = new LocalCachingContextClassLoaderFactory();
+ Map.of(PROP_CACHE_DIR, baseCacheDir.toAbsolutePath().toUri().toURL().toExternalForm(),
+ PROP_ALLOWED_URLS, "file:.*"));
+ var factory = new CachingClassLoaderFactory();
factory.init(() -> new ConfigurationImpl(acuConf));
// case 1: all URLs pass (normal case, covered by other tests)
- // case 2: context definition URL fails to match the pattern
+ // case 2: manifest URL fails to match the pattern
var ex = assertThrows(ContextClassLoaderException.class,
- () -> factory.getClassLoader(hdfsAllContext.toExternalForm()));
+ () -> factory.getClassLoader(hdfsAllUrl.toExternalForm()));
assertTrue(ex.getCause() instanceof IllegalArgumentException);
- assertTrue(ex.getCause().getMessage().contains("Context definition location (hdfs:"));
+ assertTrue(ex.getCause().getMessage().contains("Context manifest URL (hdfs:"));
- // case 3a: context definition URL matches, but resource URL should fail to match the pattern,
+ // case 3a: manifest URL matches, but resource URL should fail to match the pattern,
// but it works anyway, because the resources were downloaded already by a different instance
// (in this case, by a less restrictive FACTORY instance) and no new connection is made
- FACTORY.getClassLoader(hdfsAllContext.toExternalForm());
- factory.getClassLoader(localAllContext.toExternalForm()); // same resources
+ FACTORY.getClassLoader(hdfsAllUrl.toExternalForm());
+ factory.getClassLoader(localAllUrl.toExternalForm()); // same resources
- // case 3b: context definition URL matches, but resource URL fails to match the pattern
- // in this case, we use a new context definition, with a resource that doesn't exist locally
+ // case 3b: manifest URL matches, but resource URL fails to match the pattern
+ // in this case, we use a new manifest, with a resource that doesn't exist locally
var newResources = new LinkedHashSet<Resource>();
var badUrl = "http://localhost/some/path";
newResources.add(new Resource(new URL(badUrl), "MD5", BAD_MD5));
- var context2 = new ContextDefinition(MONITOR_INTERVAL_SECS, newResources);
+ var context2 = new Manifest(MONITOR_INTERVAL_SECS, newResources);
var disallowedContext = tempDir.resolve("context-with-disallowed-resource-url.json");
Files.writeString(disallowedContext, context2.toJson());
ex = assertThrows(ContextClassLoaderException.class,
@@ -221,32 +220,34 @@ public class LocalCachingContextClassLoaderFactoryTest {
assertTrue(ex.getCause() instanceof IllegalStateException);
assertTrue(ex.getCause().getCause() instanceof ExecutionException);
assertTrue(ex.getCause().getCause().getCause() instanceof IllegalArgumentException);
- assertTrue(ex.getCause().getCause().getCause().getMessage().contains(
- "Resource location (" + badUrl + ")"), ex.getCause().getCause().getCause()::getMessage);
+ assertTrue(
+ ex.getCause().getCause().getCause().getMessage()
+ .contains("Context resource URL (" + badUrl + ") not allowed by pattern ("),
+ ex.getCause().getCause().getCause()::getMessage);
// case 4: invalid regex for allowed url
ConfigurationCopy acuConf2 = new ConfigurationCopy(
- Map.of(CACHE_DIR_PROPERTY, baseCacheDir.toAbsolutePath().toUri().toURL().toExternalForm(),
- ALLOWED_URLS_PATTERN, "file:[a-z.*"));
- var factory2 = new LocalCachingContextClassLoaderFactory();
+ Map.of(PROP_CACHE_DIR, baseCacheDir.toAbsolutePath().toUri().toURL().toExternalForm(),
+ PROP_ALLOWED_URLS, "file:[a-z.*"));
+ var factory2 = new CachingClassLoaderFactory();
factory2.init(() -> new ConfigurationImpl(acuConf2));
ex = assertThrows(ContextClassLoaderException.class,
- () -> factory2.getClassLoader(hdfsAllContext.toExternalForm()));
+ () -> factory2.getClassLoader(hdfsAllUrl.toExternalForm()));
assertEquals(PatternSyntaxException.class, ex.getCause().getClass());
// case 5: no allowed pattern url is set
ConfigurationCopy acuConf3 = new ConfigurationCopy(
- Map.of(CACHE_DIR_PROPERTY, baseCacheDir.toAbsolutePath().toUri().toURL().toExternalForm()));
- var factory3 = new LocalCachingContextClassLoaderFactory();
+ Map.of(PROP_CACHE_DIR, baseCacheDir.toAbsolutePath().toUri().toURL().toExternalForm()));
+ var factory3 = new CachingClassLoaderFactory();
factory3.init(() -> new ConfigurationImpl(acuConf3));
ex = assertThrows(ContextClassLoaderException.class,
- () -> factory3.getClassLoader(hdfsAllContext.toExternalForm()));
- assertTrue(ex.getMessage().contains(ALLOWED_URLS_PATTERN + " not set"), ex::getMessage);
+ () -> factory3.getClassLoader(hdfsAllUrl.toExternalForm()));
+ assertTrue(ex.getMessage().contains(PROP_ALLOWED_URLS + " not set"), ex::getMessage);
}
@Test
public void testCreateFromLocal() throws Exception {
- final ClassLoader cl = FACTORY.getClassLoader(localAllContext.toString());
+ final ClassLoader cl = FACTORY.getClassLoader(localAllUrl.toString());
testClassLoads(cl, classA);
testClassLoads(cl, classB);
testClassLoads(cl, classC);
@@ -255,7 +256,7 @@ public class LocalCachingContextClassLoaderFactoryTest {
@Test
public void testCreateFromHdfs() throws Exception {
- final ClassLoader cl = FACTORY.getClassLoader(hdfsAllContext.toString());
+ final ClassLoader cl = FACTORY.getClassLoader(hdfsAllUrl.toString());
testClassLoads(cl, classA);
testClassLoads(cl, classB);
testClassLoads(cl, classC);
@@ -264,7 +265,7 @@ public class LocalCachingContextClassLoaderFactoryTest {
@Test
public void testCreateFromHttp() throws Exception {
- final ClassLoader cl = FACTORY.getClassLoader(jettyAllContext.toString());
+ final ClassLoader cl = FACTORY.getClassLoader(jettyAllUrl.toString());
testClassLoads(cl, classA);
testClassLoads(cl, classB);
testClassLoads(cl, classC);
@@ -272,7 +273,7 @@ public class LocalCachingContextClassLoaderFactoryTest {
}
@Test
- public void testInvalidContextDefinitionURL() {
+ public void testInvalidManifestURL() {
var ex =
assertThrows(ContextClassLoaderException.class, () -> FACTORY.getClassLoader("/not/a/URL"));
assertTrue(ex.getCause() instanceof UncheckedIOException);
@@ -281,43 +282,40 @@ public class LocalCachingContextClassLoaderFactoryTest {
}
@Test
- public void testInitialContextDefinitionEmpty() throws Exception {
- // Create a new context definition file in HDFS, but with no content
- final var def = createContextDefinitionFile(fs, "EmptyContextDefinitionFile.json", null);
- final URL emptyDefUrl = new URL(fs.getUri().toString() + def.toUri().toString());
+ public void testInitialManifestEmpty() throws Exception {
+ // Create a new manifest file in HDFS, but with no content
+ final var manifest = createManifestFile(fs, "empty.json", null);
+ final URL emptyUrl = new URL(fs.getUri().toString() + manifest.toUri().toString());
var ex = assertThrows(ContextClassLoaderException.class,
- () -> FACTORY.getClassLoader(emptyDefUrl.toString()));
+ () -> FACTORY.getClassLoader(emptyUrl.toString()));
assertTrue(ex.getCause() instanceof UncheckedIOException);
assertTrue(ex.getCause().getCause() instanceof EOFException);
- assertEquals(
- "InputStream does not contain a valid ContextDefinition at " + emptyDefUrl.toString(),
+ assertEquals("InputStream does not contain a valid manifest at " + emptyUrl.toString(),
ex.getCause().getCause().getMessage());
}
@Test
public void testInitialInvalidJson() throws Exception {
- // Create a new context definition file in HDFS, but with invalid content
- var def = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
+ // Create a new manifest in HDFS, but with invalid content
+ var manifest = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
// write out invalid json
- final var invalid = createContextDefinitionFile(fs, "InvalidContextDefinitionFile.json",
- def.toJson().substring(0, 4));
- final URL invalidDefUrl = new URL(fs.getUri().toString() + invalid.toUri().toString());
+ final var invalid = createManifestFile(fs, "invalid.json", manifest.toJson().substring(0, 4));
+ final URL invalidUrl = new URL(fs.getUri().toString() + invalid.toUri().toString());
var ex = assertThrows(ContextClassLoaderException.class,
- () -> FACTORY.getClassLoader(invalidDefUrl.toString()));
+ () -> FACTORY.getClassLoader(invalidUrl.toString()));
assertTrue(ex.getCause() instanceof JsonSyntaxException);
assertTrue(ex.getCause().getCause() instanceof EOFException);
}
@Test
public void testInitial() throws Exception {
- var def = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
- final var initial =
- createContextDefinitionFile(fs, "InitialContextDefinitionFile.json", def.toJson());
- final URL initialDefUrl = new URL(fs.getUri().toString() + initial.toUri().toString());
+ var manifest = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
+ final var initial = createManifestFile(fs, "initial.json", manifest.toJson());
+ final URL initialUrl = new URL(fs.getUri().toString() + initial.toUri().toString());
- ClassLoader cl = FACTORY.getClassLoader(initialDefUrl.toString());
+ ClassLoader cl = FACTORY.getClassLoader(initialUrl.toString());
testClassLoads(cl, classA);
testClassFailsToLoad(cl, classB);
@@ -336,17 +334,16 @@ public class LocalCachingContextClassLoaderFactoryTest {
Files.copy(jarAPath, jarACopy, REPLACE_EXISTING);
assertTrue(Files.exists(jarACopy));
- var def = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarACopy.toUri().toURL());
+ var manifest = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarACopy.toUri().toURL());
Files.delete(jarACopy);
assertTrue(!Files.exists(jarACopy));
- final var initial = createContextDefinitionFile(fs,
- "InitialContextDefinitionFileMissingResource.json", def.toJson());
- final URL initialDefUrl = new URL(fs.getUri().toString() + initial.toUri().toString());
+ final var initial = createManifestFile(fs, "missing-resource.json", manifest.toJson());
+ final URL initialUrl = new URL(fs.getUri().toString() + initial.toUri().toString());
var ex = assertThrows(ContextClassLoaderException.class,
- () -> FACTORY.getClassLoader(initialDefUrl.toString()));
+ () -> FACTORY.getClassLoader(initialUrl.toString()));
boolean foundExpectedException = false;
var cause = ex.getCause();
do {
@@ -365,17 +362,16 @@ public class LocalCachingContextClassLoaderFactoryTest {
resources.add(new Resource(jarAOrigLocation, "MD5", BAD_MD5));
// remove the file:// prefix from the URL
- String goodJson = new ContextDefinition(MONITOR_INTERVAL_SECS, resources).toJson();
+ String goodJson = new Manifest(MONITOR_INTERVAL_SECS, resources).toJson();
String badJson =
goodJson.replace(jarAOrigLocation.toString(), jarAOrigLocation.toString().substring(6));
assertNotEquals(goodJson, badJson);
- final var initial =
- createContextDefinitionFile(fs, "InitialContextDefinitionBadResourceURL.json", badJson);
- final URL initialDefUrl = new URL(fs.getUri().toString() + initial.toUri().toString());
+ final var initial = createManifestFile(fs, "bad-resource-url.json", badJson);
+ final URL initialUrl = new URL(fs.getUri().toString() + initial.toUri().toString());
var ex = assertThrows(ContextClassLoaderException.class,
- () -> FACTORY.getClassLoader(initialDefUrl.toString()));
+ () -> FACTORY.getClassLoader(initialUrl.toString()));
assertTrue(ex.getMessage().startsWith("Error getting classloader for context:"),
ex::getMessage);
assertTrue(ex.getCause() instanceof JsonSyntaxException);
@@ -390,14 +386,13 @@ public class LocalCachingContextClassLoaderFactoryTest {
LinkedHashSet<Resource> resources = new LinkedHashSet<>();
resources.add(r);
- var def = new ContextDefinition(MONITOR_INTERVAL_SECS, resources);
+ var manifest = new Manifest(MONITOR_INTERVAL_SECS, resources);
- final var initial = createContextDefinitionFile(fs,
- "InitialContextDefinitionBadResourceChecksum.json", def.toJson());
- final URL initialDefUrl = new URL(fs.getUri().toString() + initial.toUri().toString());
+ final var initial = createManifestFile(fs, "bad-resource-checksum.json", manifest.toJson());
+ final URL initialUrl = new URL(fs.getUri().toString() + initial.toUri().toString());
var ex = assertThrows(ContextClassLoaderException.class,
- () -> FACTORY.getClassLoader(initialDefUrl.toString()));
+ () -> FACTORY.getClassLoader(initialUrl.toString()));
assertTrue(ex.getCause() instanceof IllegalStateException);
assertTrue(ex.getCause().getMessage().startsWith("Error copying resource from file:"),
ex::getMessage);
@@ -407,32 +402,31 @@ public class LocalCachingContextClassLoaderFactoryTest {
ex.getCause().getCause().getCause()::getMessage);
assertTrue(
ex.getCause().getCause().getCause().getMessage()
- .endsWith("TestA.jar does not match checksum in context definition " + BAD_MD5),
+ .endsWith("TestA.jar does not match checksum in the manifest " + BAD_MD5),
ex.getCause().getCause().getCause()::getMessage);
}
@Test
public void testUpdate() throws Exception {
- final var def = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
- final var defFilePath =
- createContextDefinitionFile(fs, "UpdateContextDefinitionFile.json", def.toJson());
- final URL updateDefUrl = new URL(fs.getUri().toString() + defFilePath.toUri().toString());
+ final var manifest = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
+ final var manifestPath = createManifestFile(fs, "update.json", manifest.toJson());
+ final URL manifestUrl = new URL(fs.getUri().toString() + manifestPath.toUri().toString());
- final ClassLoader cl = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl = FACTORY.getClassLoader(manifestUrl.toString());
testClassLoads(cl, classA);
testClassFailsToLoad(cl, classB);
testClassFailsToLoad(cl, classC);
testClassFailsToLoad(cl, classD);
- // Update the contents of the context definition json file
- var updateDef = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarDOrigLocation);
- updateContextDefinitionFile(fs, defFilePath, updateDef.toJson());
+ // Update the contents
+ var update = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarDOrigLocation);
+ updateManifestFile(fs, manifestPath, update.toJson());
// wait 2x the monitor interval
Thread.sleep(MONITOR_INTERVAL_SECS * 2 * 1000);
- final ClassLoader cl2 = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl2 = FACTORY.getClassLoader(manifestUrl.toString());
assertNotEquals(cl, cl2);
@@ -444,26 +438,25 @@ public class LocalCachingContextClassLoaderFactoryTest {
@Test
public void testUpdateSameClassNameDifferentContent() throws Exception {
- final var def = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
- final var defFilePath =
- createContextDefinitionFile(fs, "UpdateContextDefinitionFile.json", def.toJson());
- final URL updateDefUrl = new URL(fs.getUri().toString() + defFilePath.toUri().toString());
+ final var manifest = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
+ final var manifestPath = createManifestFile(fs, "update-same-name.json", manifest.toJson());
+ final URL manifestUrl = new URL(fs.getUri().toString() + manifestPath.toUri().toString());
- final ClassLoader cl = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl = FACTORY.getClassLoader(manifestUrl.toString());
testClassLoads(cl, classA);
testClassFailsToLoad(cl, classB);
testClassFailsToLoad(cl, classC);
testClassFailsToLoad(cl, classD);
- // Update the contents of the context definition json file
- var updateDef = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarEOrigLocation);
- updateContextDefinitionFile(fs, defFilePath, updateDef.toJson());
+ // Update the contents
+ var update = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarEOrigLocation);
+ updateManifestFile(fs, manifestPath, update.toJson());
// wait 2x the monitor interval
Thread.sleep(MONITOR_INTERVAL_SECS * 2 * 1000);
- final ClassLoader cl2 = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl2 = FACTORY.getClassLoader(manifestUrl.toString());
assertNotEquals(cl, cl2);
@@ -476,26 +469,25 @@ public class LocalCachingContextClassLoaderFactoryTest {
}
@Test
- public void testUpdateContextDefinitionEmpty() throws Exception {
- final var def = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
- final var defFilePath =
- createContextDefinitionFile(fs, "UpdateEmptyContextDefinitionFile.json", def.toJson());
- final URL updateDefUrl = new URL(fs.getUri().toString() + defFilePath.toUri().toString());
+ public void testUpdateManifestEmpty() throws Exception {
+ final var manifest = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
+ final var manifestPath = createManifestFile(fs, "update-empty.json", manifest.toJson());
+ final URL manifestUrl = new URL(fs.getUri().toString() + manifestPath.toUri().toString());
- final ClassLoader cl = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl = FACTORY.getClassLoader(manifestUrl.toString());
testClassLoads(cl, classA);
testClassFailsToLoad(cl, classB);
testClassFailsToLoad(cl, classC);
testClassFailsToLoad(cl, classD);
- // Update the contents of the context definition json file with an empty file
- updateContextDefinitionFile(fs, defFilePath, null);
+ // Update the contents with an empty file
+ updateManifestFile(fs, manifestPath, null);
// wait 2x the monitor interval
Thread.sleep(MONITOR_INTERVAL_SECS * 2 * 1000);
- final ClassLoader cl2 = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl2 = FACTORY.getClassLoader(manifestUrl.toString());
// validate that the classloader has not updated
assertEquals(cl, cl2);
@@ -508,12 +500,12 @@ public class LocalCachingContextClassLoaderFactoryTest {
@Test
public void testUpdateNonExistentResource() throws Exception {
- final var def = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
- final var defFilePath =
- createContextDefinitionFile(fs, "UpdateNonExistentResource.json", def.toJson());
- final URL updateDefUrl = new URL(fs.getUri().toString() + defFilePath.toUri().toString());
+ final var manifest = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
+ final var manifestPath =
+ createManifestFile(fs, "UpdateNonExistentResource.json", manifest.toJson());
+ final URL manifestUrl = new URL(fs.getUri().toString() + manifestPath.toUri().toString());
- final ClassLoader cl = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl = FACTORY.getClassLoader(manifestUrl.toString());
testClassLoads(cl, classA);
testClassFailsToLoad(cl, classB);
@@ -521,7 +513,7 @@ public class LocalCachingContextClassLoaderFactoryTest {
testClassFailsToLoad(cl, classD);
// copy jarA to jarACopy
- // create a ContextDefinition that references it
+ // create a manifest that references it
// delete jarACopy
var jarAPath = Path.of(jarAOrigLocation.toURI());
var jarAPathParent = jarAPath.getParent();
@@ -530,16 +522,16 @@ public class LocalCachingContextClassLoaderFactoryTest {
assertTrue(!Files.exists(jarACopy));
Files.copy(jarAPath, jarACopy, REPLACE_EXISTING);
assertTrue(Files.exists(jarACopy));
- var def2 = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarACopy.toUri().toURL());
+ var manifest2 = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarACopy.toUri().toURL());
Files.delete(jarACopy);
assertTrue(!Files.exists(jarACopy));
- updateContextDefinitionFile(fs, defFilePath, def2.toJson());
+ updateManifestFile(fs, manifestPath, manifest2.toJson());
// wait 2x the monitor interval
Thread.sleep(MONITOR_INTERVAL_SECS * 2 * 1000);
- final ClassLoader cl2 = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl2 = FACTORY.getClassLoader(manifestUrl.toString());
// validate that the classloader has not updated
assertEquals(cl, cl2);
@@ -551,12 +543,12 @@ public class LocalCachingContextClassLoaderFactoryTest {
@Test
public void testUpdateBadResourceChecksum() throws Exception {
- final var def = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
- final var defFilePath =
- createContextDefinitionFile(fs, "UpdateBadResourceChecksum.json", def.toJson());
- final URL updateDefUrl = new URL(fs.getUri().toString() + defFilePath.toUri().toString());
+ final var manifest = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
+ final var manifestPath =
+ createManifestFile(fs, "UpdateBadResourceChecksum.json", manifest.toJson());
+ final URL manifestUrl = new URL(fs.getUri().toString() + manifestPath.toUri().toString());
- final ClassLoader cl = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl = FACTORY.getClassLoader(manifestUrl.toString());
testClassLoads(cl, classA);
testClassFailsToLoad(cl, classB);
@@ -567,14 +559,14 @@ public class LocalCachingContextClassLoaderFactoryTest {
LinkedHashSet<Resource> resources = new LinkedHashSet<>();
resources.add(r);
- var def2 = new ContextDefinition(MONITOR_INTERVAL_SECS, resources);
+ var manifest2 = new Manifest(MONITOR_INTERVAL_SECS, resources);
- updateContextDefinitionFile(fs, defFilePath, def2.toJson());
+ updateManifestFile(fs, manifestPath, manifest2.toJson());
// wait 2x the monitor interval
Thread.sleep(MONITOR_INTERVAL_SECS * 2 * 1000);
- final ClassLoader cl2 = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl2 = FACTORY.getClassLoader(manifestUrl.toString());
// validate that the classloader has not updated
assertEquals(cl, cl2);
@@ -586,12 +578,12 @@ public class LocalCachingContextClassLoaderFactoryTest {
@Test
public void testUpdateBadResourceURL() throws Exception {
- final var def = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
- final var defFilePath =
- createContextDefinitionFile(fs, "UpdateBadResourceChecksum.json", def.toJson());
- final URL updateDefUrl = new URL(fs.getUri().toString() + defFilePath.toUri().toString());
+ final var manifest = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
+ final var manifestPath =
+ createManifestFile(fs, "UpdateBadResourceChecksum.json", manifest.toJson());
+ final URL manifestUrl = new URL(fs.getUri().toString() + manifestPath.toUri().toString());
- final ClassLoader cl = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl = FACTORY.getClassLoader(manifestUrl.toString());
testClassLoads(cl, classA);
testClassFailsToLoad(cl, classB);
@@ -601,17 +593,17 @@ public class LocalCachingContextClassLoaderFactoryTest {
// remove the file:// prefix from the URL
LinkedHashSet<Resource> resources = new LinkedHashSet<>();
resources.add(new Resource(jarAOrigLocation, "MD5", BAD_MD5));
- String goodJson = new ContextDefinition(MONITOR_INTERVAL_SECS, resources).toJson();
+ String goodJson = new Manifest(MONITOR_INTERVAL_SECS, resources).toJson();
String badJson =
goodJson.replace(jarAOrigLocation.toString(), jarAOrigLocation.toString().substring(6));
assertNotEquals(goodJson, badJson);
- updateContextDefinitionFile(fs, defFilePath, badJson);
+ updateManifestFile(fs, manifestPath, badJson);
// wait 2x the monitor interval
Thread.sleep(MONITOR_INTERVAL_SECS * 2 * 1000);
- final ClassLoader cl2 = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl2 = FACTORY.getClassLoader(manifestUrl.toString());
// validate that the classloader has not updated
assertEquals(cl, cl2);
@@ -623,25 +615,24 @@ public class LocalCachingContextClassLoaderFactoryTest {
@Test
public void testUpdateInvalidJson() throws Exception {
- final var def = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
- final var defFilePath =
- createContextDefinitionFile(fs, "UpdateInvalidContextDefinitionFile.json", def.toJson());
- final URL updateDefUrl = new URL(fs.getUri().toString() + defFilePath.toUri().toString());
+ final var manifest = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
+ final var manifestPath = createManifestFile(fs, "update-invalid.json", manifest.toJson());
+ final URL updateUrl = new URL(fs.getUri().toString() + manifestPath.toUri().toString());
- final ClassLoader cl = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl = FACTORY.getClassLoader(updateUrl.toString());
testClassLoads(cl, classA);
testClassFailsToLoad(cl, classB);
testClassFailsToLoad(cl, classC);
testClassFailsToLoad(cl, classD);
- var updateDef = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarDOrigLocation);
- updateContextDefinitionFile(fs, defFilePath, updateDef.toJson().substring(0, 4));
+ var update = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarDOrigLocation);
+ updateManifestFile(fs, manifestPath, update.toJson().substring(0, 4));
// wait 2x the monitor interval
Thread.sleep(MONITOR_INTERVAL_SECS * 2 * 1000);
- final ClassLoader cl2 = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl2 = FACTORY.getClassLoader(updateUrl.toString());
// validate that the classloader has not updated
assertEquals(cl, cl2);
@@ -650,13 +641,13 @@ public class LocalCachingContextClassLoaderFactoryTest {
testClassFailsToLoad(cl2, classC);
testClassFailsToLoad(cl2, classD);
- // Re-write the updated context definition such that it is now valid
- updateContextDefinitionFile(fs, defFilePath, updateDef.toJson());
+ // Re-write the updated file such that it is now valid
+ updateManifestFile(fs, manifestPath, update.toJson());
// wait 2x the monitor interval
Thread.sleep(MONITOR_INTERVAL_SECS * 2 * 1000);
- final ClassLoader cl3 = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl3 = FACTORY.getClassLoader(updateUrl.toString());
assertEquals(cl, cl2);
assertNotEquals(cl, cl3);
@@ -668,13 +659,12 @@ public class LocalCachingContextClassLoaderFactoryTest {
@Test
public void testChangingContext() throws Exception {
- var def = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation,
+ var manifest = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation,
jarBOrigLocation, jarCOrigLocation, jarDOrigLocation);
- final var update =
- createContextDefinitionFile(fs, "UpdateChangingContextDefinition.json", def.toJson());
- final URL updatedDefUrl = new URL(fs.getUri().toString() + update.toUri().toString());
+ final var manifestPath = createManifestFile(fs, "update-changing.json", manifest.toJson());
+ final URL manifestUrl = new URL(fs.getUri().toString() + manifestPath.toUri().toString());
- final ClassLoader cl = FACTORY.getClassLoader(updatedDefUrl.toString());
+ final ClassLoader cl = FACTORY.getClassLoader(manifestUrl.toString());
testClassLoads(cl, classA);
testClassLoads(cl, classB);
testClassLoads(cl, classC);
@@ -694,15 +684,15 @@ public class LocalCachingContextClassLoaderFactoryTest {
Collections.shuffle(updatedList);
final URL removed = updatedList.remove(0);
- // Update the contents of the context definition json file
- var updateDef = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512",
- updatedList.toArray(new URL[0]));
- updateContextDefinitionFile(fs, update, updateDef.toJson());
+ // Update the contents
+ var update =
+ Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", updatedList.toArray(new URL[0]));
+ updateManifestFile(fs, manifestPath, update.toJson());
// wait 2x the monitor interval
Thread.sleep(MONITOR_INTERVAL_SECS * 2 * 1000);
- final ClassLoader updatedClassLoader = FACTORY.getClassLoader(updatedDefUrl.toString());
+ final ClassLoader updatedClassLoader = FACTORY.getClassLoader(manifestUrl.toString());
if (updatedList.equals(priorList)) {
assertEquals(priorCL, updatedClassLoader);
@@ -740,12 +730,12 @@ public class LocalCachingContextClassLoaderFactoryTest {
@Test
public void testGracePeriod() throws Exception {
- final var def = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
- final var defFilePath =
- createContextDefinitionFile(fs, "UpdateNonExistentResource.json", def.toJson());
- final URL updateDefUrl = new URL(fs.getUri().toString() + defFilePath.toUri().toString());
+ final var manifest = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation);
+ final var manifestPath =
+ createManifestFile(fs, "UpdateNonExistentResource.json", manifest.toJson());
+ final URL manifestUrl = new URL(fs.getUri().toString() + manifestPath.toUri().toString());
- final ClassLoader cl = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl = FACTORY.getClassLoader(manifestUrl.toString());
testClassLoads(cl, classA);
testClassFailsToLoad(cl, classB);
@@ -753,7 +743,7 @@ public class LocalCachingContextClassLoaderFactoryTest {
testClassFailsToLoad(cl, classD);
// copy jarA to jarACopy
- // create a ContextDefinition that references it
+ // create a manifest that references it
// delete jarACopy
var jarAPath = Path.of(jarAOrigLocation.toURI());
var jarAPathParent = jarAPath.getParent();
@@ -762,16 +752,16 @@ public class LocalCachingContextClassLoaderFactoryTest {
assertTrue(!Files.exists(jarACopy));
Files.copy(jarAPath, jarACopy, REPLACE_EXISTING);
assertTrue(Files.exists(jarACopy));
- var def2 = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarACopy.toUri().toURL());
+ var manifest2 = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarACopy.toUri().toURL());
Files.delete(jarACopy);
assertTrue(!Files.exists(jarACopy));
- updateContextDefinitionFile(fs, defFilePath, def2.toJson());
+ updateManifestFile(fs, manifestPath, manifest2.toJson());
// wait 2x the monitor interval
Thread.sleep(MONITOR_INTERVAL_SECS * 2 * 1000);
- final ClassLoader cl2 = FACTORY.getClassLoader(updateDefUrl.toString());
+ final ClassLoader cl2 = FACTORY.getClassLoader(manifestUrl.toString());
// validate that the classloader has not updated
assertEquals(cl, cl2);
@@ -784,7 +774,7 @@ public class LocalCachingContextClassLoaderFactoryTest {
Thread.sleep(120_000);
var ex = assertThrows(ContextClassLoaderException.class,
- () -> FACTORY.getClassLoader(updateDefUrl.toString()));
+ () -> FACTORY.getClassLoader(manifestUrl.toString()));
boolean foundExpectedException = false;
var cause = ex.getCause();
do {
@@ -799,42 +789,41 @@ public class LocalCachingContextClassLoaderFactoryTest {
@Test
public void testExternalFileModification() throws Exception {
- var def = ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation,
+ var manifest = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarAOrigLocation,
jarBOrigLocation, jarCOrigLocation, jarDOrigLocation);
- final var update =
- createContextDefinitionFile(fs, "UpdateChangingContextDefinition.json", def.toJson());
- final URL updatedDefUrl = new URL(fs.getUri().toString() + update.toUri().toString());
+ final var manifestPath =
+ createManifestFile(fs, "update-external-modified.json", manifest.toJson());
+ final URL manifestUrl = new URL(fs.getUri().toString() + manifestPath.toUri().toString());
- final ClassLoader cl = FACTORY.getClassLoader(updatedDefUrl.toString());
+ final ClassLoader cl = FACTORY.getClassLoader(manifestUrl.toString());
testClassLoads(cl, classA);
testClassLoads(cl, classB);
testClassLoads(cl, classC);
testClassLoads(cl, classD);
var resources = baseCacheDir.resolve(RESOURCES_DIR);
- List<Path> files =
- def.getResources().stream().map(r -> resources.resolve(LocalStore.localResourceName(r)))
- .limit(2).collect(Collectors.toList());
+ List<Path> files = manifest.getResources().stream()
+ .map(r -> resources.resolve(LocalStore.localResourceName(r))).limit(2)
+ .collect(Collectors.toList());
assertEquals(2, files.size());
// overwrite one downloaded jar with others content
Files.copy(files.get(0), files.get(1), REPLACE_EXISTING);
- final var update2 =
- createContextDefinitionFile(fs, "UpdateChangingContextDefinition2.json", def.toJson());
- final URL updatedDefUrl2 = new URL(fs.getUri().toString() + update2.toUri().toString());
+ final var update2 = createManifestFile(fs, "update-external-modified2.json", manifest.toJson());
+ final URL updateUrl2 = new URL(fs.getUri().toString() + update2.toUri().toString());
// The classloader should fail to create because one of the files in the local filesystem cache
// has a checksum mismatch
var exception = assertThrows(ContextClassLoaderException.class,
- () -> FACTORY.getClassLoader(updatedDefUrl2.toString()));
+ () -> FACTORY.getClassLoader(updateUrl2.toString()));
assertTrue(exception.getMessage().contains("Checksum"), exception::getMessage);
// clean up corrupt file
Files.delete(files.get(1));
// ensure it works now
- FACTORY.getClassLoader(updatedDefUrl2.toString());
+ FACTORY.getClassLoader(updateUrl2.toString());
}
@Test
@@ -857,7 +846,7 @@ public class LocalCachingContextClassLoaderFactoryTest {
return null;
});
- var def = ContextDefinition.create(100, "SHA-512", jarAOrigLocation, jarBOrigLocation,
+ var manifest = Manifest.create(100, "SHA-512", jarAOrigLocation, jarBOrigLocation,
jarCOrigLocation, jarDOrigLocation);
List<Future<?>> futures = new ArrayList<>();
@@ -871,11 +860,11 @@ public class LocalCachingContextClassLoaderFactoryTest {
int j = 0;
ClassLoader lastCl = null;
while (!timer.hasElapsed(10, TimeUnit.SECONDS)) {
- var contextFile = tempDir.resolve("context-cd-" + threadNum + "_" + j + ".json");
- Files.writeString(contextFile, def.toJson());
- var contextUrl = contextFile.toUri().toURL().toExternalForm();
+ var file = tempDir.resolve("context-cd-" + threadNum + "_" + j + ".json");
+ Files.writeString(file, manifest.toJson());
+ var url = file.toUri().toURL().toExternalForm();
- final ClassLoader cl = FACTORY.getClassLoader(contextUrl);
+ final ClassLoader cl = FACTORY.getClassLoader(url);
// This test is assuming that each call above creates a new classloader which in turn
// creates new hard links. This is checking that assumption in case the impl changes and
// this test needs to be reevaluated.
diff --git a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/util/DeduplicationCacheTest.java b/modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/DeduplicationCacheTest.java
similarity index 98%
rename from modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/util/DeduplicationCacheTest.java
rename to modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/DeduplicationCacheTest.java
index 5d17fea..79179bb 100644
--- a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/util/DeduplicationCacheTest.java
+++ b/modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/DeduplicationCacheTest.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.accumulo.classloader.lcc.util;
+package org.apache.accumulo.classloader.ccl;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
diff --git a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/util/LocalStoreTest.java b/modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/LocalStoreTest.java
similarity index 77%
rename from modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/util/LocalStoreTest.java
rename to modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/LocalStoreTest.java
index 9f4e223..22a1cb3 100644
--- a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/util/LocalStoreTest.java
+++ b/modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/LocalStoreTest.java
@@ -16,15 +16,16 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.accumulo.classloader.lcc.util;
-
-import static org.apache.accumulo.classloader.lcc.TestUtils.testClassFailsToLoad;
-import static org.apache.accumulo.classloader.lcc.TestUtils.testClassLoads;
-import static org.apache.accumulo.classloader.lcc.util.LccUtils.checksumForFileName;
-import static org.apache.accumulo.classloader.lcc.util.LocalStore.CONTEXTS_DIR;
-import static org.apache.accumulo.classloader.lcc.util.LocalStore.RESOURCES_DIR;
-import static org.apache.accumulo.classloader.lcc.util.LocalStore.WORKING_DIR;
-import static org.apache.accumulo.classloader.lcc.util.LocalStore.localResourceName;
+package org.apache.accumulo.classloader.ccl;
+
+import static org.apache.accumulo.classloader.ccl.CachingClassLoaderFactory.createClassLoader;
+import static org.apache.accumulo.classloader.ccl.LocalStore.MANIFESTS_DIR;
+import static org.apache.accumulo.classloader.ccl.LocalStore.RESOURCES_DIR;
+import static org.apache.accumulo.classloader.ccl.LocalStore.WORKING_DIR;
+import static org.apache.accumulo.classloader.ccl.LocalStore.checksumForFileName;
+import static org.apache.accumulo.classloader.ccl.LocalStore.localResourceName;
+import static org.apache.accumulo.classloader.ccl.TestUtils.testClassFailsToLoad;
+import static org.apache.accumulo.classloader.ccl.TestUtils.testClassLoads;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotEquals;
@@ -41,10 +42,9 @@ import java.util.concurrent.CountDownLatch;
import java.util.function.BiConsumer;
import java.util.stream.Collectors;
-import org.apache.accumulo.classloader.lcc.TestUtils;
-import org.apache.accumulo.classloader.lcc.TestUtils.TestClassInfo;
-import org.apache.accumulo.classloader.lcc.definition.ContextDefinition;
-import org.apache.accumulo.classloader.lcc.definition.Resource;
+import org.apache.accumulo.classloader.ccl.TestUtils.TestClassInfo;
+import org.apache.accumulo.classloader.ccl.manifest.Manifest;
+import org.apache.accumulo.classloader.ccl.manifest.Resource;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.eclipse.jetty.server.Server;
@@ -67,7 +67,7 @@ public class LocalStoreTest {
private static final int MONITOR_INTERVAL_SECS = 5;
private static MiniDFSCluster hdfs;
private static Server jetty;
- private static ContextDefinition def;
+ private static Manifest manifest;
private static TestClassInfo classA;
private static TestClassInfo classB;
private static TestClassInfo classC;
@@ -98,7 +98,7 @@ public class LocalStoreTest {
jetty = TestUtils.getJetty(jarCParentDirectory);
final URL jarCNewLocation = jetty.getURI().resolve("TestC.jar").toURL();
- // Create ContextDefinition with all three resources
+ // Create with all three resources
final LinkedHashSet<Resource> resources = new LinkedHashSet<>();
resources.add(new Resource(jarAOrigLocation, "SHA-256",
TestUtils.computeResourceChecksum("SHA-256", jarAOrigLocation)));
@@ -107,7 +107,7 @@ public class LocalStoreTest {
resources.add(new Resource(jarCNewLocation, "SHA-1",
TestUtils.computeResourceChecksum("SHA-1", jarCOrigLocation)));
- def = new ContextDefinition(MONITOR_INTERVAL_SECS, resources);
+ manifest = new Manifest(MONITOR_INTERVAL_SECS, resources);
classA = new TestClassInfo("test.TestObjectA", "Hello from A");
classB = new TestClassInfo("test.TestObjectB", "Hello from B");
classC = new TestClassInfo("test.TestObjectC", "Hello from C");
@@ -141,17 +141,17 @@ public class LocalStoreTest {
@Test
public void testCreateBaseDirs() throws Exception {
// these dirs are documented, so need to update docs when changing constant values
+ assertEquals("manifests", MANIFESTS_DIR);
assertEquals("resources", RESOURCES_DIR);
assertEquals("working", WORKING_DIR);
- assertEquals("contexts", CONTEXTS_DIR);
assertFalse(Files.exists(baseCacheDir));
var localStore = new LocalStore(baseCacheDir, ALLOW_ALL_URLS);
assertTrue(Files.exists(baseCacheDir));
- assertTrue(Files.exists(baseCacheDir.resolve(CONTEXTS_DIR)));
+ assertTrue(Files.exists(baseCacheDir.resolve(MANIFESTS_DIR)));
assertTrue(Files.exists(baseCacheDir.resolve(RESOURCES_DIR)));
assertTrue(Files.exists(baseCacheDir.resolve(WORKING_DIR)));
- assertEquals(baseCacheDir.resolve(CONTEXTS_DIR), localStore.contextsDir());
+ assertEquals(baseCacheDir.resolve(MANIFESTS_DIR), localStore.manifestsDir());
assertEquals(baseCacheDir.resolve(RESOURCES_DIR), localStore.resourcesDir());
assertEquals(baseCacheDir.resolve(WORKING_DIR), localStore.workingDir());
}
@@ -226,15 +226,15 @@ public class LocalStoreTest {
}
@Test
- public void testStoreContextResources() throws Exception {
+ public void testStoreContext() throws Exception {
var localStore = new LocalStore(baseCacheDir, ALLOW_ALL_URLS);
- localStore.storeContextResources(def);
+ localStore.storeContext(manifest);
// Confirm the 3 jars are cached locally
assertTrue(Files.exists(baseCacheDir));
- assertTrue(Files
- .exists(baseCacheDir.resolve(CONTEXTS_DIR).resolve(checksumForFileName(def) + ".json")));
- for (Resource r : def.getResources()) {
+ assertTrue(Files.exists(
+ baseCacheDir.resolve(MANIFESTS_DIR).resolve(checksumForFileName(manifest) + ".json")));
+ for (Resource r : manifest.getResources()) {
assertTrue(Files.exists(baseCacheDir.resolve(RESOURCES_DIR).resolve(localResourceName(r))));
}
}
@@ -242,31 +242,32 @@ public class LocalStoreTest {
@Test
public void testClassLoader() throws Exception {
var localStore = new LocalStore(baseCacheDir, ALLOW_ALL_URLS);
- localStore.storeContextResources(def);
- var cacheKey = new ContextCacheKey("loc", def);
- var contextClassLoader = LccUtils.createClassLoader(cacheKey, localStore);
+ localStore.storeContext(manifest);
+ var cacheKey = new DeduplicationCacheKey("loc", manifest);
+ var cl = createClassLoader(cacheKey, localStore);
- testClassLoads(contextClassLoader, classA);
- testClassLoads(contextClassLoader, classB);
- testClassLoads(contextClassLoader, classC);
+ testClassLoads(cl, classA);
+ testClassLoads(cl, classB);
+ testClassLoads(cl, classC);
}
@Test
public void testClassLoaderUpdate() throws Exception {
var localStore = new LocalStore(baseCacheDir, ALLOW_ALL_URLS);
- localStore.storeContextResources(def);
- var cacheKey = new ContextCacheKey("loc", def);
- final var contextClassLoader = LccUtils.createClassLoader(cacheKey, localStore);
+ localStore.storeContext(manifest);
+ var cacheKey = new DeduplicationCacheKey("loc", manifest);
+ final var cl = createClassLoader(cacheKey, localStore);
- testClassLoads(contextClassLoader, classA);
- testClassLoads(contextClassLoader, classB);
- testClassLoads(contextClassLoader, classC);
+ testClassLoads(cl, classA);
+ testClassLoads(cl, classB);
+ testClassLoads(cl, classC);
// keep all but C
- var updatedResources = def.getResources().stream().limit(def.getResources().size() - 1)
- .collect(Collectors.toCollection(LinkedHashSet::new));
- assertEquals(def.getResources().size() - 1, updatedResources.size());
- var removedResource = def.getResources().stream().reduce((a, b) -> b).orElseThrow();
+ var updatedResources =
+ manifest.getResources().stream().limit(manifest.getResources().size() - 1)
+ .collect(Collectors.toCollection(LinkedHashSet::new));
+ assertEquals(manifest.getResources().size() - 1, updatedResources.size());
+ var removedResource = manifest.getResources().stream().reduce((a, b) -> b).orElseThrow();
// Add D
final URL jarDOrigLocation = LocalStoreTest.class.getResource("/ClassLoaderTestD/TestD.jar");
@@ -274,13 +275,13 @@ public class LocalStoreTest {
updatedResources.add(new Resource(jarDOrigLocation, "SHA-512",
TestUtils.computeResourceChecksum("SHA-512", jarDOrigLocation)));
- var updatedDef = new ContextDefinition(MONITOR_INTERVAL_SECS, updatedResources);
- localStore.storeContextResources(updatedDef);
+ var update = new Manifest(MONITOR_INTERVAL_SECS, updatedResources);
+ localStore.storeContext(update);
// Confirm the 3 jars are cached locally
assertTrue(Files.exists(
- baseCacheDir.resolve(CONTEXTS_DIR).resolve(checksumForFileName(updatedDef) + ".json")));
- for (Resource r : updatedDef.getResources()) {
+ baseCacheDir.resolve(MANIFESTS_DIR).resolve(checksumForFileName(update) + ".json")));
+ for (Resource r : update.getResources()) {
assertFalse(r.getFileName().contains("C"));
assertTrue(Files.exists(baseCacheDir.resolve(RESOURCES_DIR).resolve(localResourceName(r))));
}
@@ -290,37 +291,37 @@ public class LocalStoreTest {
assertTrue(Files
.exists(baseCacheDir.resolve(RESOURCES_DIR).resolve(localResourceName(removedResource))));
- cacheKey = new ContextCacheKey("loc", updatedDef);
- final var updatedContextClassLoader = LccUtils.createClassLoader(cacheKey, localStore);
+ cacheKey = new DeduplicationCacheKey("loc", update);
+ final var updatedCl = createClassLoader(cacheKey, localStore);
- assertNotEquals(contextClassLoader, updatedContextClassLoader);
- testClassLoads(updatedContextClassLoader, classA);
- testClassLoads(updatedContextClassLoader, classB);
- testClassFailsToLoad(updatedContextClassLoader, classC);
- testClassLoads(updatedContextClassLoader, classD);
+ assertNotEquals(cl, updatedCl);
+ testClassLoads(updatedCl, classA);
+ testClassLoads(updatedCl, classB);
+ testClassFailsToLoad(updatedCl, classC);
+ testClassLoads(updatedCl, classD);
}
@Test
public void testClassLoaderDirCleanup() throws Exception {
var localStore = new LocalStore(baseCacheDir, ALLOW_ALL_URLS);
assertEquals(0, Files.list(localStore.workingDir()).filter(Files::isDirectory).count());
- localStore.storeContextResources(def);
- var cacheKey = new ContextCacheKey("loc", def);
+ localStore.storeContext(manifest);
+ var cacheKey = new DeduplicationCacheKey("loc", manifest);
assertEquals(0, Files.list(localStore.workingDir()).filter(Files::isDirectory).count());
final var endBackgroundThread = new CountDownLatch(1);
var createdClassLoader = new CountDownLatch(1);
Thread t = new Thread(() -> {
- final var contextClassLoader = LccUtils.createClassLoader(cacheKey, localStore);
+ final var cl = createClassLoader(cacheKey, localStore);
createdClassLoader.countDown();
try {
- testClassLoads(contextClassLoader, classA);
+ testClassLoads(cl, classA);
endBackgroundThread.await();
} catch (Exception e) {
fail(e);
}
// hold a strong reference in the background thread long enough to prevent cleanup
- assertNotNull(contextClassLoader);
+ assertNotNull(cl);
});
t.start();
createdClassLoader.await();
diff --git a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/MiniAccumuloClusterClassLoaderFactoryTest.java b/modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/MiniAccumuloClusterClassLoaderFactoryTest.java
similarity index 80%
rename from modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/MiniAccumuloClusterClassLoaderFactoryTest.java
rename to modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/MiniAccumuloClusterClassLoaderFactoryTest.java
index 8939309..630b5fd 100644
--- a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/MiniAccumuloClusterClassLoaderFactoryTest.java
+++ b/modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/MiniAccumuloClusterClassLoaderFactoryTest.java
@@ -16,14 +16,14 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.accumulo.classloader.lcc;
+package org.apache.accumulo.classloader.ccl;
import static java.nio.charset.StandardCharsets.UTF_8;
import static java.nio.file.StandardCopyOption.REPLACE_EXISTING;
-import static org.apache.accumulo.classloader.lcc.LocalCachingContextClassLoaderFactory.ALLOWED_URLS_PATTERN;
-import static org.apache.accumulo.classloader.lcc.LocalCachingContextClassLoaderFactory.CACHE_DIR_PROPERTY;
-import static org.apache.accumulo.classloader.lcc.LocalCachingContextClassLoaderFactory.UPDATE_FAILURE_GRACE_PERIOD_MINS;
-import static org.apache.accumulo.classloader.lcc.util.LocalStore.WORKING_DIR;
+import static org.apache.accumulo.classloader.ccl.CachingClassLoaderFactory.PROP_ALLOWED_URLS;
+import static org.apache.accumulo.classloader.ccl.CachingClassLoaderFactory.PROP_CACHE_DIR;
+import static org.apache.accumulo.classloader.ccl.CachingClassLoaderFactory.PROP_GRACE_PERIOD;
+import static org.apache.accumulo.classloader.ccl.LocalStore.WORKING_DIR;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertIterableEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
@@ -44,9 +44,8 @@ import java.util.Set;
import java.util.TreeSet;
import java.util.stream.Collectors;
-import org.apache.accumulo.classloader.lcc.definition.ContextDefinition;
-import org.apache.accumulo.classloader.lcc.definition.Resource;
-import org.apache.accumulo.classloader.lcc.util.LocalStore;
+import org.apache.accumulo.classloader.ccl.manifest.Manifest;
+import org.apache.accumulo.classloader.ccl.manifest.Resource;
import org.apache.accumulo.core.client.Accumulo;
import org.apache.accumulo.core.client.AccumuloClient;
import org.apache.accumulo.core.client.AccumuloException;
@@ -87,10 +86,10 @@ public class MiniAccumuloClusterClassLoaderFactoryTest extends SharedMiniCluster
cfg.setNumTservers(3);
cfg.setProperty(Property.TSERV_NATIVEMAP_ENABLED.getKey(), "false");
cfg.setProperty(Property.GENERAL_CONTEXT_CLASSLOADER_FACTORY.getKey(),
- LocalCachingContextClassLoaderFactory.class.getName());
- cfg.setProperty(CACHE_DIR_PROPERTY, baseCacheDir.toUri().toString());
- cfg.setProperty(ALLOWED_URLS_PATTERN, ".*");
- cfg.setProperty(UPDATE_FAILURE_GRACE_PERIOD_MINS, "1");
+ CachingClassLoaderFactory.class.getName());
+ cfg.setProperty(PROP_CACHE_DIR, baseCacheDir.toUri().toString());
+ cfg.setProperty(PROP_ALLOWED_URLS, ".*");
+ cfg.setProperty(PROP_GRACE_PERIOD, "1");
}
}
@@ -100,7 +99,7 @@ public class MiniAccumuloClusterClassLoaderFactoryTest extends SharedMiniCluster
private static final String ITER_CLASS_NAME =
"org.apache.accumulo.classloader.vfs.examples.ExampleIterator";
private static final int MONITOR_INTERVAL_SECS =
- LocalCachingContextClassLoaderFactoryTest.MONITOR_INTERVAL_SECS;
+ CachingClassLoaderFactoryTest.MONITOR_INTERVAL_SECS;
private static URL jarAOrigLocation;
private static URL jarBOrigLocation;
@@ -132,15 +131,14 @@ public class MiniAccumuloClusterClassLoaderFactoryTest extends SharedMiniCluster
final var jsonDirPath = tempDir.resolve("simulatedRemoteContextFiles");
Files.createDirectory(jsonDirPath);
- // Create a context definition that only references jar A
- final var testContextDef =
- ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-256", jarAOrigLocation);
- final String testContextDefJson = testContextDef.toJson();
- final File testContextDefFile = jsonDirPath.resolve("testContextDefinition.json").toFile();
- Files.writeString(testContextDefFile.toPath(), testContextDefJson);
- assertTrue(Files.exists(testContextDefFile.toPath()));
+ // Create a manifest that only references jar A
+ final var manifest = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-256", jarAOrigLocation);
+ final String manifestJson = manifest.toJson();
+ final File testFile = jsonDirPath.resolve("testManifest.json").toFile();
+ Files.writeString(testFile.toPath(), manifestJson);
+ assertTrue(Files.exists(testFile.toPath()));
- Resource jarAResource = testContextDef.getResources().iterator().next();
+ Resource jarAResource = manifest.getResources().iterator().next();
String jarALocalFileName = LocalStore.localResourceName(jarAResource);
final String[] names = this.getUniqueNames(1);
@@ -189,8 +187,8 @@ public class MiniAccumuloClusterClassLoaderFactoryTest extends SharedMiniCluster
vp.cols = params.cols;
VerifyIngest.verifyIngest(client, vp);
- // Set the table classloader context. The context is the URL to the context definition file
- final String contextURL = testContextDefFile.toURI().toURL().toString();
+ // Set the table classloader context. The context is the URL to the manifest file
+ final String contextURL = testFile.toURI().toURL().toString();
client.tableOperations().setProperty(tableName, Property.TABLE_CLASSLOADER_CONTEXT.getKey(),
contextURL);
@@ -216,14 +214,13 @@ public class MiniAccumuloClusterClassLoaderFactoryTest extends SharedMiniCluster
assertEquals(1, refFiles.size(), refFiles::toString);
assertTrue(refFiles.stream().anyMatch(p -> p.endsWith(jarALocalFileName)));
- // Update the context definition to point to jar B
- final ContextDefinition testContextDefUpdate =
- ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarBOrigLocation);
- final String testContextDefUpdateJson = testContextDefUpdate.toJson();
- Files.writeString(testContextDefFile.toPath(), testContextDefUpdateJson);
- assertTrue(Files.exists(testContextDefFile.toPath()));
+ // Update to point to jar B
+ final Manifest update = Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarBOrigLocation);
+ final String updateJson = update.toJson();
+ Files.writeString(testFile.toPath(), updateJson);
+ assertTrue(Files.exists(testFile.toPath()));
- Resource jarBResource = testContextDefUpdate.getResources().iterator().next();
+ Resource jarBResource = update.getResources().iterator().next();
String jarBLocalFileName = LocalStore.localResourceName(jarBResource);
// Wait 2x the monitor interval
@@ -239,9 +236,8 @@ public class MiniAccumuloClusterClassLoaderFactoryTest extends SharedMiniCluster
assertTrue(refFiles.stream().anyMatch(p -> p.endsWith(jarALocalFileName)));
assertTrue(refFiles.stream().anyMatch(p -> p.endsWith(jarBLocalFileName)));
- // Copy jar A, create a context definition using the copy, then
- // remove the copy so that it's not found when the context classloader
- // updates.
+ // Copy jar A, create a manifest using the copy, then remove the copy so that it's not
+ // found when the context classloader updates.
var jarAPath = Path.of(jarAOrigLocation.toURI());
var jarAPathParent = jarAPath.getParent();
assertNotNull(jarAPathParent);
@@ -250,14 +246,14 @@ public class MiniAccumuloClusterClassLoaderFactoryTest extends SharedMiniCluster
Files.copy(jarAPath, jarACopy, REPLACE_EXISTING);
assertTrue(Files.exists(jarACopy));
- final ContextDefinition testContextDefUpdate2 =
- ContextDefinition.create(MONITOR_INTERVAL_SECS, "SHA-512", jarACopy.toUri().toURL());
+ final var update2 =
+ Manifest.create(MONITOR_INTERVAL_SECS, "SHA-512", jarACopy.toUri().toURL());
Files.delete(jarACopy);
assertTrue(!Files.exists(jarACopy));
- final String testContextDefUpdateJson2 = testContextDefUpdate2.toJson();
- Files.writeString(testContextDefFile.toPath(), testContextDefUpdateJson2);
- assertTrue(Files.exists(testContextDefFile.toPath()));
+ final String updateJson2 = update2.toJson();
+ Files.writeString(testFile.toPath(), updateJson2);
+ assertTrue(Files.exists(testFile.toPath()));
// Wait 2x the monitor interval
Thread.sleep(2 * MONITOR_INTERVAL_SECS * 1000);
diff --git a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/TestUtils.java b/modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/TestUtils.java
similarity index 83%
rename from modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/TestUtils.java
rename to modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/TestUtils.java
index b0e2d8c..4cc2691 100644
--- a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/TestUtils.java
+++ b/modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/TestUtils.java
@@ -16,10 +16,9 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.accumulo.classloader.lcc;
+package org.apache.accumulo.classloader.ccl;
import static java.nio.charset.StandardCharsets.UTF_8;
-import static org.apache.accumulo.classloader.lcc.util.LccUtils.getDigester;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
@@ -33,6 +32,7 @@ import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Path;
+import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.io.IOUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
@@ -66,38 +66,38 @@ public class TestUtils {
}
}
- public static org.apache.hadoop.fs.Path createContextDefinitionFile(FileSystem fs, String name,
+ public static org.apache.hadoop.fs.Path createManifestFile(FileSystem fs, String name,
String contents) throws Exception {
- var baseHdfsPath = new org.apache.hadoop.fs.Path("/contextDefs");
+ var baseHdfsPath = new org.apache.hadoop.fs.Path("/remote-manifests");
assertTrue(fs.mkdirs(baseHdfsPath));
- var newContextDefinitionFile = new org.apache.hadoop.fs.Path(baseHdfsPath, name);
+ var newFile = new org.apache.hadoop.fs.Path(baseHdfsPath, name);
if (contents == null) {
- assertTrue(fs.createNewFile(newContextDefinitionFile));
+ assertTrue(fs.createNewFile(newFile));
} else {
- try (FSDataOutputStream out = fs.create(newContextDefinitionFile)) {
+ try (FSDataOutputStream out = fs.create(newFile)) {
out.writeBytes(contents);
}
}
- assertTrue(fs.exists(newContextDefinitionFile));
- return newContextDefinitionFile;
+ assertTrue(fs.exists(newFile));
+ return newFile;
}
- public static void updateContextDefinitionFile(FileSystem fs,
- org.apache.hadoop.fs.Path defFilePath, String contents) throws Exception {
- // Update the contents of the context definition json file
- assertTrue(fs.exists(defFilePath));
- fs.delete(defFilePath, false);
- assertFalse(fs.exists(defFilePath));
+ public static void updateManifestFile(FileSystem fs, org.apache.hadoop.fs.Path path,
+ String contents) throws Exception {
+ // Update the contents
+ assertTrue(fs.exists(path));
+ fs.delete(path, false);
+ assertFalse(fs.exists(path));
if (contents == null) {
- assertTrue(fs.createNewFile(defFilePath));
+ assertTrue(fs.createNewFile(path));
} else {
- try (FSDataOutputStream out = fs.create(defFilePath)) {
+ try (FSDataOutputStream out = fs.create(path)) {
out.writeBytes(contents);
}
}
- assertTrue(fs.exists(defFilePath));
+ assertTrue(fs.exists(path));
}
@@ -172,7 +172,7 @@ public class TestUtils {
public static String computeResourceChecksum(String algorithm, URL resourceLocation)
throws IOException {
try (InputStream is = resourceLocation.openStream()) {
- return getDigester(algorithm).digestAsHex(is);
+ return new DigestUtils(algorithm).digestAsHex(is);
}
}
diff --git a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/URLTypesTest.java b/modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/URLTypesTest.java
similarity index 98%
rename from modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/URLTypesTest.java
rename to modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/URLTypesTest.java
index 54a122f..c959eb0 100644
--- a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/URLTypesTest.java
+++ b/modules/caching-class-loader/src/test/java/org/apache/accumulo/classloader/ccl/URLTypesTest.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.accumulo.classloader.lcc;
+package org.apache.accumulo.classloader.ccl;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
diff --git a/modules/local-caching-classloader/src/test/java/test/HelloWorldTemplate b/modules/caching-class-loader/src/test/java/test/HelloWorldTemplate
similarity index 100%
rename from modules/local-caching-classloader/src/test/java/test/HelloWorldTemplate
rename to modules/caching-class-loader/src/test/java/test/HelloWorldTemplate
diff --git a/modules/local-caching-classloader/src/test/java/test/Test.java b/modules/caching-class-loader/src/test/java/test/Test.java
similarity index 100%
rename from modules/local-caching-classloader/src/test/java/test/Test.java
rename to modules/caching-class-loader/src/test/java/test/Test.java
diff --git a/modules/local-caching-classloader/src/test/java/test/TestTemplate b/modules/caching-class-loader/src/test/java/test/TestTemplate
similarity index 100%
rename from modules/local-caching-classloader/src/test/java/test/TestTemplate
rename to modules/caching-class-loader/src/test/java/test/TestTemplate
diff --git a/modules/local-caching-classloader/src/test/resources/log4j2-test.properties b/modules/caching-class-loader/src/test/resources/log4j2-test.properties
similarity index 92%
rename from modules/local-caching-classloader/src/test/resources/log4j2-test.properties
rename to modules/caching-class-loader/src/test/resources/log4j2-test.properties
index 46e83a6..20ba2c2 100644
--- a/modules/local-caching-classloader/src/test/resources/log4j2-test.properties
+++ b/modules/caching-class-loader/src/test/resources/log4j2-test.properties
@@ -19,7 +19,7 @@
status = error
dest = err
-name = LocalCachineClassLoaderTestLoggingProperties
+name = CachingClassLoaderTestLoggingProperties
appender.console.type = Console
appender.console.name = STDOUT
@@ -30,7 +30,7 @@ appender.console.layout.pattern = %d{ISO8601} [%-8c{2}] %-5p: %m%n
logger.01.name = org.apache.accumulo
logger.01.level = debug
-logger.02.name = org.apache.accumulo.classloader.lcc
+logger.02.name = org.apache.accumulo.classloader.ccl
logger.02.level = trace
logger.03.name = org.apache.hadoop
diff --git a/modules/local-caching-classloader/src/test/shell/makeHelloWorldJars.sh b/modules/caching-class-loader/src/test/shell/makeHelloWorldJars.sh
similarity index 100%
rename from modules/local-caching-classloader/src/test/shell/makeHelloWorldJars.sh
rename to modules/caching-class-loader/src/test/shell/makeHelloWorldJars.sh
diff --git a/modules/local-caching-classloader/src/test/shell/makeTestJars.sh b/modules/caching-class-loader/src/test/shell/makeTestJars.sh
similarity index 100%
rename from modules/local-caching-classloader/src/test/shell/makeTestJars.sh
rename to modules/caching-class-loader/src/test/shell/makeTestJars.sh
diff --git a/modules/local-caching-classloader/README.md b/modules/local-caching-classloader/README.md
deleted file mode 100644
index 02651f8..0000000
--- a/modules/local-caching-classloader/README.md
+++ /dev/null
@@ -1,290 +0,0 @@
-<!--
-
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- https://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
--->
-# Local Caching ClassLoader
-
-`LocalCachingContextClassLoaderFactory` implements Accumulo's
-`ContextClassLoaderFactory` SPI. Given a context parameter, supplied as a
-String containing a URL to a remote context definition file, it will produce
-and return `ClassLoader` instances to load classes and resources for a Java
-application, based on the list of resource URLs contained in the remote context
-definition. It will also monitor the URL to the context definition file for any
-changes, at the monitoring interval specified in the context definition file.
-
-## Introduction
-
-This factory creates `ClassLoader` instances that point to locally cached
-copies of remote resource files. In this way, this factory allows placing
-common resources in a remote location for use across many hosts, but without
-many of the problems that can occur when loading resources from a remote
-location.
-
-This factory uses a storage cache in the local filesystem for any files it
-downloads from a remote URL.
-
-To use this factory, one must store resource files in a location that can be
-specified by a supported URL, and then must create a JSON-formatted context
-definition file that contains a monitoring interval (in seconds, greater than
-0), and a list of resource URLs along with a checksum for each resource file.
-This context definition file must then be stored somewhere where this factory
-can download it, and use the URL to that context definition file as the
-`context` parameter for this factory's `getClassLoader(String context)` method.
-
-This factory can handle context and resource URLs of any type that are
-supported by your application via a registered [URLStreamHandlerProvider][1],
-such as the built-in `file:` and `http:` types. A provider that handles `hdfs:`
-URL types must be provided by the user. This may be provided by the Apache
-Hadoop project, or by another library. A reference implementation is available
-[elsewhere in this project][2].
-
-Here is an example context definition file:
-
-```json
-{
- "monitorIntervalSeconds": 5,
- "resources": [
- {
- "location": "file:/home/user/ClassLoaderTestA/TestA.jar",
- "algorithm": "MD5",
- "checksum": "ae5e8248a9243751d60dbcaaedeb93ba"
- },
- {
- "location": "hdfs://localhost:8020/contextB/TestB.jar",
- "algorithm": "SHA-256",
- "checksum": "ed95fe130090fd64c2caddc3e37555ada8ba49a91bfd8ec1fd5c989d340ad0e0"
- },
- {
- "location": "http://localhost:80/TestC.jar",
- "algorithm": "SHA3-224",
- "checksum": "958f12ddc5acf87c2fe0ceed645327bb0c92e268acf915c4a374c14b"
- },
- {
- "location": "https://localhost:80/TestD.jar",
- "algorithm": "SHA-512/224",
- "checksum": "f7f982521ceb8ca97662973ada9b92b86de6bbaf233f14fd47efd792"
- }
- ]
-}
-```
-
-## How it Works
-
-When this factory receives a request for a `ClassLoader` for a given URL, it
-downloads a copy of the context definition file and parses it. If it has
-recently acquired that context definition file, based on the monitoring
-interval from a previous retrieval, it can skip this step and use the
-definition from the earlier retrieval, which is kept up-to-date by the
-background monitoring process that started when it was previously retrieved.
-Once it has the context definition, it then returns a `ClassLoader` instance
-containing the resources defined in that context definition file, first
-downloading any missing resources and verifying them using the checksums in the
-context definition file.
-
-`ClassLoader` instances are stored in a de-deduplicating cache in memory with a
-minimum lifetime of 24 hours. So, no two instances will ever exist in a process
-for the same context definition.
-
-If this context definition had not previously been downloaded, a background
-monitoring task is set up to ensure the URL is watched for any changes to the
-context definition. This monitoring continues for as long as there exists
-`ClassLoader` instances in the system that were constructed from the definition
-file at that URL (at least 24 hours, since that is the minimum time they will
-exist in the de-duplicating cache).
-
-## Local Storage Cache
-
-The local storage cache location is configured by the user by setting the
-required Accumulo property named `general.custom.classloader.lcc.cache.dir` to
-a directory on the local filesystem. This location may be specified as an
-absolute path or as a URL representing an absolute path with the `file` scheme.
-The location, and its directory structure will be created on first use, if it
-does not already exist. This will cause an error if the application does not
-have permission to create the directories.
-
-The selected location should be a persistent location with plenty of space to
-store downloaded resources (usually jar files), and should be writable by all
-the processes which use this factory to share the same resources. You may wish
-to pre-create the base directory specified by the property, and the three
-sub-directories, `contexts`, `resources`, and `working`, to set the appropriate
-permissions and ACLs.
-
-Resources downloaded to this cache may be used by multiple contexts, threads,
-and processes, so be very careful when removing old contents to ensure that
-they are no longer needed. If a resource file is deleted from the local storage
-cache while a `ClassLoader` exists that references it, that `ClassLoader` may,
-and probably will, stop working correctly. Similarly, files that have been
-downloaded should not be modified, because any modification will likely cause
-unexpected behavior to classloaders still using the file.
-
-* Do **NOT** use a temporary directory for the local storage cache location.
-* The local storage cache location **MUST** use a filesystem that supports
- atomic moves and hard links.
-
-## Security
-
-The Accumulo property `general.custom.classloader.lcc.allowed.urls.pattern` is
-another required parameter. It is used to limit the allowed URLs that can be
-fetched when downloading context definitions or context resources. Since the
-process using this factory will be using its own permissions to fetch
-resources, and placing a copy of those resources in a local directory where
-others may access them, that presents presents a potential file disclosure
-security risk. This property allows a system administrator to mitigate that
-risk by restricting access to only approved URLs. (e.g. to exclude non-approved
-locations like `file:/path/to/accumulo.properties` or
-`hdfs://host/path/to/accumulo/rfile.rf`).
-
-An example value for this property might look like:
-`https://example.com/path/to/contexts/.*` or
-`(file:/etc|hdfs://example[.]com:9000)/path/to/contexts/.*`
-
-Note: this property affects all URLs fetched by this factory, including context
-definition URLs and any resource URLs defined inside any fetched context
-definition. It should be updated by a system maintainer if any new context
-definitions have need to use new locations. It may be updated on a running
-system, and will take effect after approximately a minute.
-
-## Creating a ContextDefinition file
-
-Users may take advantage of the `ContextDefinition.create(int,String,URL[])`
-method to construct a `ContextDefinition` object, programmatically. This will
-calculate the checksums of the classpath elements. `ContextDefinition.toJson()`
-can be used to serialize the `ContextDefinition` to a `String` to store in a
-file.
-
-Alternatively, if this library's jar is built and placed onto Accumulo's
-`CLASSPATH`, then one can run `bin/accumulo create-context-definition` to
-create the ContextDefinition json file using the command-line. The resulting
-json is printed to stdout and can be redirected to a file. The command takes
-two arguments:
-
-1. the monitor interval, in seconds (e.g. `-i 300`),
-2. an optional checksum algorithm to use (e.g. `-a 'SHA3-512'`), and
-3. a list of file URLs (e.g. `hdfs://host:port/path/to/one.jar file://host/path/to/two.jar`)
-
-## Updating a ContextDefinition file
-
-This factory uses a background thread to fetch the context definition file at
-its initial URL using the interval specified in the definition file the last
-time it was retrieved. The definition file at a watched URL can be replaced at
-any time with any changes, including to changes to the monitor interval and
-resources. When the context definition is next retrieved, the new context
-definition file will be used as though it were an entirely new context at that
-URL. The next retrieval will occur after the monitor interval read from the
-most recent retrieval elapses. Changes to the context resources in any way will
-result in those new resources being downloaded, verified, and a new
-`ClassLoader` instance created and ready to be returned the next time
-`getClassLoader(String context)` is called with that context URL.
-
-Note: if the contents of a context definition file change in only
-inconsequential ways, such as JSON formatting changes, then those changes will
-not trigger any new downloads or `ClassLoader` staging. The is because the
-context definition JSON files are normalized prior to computing their checksum
-to determine if any changes have occurred.
-
-## Error Handling
-
-If there is an exception in creating the initial `ClassLoader`, then a
-`ContextClassLoaderException` is thrown. If there is an exception when updating
-the classloader, then the exception is logged and the classloader is not
-updated. Calls to `getClassLoader(String context)` will return the most recent
-classloader with valid contents. If the checksum of a downloaded resource does
-not match the checksum in the context definition file, then the downloaded
-version of the file is deleted from the context cache directory so that it can
-be retried at the next interval.
-
-The property `general.custom.classloader.lcc.update.grace.minutes` determines
-how long the update process continues to return the most recent valid
-classloader when an exception occurs in the background update thread. A zero
-value (default) will cause the most recent valid classloader to be returned.
-Otherwise, if a non-zero number is configured, then monitoring will stop after
-the update has failed for that number of minutes. Once monitoring has stopped,
-any subsequent calls to `getClassLoader(String context)` will behave as it
-would during an initial request, throwing a `ContextClassLoaderException` if
-the context definition cannot be retrieved or a `ClassLoader` cannot be
-constructed at that time.
-
-## Cleanup
-
-Because the cache directory is shared among multiple processes, and one process
-can't know what the other processes are doing, this class cannot always clean
-up the shared cache directory of unused resources automatically. It is left to
-the user to remove unused files from the cache. The local storage is organized
-into several directories, which are explained here to aid in understanding when
-unused files can be safely removed.
-
-### Contexts
-
-The `contexts` directory contents are always safe to delete. These contain only
-copies of the JSON files from which a ClassLoader was constructed. It is never
-used by this factory, and copies are placed here solely to provide more
-information to a user. Because these files are small text files and contain
-useful information, it is generally not recommended to delete these, because it
-may impair troubleshooting.
-
-### Resources
-
-The `resources` directory contains a shared pool of remote resource files that
-have been fetched for all contexts (typically, `.jar` files). The files in this
-directory are generally safe to delete any time. However, some considerations
-should be made:
-
-1. Deleting resources that are still needed will cause them to be downloaded
- again the next time they are needed, which may cause an increase in network
- activity.
-2. If any of the removed files had hard-linked "copies" in the `working`
- directory, the newly downloaded copy will increase the total amount of
- storage (whereas the original would have shared storage space with the
- hard-linked "copies").
-
-### Working
-
-The `working` directory contains temporary files for files currently being
-downloaded, and temporary directories containing hard-linked "copies" of files
-from the `resources` directory. These files and directories contain the process
-ID (PID) for the process that created them. Normally, these files are
-automatically cleaned up, but if a process is killed before that can happen,
-they may be left behind. The files with the PID in them can safely be removed,
-so long as the process that created them has been terminated.
-
-This directory also contains files that do not contain a PID. These files end
-with the `.downloading` suffix and exist to signal that a resource file is
-currently being downloaded by a process. These files are very small, containing
-only the PID of the most recent process to attempt downloading the file. They
-are removed when a download completes, or whenever the next time the
-corresponding resource file is used, if it has already been successfully
-downloaded by a previously failed process. Removing them won't break the
-application in any way, but doing so may result in a redundant download, which
-can result in increased network activity or storage space (see the previous
-section for considerations regarding the `resources` directory).
-
-## Accumulo Configuration
-
-To use this with Accumulo:
-
-1. Set the following Accumulo properties:
- * `general.context.class.loader.factory=org.apache.accumulo.classloader.lcc.LocalCachingContextClassLoaderFactory`
- * `general.custom.classloader.lcc.cache.dir=file://path/to/some/directory`
- * `general.custom.classloader.lcc.allowed.urls.pattern=someRegexPatternForAllowedUrls`
-2. Set the following table property to link to a context definition file. For example:
- * `table.class.loader.context=(file|hdfs|http|https)://path/to/context/definition.json`
-
-
-[1]: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/spi/URLStreamHandlerProvider.html
-[2]: https://github.com/apache/accumulo-classloaders/tree/main/modules/hdfs-urlstreamhandler-provider
diff --git a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactory.java b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactory.java
deleted file mode 100644
index 437352c..0000000
--- a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactory.java
+++ /dev/null
@@ -1,321 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * https://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.accumulo.classloader.lcc;
-
-import static java.util.Objects.requireNonNull;
-
-import java.io.IOException;
-import java.io.UncheckedIOException;
-import java.net.URISyntaxException;
-import java.net.URL;
-import java.net.URLClassLoader;
-import java.nio.file.Path;
-import java.time.Duration;
-import java.util.HashMap;
-import java.util.Map;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.Executors;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicReference;
-import java.util.function.BiConsumer;
-import java.util.function.Supplier;
-import java.util.regex.Pattern;
-
-import org.apache.accumulo.classloader.lcc.definition.ContextDefinition;
-import org.apache.accumulo.classloader.lcc.definition.Resource;
-import org.apache.accumulo.classloader.lcc.util.ContextCacheKey;
-import org.apache.accumulo.classloader.lcc.util.DeduplicationCache;
-import org.apache.accumulo.classloader.lcc.util.LccUtils;
-import org.apache.accumulo.classloader.lcc.util.LocalStore;
-import org.apache.accumulo.core.spi.common.ContextClassLoaderEnvironment;
-import org.apache.accumulo.core.spi.common.ContextClassLoaderFactory;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.base.Preconditions;
-import com.google.common.base.Stopwatch;
-import com.google.common.base.Suppliers;
-
-/**
- * A ContextClassLoaderFactory implementation that creates and maintains a ClassLoader for a context
- * definition at a remote URL, and referencing remotely stored resources. This factory expects the
- * parameter passed to {@link #getClassLoader(String)} to be the URL of a JSON-formatted
- * {@link ContextDefinition} file. The file contains an interval at which this class should monitor
- * the file for changes and a list of {@link Resource} objects. If the monitoring fails for a period
- * configurable with the {@link #UPDATE_FAILURE_GRACE_PERIOD_MINS} property, then monitoring will
- * discontinue until the next use of that context. Each resource is defined by a URL to the file, a
- * checksum algorithm, and a checksum.
- * <p>
- * The URLs supplied for the context definition file and for the resources may use any URL type with
- * a registered provider in your application, such as: file, http, https, or hdfs.
- * <p>
- * As this class processes the ContextDefinition, it fetches the contents of the resource from the
- * resource URL and caches it in a directory on the local filesystem. This class uses the value of
- * the property {@link #CACHE_DIR_PROPERTY} passed via {@link #init(ContextClassLoaderEnvironment)}
- * as the root directory and creates a sub-directory for context definition files, and another for
- * resource files. All cached files have a name that includes their checksum. The required property,
- * {@link #ALLOWED_URLS_PATTERN}, is used to specify a pattern for allowed URLs to be fetched.
- * <p>
- * An in-progress signal file is used for each resource file while it is being downloaded, to allow
- * multiple processes or threads to try to avoid redundant downloads. Atomic filesystem moves are
- * used to guarantee correctness if multiple downloads do occur.
- * <p>
- * Note that because the cache directory is shared among multiple processes, and one process can't
- * know what the other processes are doing, this class cannot clean up the shared cache directory.
- * It is left to the user to remove unused old files.
- */
-public class LocalCachingContextClassLoaderFactory implements ContextClassLoaderFactory {
-
- public static final String CACHE_DIR_PROPERTY = "general.custom.classloader.lcc.cache.dir";
-
- public static final String UPDATE_FAILURE_GRACE_PERIOD_MINS =
- "general.custom.classloader.lcc.update.grace.minutes";
-
- public static final String ALLOWED_URLS_PATTERN =
- "general.custom.classloader.lcc.allowed.urls.pattern";
-
- private static final Logger LOG =
- LoggerFactory.getLogger(LocalCachingContextClassLoaderFactory.class);
-
- private final ScheduledExecutorService executor = Executors.newScheduledThreadPool(0);
-
- // stores the latest seen ContextDefinition for a remote URL location; String types are used here
- // for the key instead of URL because URL.hashCode could trigger network activity for hostname
- // lookups
- private final ConcurrentHashMap<String,ContextDefinition> contextDefs = new ConcurrentHashMap<>();
-
- // to keep this coherent with the contextDefs, updates to this should be done in the compute
- // method of contextDefs
- private final DeduplicationCache<ContextCacheKey,LocalStore,URLClassLoader> classloaders =
- new DeduplicationCache<>(LccUtils::createClassLoader, Duration.ofHours(24), null);
-
- private final AtomicReference<LocalStore> localStore = new AtomicReference<>();
-
- private final Map<String,Stopwatch> classloaderFailures = new HashMap<>();
- private volatile Duration updateFailureGracePeriodMins;
-
- // this is a BiConsumer so we can pass a type in the String
- private volatile BiConsumer<String,URL> allowedUrlChecker;
-
- /**
- * Schedule a task to execute at {@code interval} seconds to update the LocalCachingContext if the
- * ContextDefinition has changed. The task schedules a follow-on task at the update interval value
- * (if it changed) and the task is successful or throws a handled exception. When an unhandled
- * exception is thrown, then the corresponding entry in the contextDefs map is cleared. The next
- * call to {@code #getClassLoader(String)} will recreate the contextDefs map entry and schedule
- * the monitor task.
- */
- private void monitorContext(final String contextLocation, long interval) {
- LOG.trace("Monitoring context definition file {} for changes at {} second intervals",
- contextLocation, interval);
- executor.schedule(() -> {
- try {
- checkMonitoredLocation(contextLocation, interval);
- } catch (Throwable t) {
- LOG.error("Unhandled exception occurred in context definition monitor thread. Removing"
- + " context definition {}.", contextLocation, t);
- contextDefs.remove(contextLocation);
- throw t;
- }
- }, interval, TimeUnit.SECONDS);
- }
-
- @Override
- public void init(ContextClassLoaderEnvironment env) {
- String value = requireNonNull(env.getConfiguration().get(CACHE_DIR_PROPERTY),
- "Property " + CACHE_DIR_PROPERTY + " not set, cannot create cache directory.");
- String graceProp = env.getConfiguration().get(UPDATE_FAILURE_GRACE_PERIOD_MINS);
- long graceMins = graceProp == null ? 0 : Long.parseLong(graceProp);
- updateFailureGracePeriodMins = Duration.ofMinutes(graceMins);
- // limit the frequency at which we check the config and re-compile the pattern
- Supplier<Pattern> allowedUrlsPattern = Suppliers.memoizeWithExpiration(
- () -> Pattern.compile(requireNonNull(env.getConfiguration().get(ALLOWED_URLS_PATTERN),
- "Property " + ALLOWED_URLS_PATTERN + " not set, no URLs are allowed")),
- Duration.ofMinutes(1));
- allowedUrlChecker = (locationType, url) -> {
- var p = allowedUrlsPattern.get();
- Preconditions.checkArgument(p.matcher(url.toExternalForm()).matches(),
- "%s location (%s) not allowed by pattern (%s)", locationType, url.toExternalForm(),
- p.pattern());
- };
- try {
- // check the allowed URLs pattern, getting it ready for first use, and warning if it is bad
- allowedUrlsPattern.get();
- } catch (RuntimeException npe) {
- LOG.warn(
- "Property {} is not set or contains an invalid pattern ()."
- + " No ClassLoader instances will be created until it is set.",
- ALLOWED_URLS_PATTERN, env.getConfiguration().get(ALLOWED_URLS_PATTERN), npe);
- }
- final Path baseCacheDir;
- if (value.startsWith("file:")) {
- try {
- baseCacheDir = Path.of(new URL(value).toURI());
- } catch (IOException | URISyntaxException e) {
- throw new IllegalArgumentException(
- "Malformed file: URL specified for base directory: " + value, e);
- }
- } else if (value.startsWith("/")) {
- baseCacheDir = Path.of(value);
- } else {
- throw new IllegalArgumentException(
- "Base directory is neither a file URL nor an absolute file path: " + value);
- }
- try {
- localStore.set(new LocalStore(baseCacheDir, allowedUrlChecker));
- } catch (IOException e) {
- throw new UncheckedIOException("Unable to create the local storage area at " + baseCacheDir,
- e);
- }
- }
-
- @Override
- public ClassLoader getClassLoader(final String contextLocation)
- throws ContextClassLoaderException {
- Preconditions.checkState(localStore.get() != null,
- "init not called before calling getClassLoader");
- requireNonNull(contextLocation, "context location must be supplied");
- final var classloader = new AtomicReference<URLClassLoader>();
- try {
- // get the current definition, or create it from the location if it doesn't exist; this has
- // the side effect of creating and caching a URLClassLoader instance if it doesn't exist for
- // the computed definition
- contextDefs.compute(contextLocation,
- (contextLocationKey, previousDefinition) -> computeDefinitionAndClassLoader(classloader,
- contextLocationKey, previousDefinition));
- } catch (RuntimeException e) {
- throw new ContextClassLoaderException(e.getMessage(), e);
- }
- return classloader.get();
- }
-
- private ContextDefinition computeDefinitionAndClassLoader(
- AtomicReference<URLClassLoader> resultHolder, String contextLocation,
- ContextDefinition previousDefinition) {
- ContextDefinition computedDefinition;
- if (previousDefinition == null) {
- try {
- computedDefinition = getDefinition(contextLocation);
- // we can set up monitoring now, but it will be blocked from doing anything yet, until this
- // finishes, since this code and the monitoring code both use contextDefs.compute(), which
- // is atomic/blocking for the same key
- monitorContext(contextLocation, computedDefinition.getMonitorIntervalSeconds());
- } catch (IOException e) {
- throw new UncheckedIOException(e);
- }
- } else {
- computedDefinition = previousDefinition;
- }
- resultHolder.set(classloaders.computeIfAbsent(
- new ContextCacheKey(contextLocation, computedDefinition), localStore::get));
- return computedDefinition;
- }
-
- private ContextDefinition getDefinition(String contextLocation) throws IOException {
- LOG.trace("Retrieving context definition file from {}", contextLocation);
- URL url = new URL(contextLocation);
- allowedUrlChecker.accept("Context definition", url);
- return ContextDefinition.fromRemoteURL(url);
- }
-
- private void checkMonitoredLocation(String contextLocation, long interval) {
- ContextDefinition currentDef =
- contextDefs.compute(contextLocation, (contextLocationKey, previousDefinition) -> {
- if (previousDefinition == null) {
- // context has been removed from the map, no need to check for update
- LOG.debug(
- "ContextDefinition for context {} not present, no longer monitoring for changes",
- contextLocation);
- return null;
- }
- // check for any classloaders still in the cache that were created for a context
- // definition found at this URL
- if (!classloaders.anyMatch(cacheKey -> cacheKey.getLocation().equals(contextLocation))) {
- LOG.debug("ClassLoader for context {} not present, no longer monitoring for changes",
- contextLocation);
- return null;
- }
- return previousDefinition;
- });
- if (currentDef == null) {
- return;
- }
- long nextInterval = interval;
- try {
- final ContextDefinition update = getDefinition(contextLocation);
- if (!currentDef.getChecksum().equals(update.getChecksum())) {
- LOG.debug("Context definition for {} has changed", contextLocation);
- localStore.get().storeContextResources(update);
- contextDefs.put(contextLocation, update);
- nextInterval = update.getMonitorIntervalSeconds();
- classloaderFailures.remove(contextLocation);
- } else {
- LOG.trace("Context definition for {} has not changed", contextLocation);
- }
- // reschedule this task to run if the context definition exists.
- // Atomically lock on the context key and only reschedule if the context is present.
- final long finalMonitorInterval = nextInterval;
- contextDefs.compute(contextLocation, (k, v) -> {
- if (v != null) {
- monitorContext(contextLocation, finalMonitorInterval);
- }
- return v;
- });
- } catch (IOException | RuntimeException e) {
- LOG.error("Error parsing updated context definition at {}. Classloader NOT updated!",
- contextLocation, e);
- final Stopwatch failureTimer = classloaderFailures.get(contextLocation);
- if (updateFailureGracePeriodMins.isZero()) {
- // failure monitoring is disabled
- LOG.debug("Property {} not set, not tracking classloader failures for context {}",
- UPDATE_FAILURE_GRACE_PERIOD_MINS, contextLocation);
- } else if (failureTimer == null) {
- // first failure, start the timer
- classloaderFailures.put(contextLocation, Stopwatch.createStarted());
- LOG.debug(
- "Tracking classloader failures for context {}, will NOT return working classloader if failures continue for {} minutes",
- contextLocation, updateFailureGracePeriodMins.toMinutes());
- } else if (failureTimer.elapsed().compareTo(updateFailureGracePeriodMins) > 0) {
- // has been failing for the grace period
- // unset the classloader reference so that the failure
- // will return from getClassLoader in the calling thread
- LOG.info("Grace period for failing classloader has elapsed for context {}",
- contextLocation);
- contextDefs.remove(contextLocation);
- classloaderFailures.remove(contextLocation);
- } else {
- LOG.trace("Failing to update classloader for context {} within the grace period",
- contextLocation, e);
- }
- // reschedule this task to run if the context definition exists.
- // Don't put this in finally block as we only want to reschedule
- // on success or handled exception
- // Atomically lock on the context key and only reschedule if the context is present.
- final long finalMonitorInterval = nextInterval;
- contextDefs.compute(contextLocation, (k, v) -> {
- if (v != null) {
- monitorContext(contextLocation, finalMonitorInterval);
- }
- return v;
- });
- }
- }
-
-}
diff --git a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/LccUtils.java b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/LccUtils.java
deleted file mode 100644
index 85c0791..0000000
--- a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/LccUtils.java
+++ /dev/null
@@ -1,133 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * https://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.accumulo.classloader.lcc.util;
-
-import java.io.File;
-import java.io.IOException;
-import java.io.UncheckedIOException;
-import java.lang.ref.Cleaner;
-import java.net.MalformedURLException;
-import java.net.URL;
-import java.net.URLClassLoader;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.util.Comparator;
-import java.util.LinkedHashSet;
-
-import org.apache.accumulo.classloader.lcc.LocalCachingContextClassLoaderFactory;
-import org.apache.accumulo.classloader.lcc.definition.ContextDefinition;
-import org.apache.accumulo.classloader.lcc.definition.Resource;
-import org.apache.accumulo.classloader.lcc.util.LocalStore.HardLinkFailedException;
-import org.apache.commons.codec.digest.DigestUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
-
-public class LccUtils {
- private static final Logger LOG = LoggerFactory.getLogger(LccUtils.class);
-
- private static final Cleaner CLEANER = Cleaner.create();
-
- // these instances are not thread-safe; calls to getClassLoader(context) and the background
- // monitor threads need distinct instances of these
- public static DigestUtils getDigester(String algorithm) {
- return new DigestUtils(algorithm);
- }
-
- private static String checksumForFileName(String algorithm, String checksum) {
- return algorithm.replace('/', '_') + "-" + checksum;
- }
-
- public static String checksumForFileName(ContextDefinition definition) {
- return checksumForFileName(definition.getChecksumAlgorithm(), definition.getChecksum());
- }
-
- public static String checksumForFileName(Resource definition) {
- return checksumForFileName(definition.getAlgorithm(), definition.getChecksum());
- }
-
- @SuppressFBWarnings(value = "DP_CREATE_CLASSLOADER_INSIDE_DO_PRIVILEGED",
- justification = "doPrivileged is deprecated without replacement and removed in newer Java")
- public static URLClassLoader createClassLoader(ContextCacheKey cacheKey, LocalStore localStore) {
- // use a LinkedHashSet to preserve the order of the context resources
- final var hardLinks = new LinkedHashSet<Path>();
- Path hardLinksDir = null;
-
- var def = cacheKey.getContextDefinition();
-
- // stage the downloads before attempting hard link creation
- localStore.storeContextResources(def);
-
- // keep trying to hard-link all the resources if the hard-linking fails
- while (hardLinksDir == null) {
- hardLinks.clear();
- try {
- hardLinksDir = localStore.createWorkingHardLinks(def, hardLinks::add);
- LOG.trace("Created hard links at {} for context {}", hardLinksDir, cacheKey);
- } catch (HardLinkFailedException e) {
- var failedHardLinksDir = e.getDestinationDirectory();
- LOG.warn(
- "Exception creating a hard link in {} due to missing resource {}; attempting re-download of context resources",
- failedHardLinksDir, e.getMissingResource(), e);
- try {
- LccUtils.recursiveDelete(failedHardLinksDir);
- } catch (IOException ioe) {
- LOG.warn(
- "Saw exception removing directory {} after hard link creation failure; this should be cleaned up manually",
- failedHardLinksDir, ioe);
- }
- localStore.storeContextResources(def);
- }
- }
-
- URL[] urls = hardLinks.stream().map(p -> {
- try {
- LOG.trace("Added resource {} to classpath", p);
- return p.toUri().toURL();
- } catch (MalformedURLException e) {
- // shouldn't be possible, since these are file-based URLs
- throw new UncheckedIOException(e);
- }
- }).toArray(URL[]::new);
-
- final var cl = new URLClassLoader(cacheKey.toString(), urls,
- LocalCachingContextClassLoaderFactory.class.getClassLoader());
- LOG.info("New classloader created for {}", cacheKey);
-
- final var cleanDir = hardLinksDir;
- CLEANER.register(cl, () -> {
- try {
- LccUtils.recursiveDelete(cleanDir);
- } catch (IOException e) {
- LOG.warn("Saw exception when executing cleaner on directory {}", cleanDir, e);
- }
- });
- return cl;
- }
-
- public static void recursiveDelete(Path directory) throws IOException {
- if (Files.exists(directory)) {
- try (var walker = Files.walk(directory)) {
- walker.map(Path::toFile).sorted(Comparator.reverseOrder()).forEach(File::delete);
- }
- }
- }
-
-}
diff --git a/pom.xml b/pom.xml
index bcab9b3..6d4dd53 100644
--- a/pom.xml
+++ b/pom.xml
@@ -75,9 +75,9 @@
<modules>
<module>modules/example-iterators-a</module>
<module>modules/example-iterators-b</module>
- <module>modules/vfs-class-loader</module>
- <module>modules/local-caching-classloader</module>
+ <module>modules/caching-class-loader</module>
<module>modules/hdfs-urlstreamhandler-provider</module>
+ <module>modules/vfs-class-loader</module>
</modules>
<scm>
<connection>scm:git:https://gitbox.apache.org/repos/asf/accumulo-classloaders.git</connection>