You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by ct...@apache.org on 2020/09/26 03:22:25 UTC
[accumulo-classloaders] branch main updated (19527a9 -> 99f9187)
This is an automated email from the ASF dual-hosted git repository.
ctubbsii pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-classloaders.git.
from 19527a9 Drop trailing slash on homepage URL in GitHub
new 491fba1 Initial commit, tests failing
new 5566e52 Capturing testing document
new 99f9187 Make changes to get the initial code buildable
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../classloaderfactory-substitute/.gitignore | 4 +
modules/classloaderfactory-substitute/pom.xml | 39 ++
.../core/spi/common/ClassLoaderFactory.java | 61 ++
.gitignore => modules/vfs-class-loader/.gitignore | 4 +
LICENSE => modules/vfs-class-loader/LICENSE | 2 -
modules/vfs-class-loader/README.md | 36 +
modules/vfs-class-loader/WIP.txt | 32 +
modules/vfs-class-loader/license-header.txt | 16 +
modules/vfs-class-loader/pom.xml | 214 ++++++
.../accumulo/classloader/vfs/ClassPathPrinter.java | 132 ++++
.../classloader/vfs/ReloadingVFSClassLoader.java | 724 +++++++++++++++++++++
.../classloader/vfs/UniqueFileReplicator.java | 105 +++
.../classloader/vfs/VFSClassLoaderWrapper.java | 83 +++
.../accumulo/classloader/vfs/VFSManager.java | 219 +++++++
.../ReloadingVFSContextClassLoaderFactory.java | 271 ++++++++
.../accumulo/classloader/vfs/AccumuloDFSBase.java | 134 ++++
.../classloader/vfs/ClassPathPrinterTest.java | 85 +++
.../vfs/ReloadingVFSClassLoaderTest.java | 225 +++++++
.../classloader/vfs/VfsClassLoaderTest.java | 150 +++++
.../ReloadingVFSContextClassLoaderFactoryTest.java | 113 ++++
.../src/test/java/test/HelloWorldTemplate | 27 +
.../vfs-class-loader/src/test/java/test/Test.java | 27 +
.../src/test/java/test/TestTemplate | 36 +
.../src/test/resources/log4j2-test.properties | 25 +-
.../src/test/shell/makeHelloWorldJars.sh | 33 +
.../src/test/shell/makeTestJars.sh | 24 +-
pom.xml | 8 +-
27 files changed, 2801 insertions(+), 28 deletions(-)
copy .gitignore => modules/classloaderfactory-substitute/.gitignore (96%)
create mode 100644 modules/classloaderfactory-substitute/pom.xml
create mode 100644 modules/classloaderfactory-substitute/src/main/java/org/apache/accumulo/core/spi/common/ClassLoaderFactory.java
copy .gitignore => modules/vfs-class-loader/.gitignore (96%)
copy LICENSE => modules/vfs-class-loader/LICENSE (99%)
create mode 100644 modules/vfs-class-loader/README.md
create mode 100644 modules/vfs-class-loader/WIP.txt
create mode 100644 modules/vfs-class-loader/license-header.txt
create mode 100644 modules/vfs-class-loader/pom.xml
create mode 100644 modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/ClassPathPrinter.java
create mode 100644 modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/ReloadingVFSClassLoader.java
create mode 100644 modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/UniqueFileReplicator.java
create mode 100644 modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/VFSClassLoaderWrapper.java
create mode 100644 modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/VFSManager.java
create mode 100644 modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/context/ReloadingVFSContextClassLoaderFactory.java
create mode 100644 modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/AccumuloDFSBase.java
create mode 100644 modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/ClassPathPrinterTest.java
create mode 100644 modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/ReloadingVFSClassLoaderTest.java
create mode 100644 modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/VfsClassLoaderTest.java
create mode 100644 modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/context/ReloadingVFSContextClassLoaderFactoryTest.java
create mode 100644 modules/vfs-class-loader/src/test/java/test/HelloWorldTemplate
create mode 100644 modules/vfs-class-loader/src/test/java/test/Test.java
create mode 100644 modules/vfs-class-loader/src/test/java/test/TestTemplate
copy .asf.yaml => modules/vfs-class-loader/src/test/resources/log4j2-test.properties (63%)
create mode 100755 modules/vfs-class-loader/src/test/shell/makeHelloWorldJars.sh
copy .asf.yaml => modules/vfs-class-loader/src/test/shell/makeTestJars.sh (57%)
mode change 100644 => 100755
[accumulo-classloaders] 01/03: Initial commit, tests failing
Posted by ct...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ctubbsii pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-classloaders.git
commit 491fba1f7dc2616de99f5dbde46bb7e572cd9bed
Author: Dave Marion <dl...@apache.org>
AuthorDate: Fri Sep 25 21:07:50 2020 +0000
Initial commit, tests failing
---
modules/vfs-class-loader/.gitignore | 32 +
modules/vfs-class-loader/LICENSE | 201 ++++++
modules/vfs-class-loader/README.md | 36 +
modules/vfs-class-loader/license-header.txt | 16 +
modules/vfs-class-loader/pom.xml | 198 ++++++
.../accumulo/classloader/vfs/ClassPathPrinter.java | 132 ++++
.../classloader/vfs/ReloadingVFSClassLoader.java | 724 +++++++++++++++++++++
.../classloader/vfs/UniqueFileReplicator.java | 105 +++
.../classloader/vfs/VFSClassLoaderWrapper.java | 83 +++
.../accumulo/classloader/vfs/VFSManager.java | 219 +++++++
.../ReloadingVFSContextClassLoaderFactory.java | 271 ++++++++
.../accumulo/classloader/vfs/AccumuloDFSBase.java | 134 ++++
.../classloader/vfs/ClassPathPrinterTest.java | 85 +++
.../vfs/ReloadingVFSClassLoaderTest.java | 225 +++++++
.../classloader/vfs/VfsClassLoaderTest.java | 150 +++++
.../ReloadingVFSContextClassLoaderFactoryTest.java | 113 ++++
.../src/test/java/test/HelloWorldTemplate | 27 +
.../vfs-class-loader/src/test/java/test/Test.java | 27 +
.../src/test/java/test/TestTemplate | 36 +
.../src/test/resources/log4j2-test.properties | 35 +
.../src/test/shell/makeHelloWorldJars.sh | 33 +
.../src/test/shell/makeTestJars.sh | 32 +
pom.xml | 7 +-
23 files changed, 2918 insertions(+), 3 deletions(-)
diff --git a/modules/vfs-class-loader/.gitignore b/modules/vfs-class-loader/.gitignore
new file mode 100644
index 0000000..3d5bdae
--- /dev/null
+++ b/modules/vfs-class-loader/.gitignore
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Maven ignores
+/target/
+
+# IDE ignores
+/.settings/
+/.project
+/.classpath
+/.pydevproject
+/.idea
+/*.iml
+/*.ipr
+/*.iws
+/nbproject/
+/nbactions.xml
+/nb-configuration.xml
+.vscode/
+.factorypath
diff --git a/modules/vfs-class-loader/LICENSE b/modules/vfs-class-loader/LICENSE
new file mode 100644
index 0000000..261eeb9
--- /dev/null
+++ b/modules/vfs-class-loader/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/modules/vfs-class-loader/README.md b/modules/vfs-class-loader/README.md
new file mode 100644
index 0000000..8b53cdb
--- /dev/null
+++ b/modules/vfs-class-loader/README.md
@@ -0,0 +1,36 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# VFS Reloading ClassLoader
+
+This module contains a [ClassLoader](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/ClassLoader.html) implementation that can be used as the [System](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/ClassLoader.html#getSystemClassLoader()) ClassLoader.
+
+## Configuration
+
+To use this ClassLoader as the System ClassLoader you must set the JVM system property **java.system.class.loader** to the fully qualified class name (org.apache.accumulo.classloader.vfs.ReloadingVFSClassLoader). This jar and it's dependent jars must be on the **java.class.path**.
+
+To set the classpath for this ClassLoader you must define the system property **vfs.class.loader.classpath** and set it to locations that are supported by [Apache Commons VFS](http://commons.apache.org/proper/commons-vfs/filesystems.html).
+
+The ClassLoader monitors the classpath for changes at 5 minute intervals. To change this interval define the sytem property **vfs.classpath.monitor.seconds**.
+
+This ClassLoader follows the normal parent delegation model but can be set to load classes and resources first, before checking if the parent classloader can, by setting the system property **vfs.class.loader.delegation** to "post".
+
+Finally, this ClassLoader keeps a local cache of objects pulled from remote systems (via http, etc.). The default location for this cache directory is the value of the system property **java.io.tmpdir**. To change this location set the system property **vfs.cache.dir** to an existing directory.
+
+## Implementation
+
+This ClassLoader maintains a [VFSClassLoader](http://commons.apache.org/proper/commons-vfs/commons-vfs2/apidocs/org/apache/commons/vfs2/impl/VFSClassLoader.html) delegate that references the classpath (as specified by **vfs.class.loader.classpath**). The ReloadingVFSClassLoader implements [FileListener](http://commons.apache.org/proper/commons-vfs/commons-vfs2/apidocs/org/apache/commons/vfs2/FileListener.html) and creates a [DefaultFileMonitor](http://commons.apache.org/proper/commons-vf [...]
diff --git a/modules/vfs-class-loader/license-header.txt b/modules/vfs-class-loader/license-header.txt
new file mode 100644
index 0000000..60b675e
--- /dev/null
+++ b/modules/vfs-class-loader/license-header.txt
@@ -0,0 +1,16 @@
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
diff --git a/modules/vfs-class-loader/pom.xml b/modules/vfs-class-loader/pom.xml
new file mode 100644
index 0000000..e727d2f
--- /dev/null
+++ b/modules/vfs-class-loader/pom.xml
@@ -0,0 +1,198 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.accumulo</groupId>
+ <artifactId>classloader-extras</artifactId>
+ <version>1.0.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+ <artifactId>vfs-reloading-classloader</artifactId>
+ <name>VFS Reloading ClassLoader</name>
+ <properties>
+ <eclipseFormatterStyle>../../contrib/Eclipse-Accumulo-Codestyle.xml</eclipseFormatterStyle>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-vfs2</artifactId>
+ <version>2.6.0</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-1.2-api</artifactId>
+ <version>2.13.1</version>
+ </dependency>
+ <dependency>
+ <groupId>com.google.code.gson</groupId>
+ <artifactId>gson</artifactId>
+ <version>2.8.6</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ <version>2.7</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ <version>1.2</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.accumulo</groupId>
+ <artifactId>accumulo-core</artifactId>
+ <version>2.1.0-SNAPSHOT</version>
+ <scope>provided</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.accumulo</groupId>
+ <artifactId>accumulo-start</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-api</artifactId>
+ <version>3.2.1</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ <version>1.7.30</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.13</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.accumulo</groupId>
+ <artifactId>accumulo-start</artifactId>
+ <version>2.1.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-minicluster</artifactId>
+ <version>3.2.1</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-slf4j-impl</artifactId>
+ <version>2.13.1</version>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+ <build>
+ <pluginManagement>
+ <plugins>
+ <!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.-->
+ <plugin>
+ <groupId>org.eclipse.m2e</groupId>
+ <artifactId>lifecycle-mapping</artifactId>
+ <version>1.0.0</version>
+ <configuration>
+ <lifecycleMappingMetadata>
+ <pluginExecutions>
+ <pluginExecution>
+ <pluginExecutionFilter>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>exec-maven-plugin</artifactId>
+ <versionRange>[3.0.0,)</versionRange>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ </pluginExecutionFilter>
+ <action>
+ <ignore />
+ </action>
+ </pluginExecution>
+ </pluginExecutions>
+ </lifecycleMappingMetadata>
+ </configuration>
+ </plugin>
+ </plugins>
+ </pluginManagement>
+ <plugins>
+ <plugin>
+ <groupId>com.mycila</groupId>
+ <artifactId>license-maven-plugin</artifactId>
+ <version>3.0</version>
+ <configuration>
+ <header>${project.basedir}/license-header.txt</header>
+ <excludes combine.children="append">
+ <exclude>**/DEPENDENCIES</exclude>
+ <exclude>**/LICENSE</exclude>
+ <exclude>**/NOTICE</exclude>
+ <exclude>**/target/**</exclude>
+ <exclude>contrib/javadoc11.patch</exclude>
+ </excludes>
+ <mapping combine.children="append">
+ <!-- general mappings; module-specific mappings appear in their respective pom -->
+ <Makefile>SCRIPT_STYLE</Makefile>
+ <c>SLASHSTAR_STYLE</c>
+ <cc>SLASHSTAR_STYLE</cc>
+ <css>SLASHSTAR_STYLE</css>
+ <h>SLASHSTAR_STYLE</h>
+ <java>SLASHSTAR_STYLE</java>
+ <proto>SLASHSTAR_STYLE</proto>
+ <thrift>SLASHSTAR_STYLE</thrift>
+ </mapping>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>exec-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>Build Test jars</id>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <phase>process-test-classes</phase>
+ <configuration>
+ <executable>${project.basedir}/src/test/shell/makeTestJars.sh</executable>
+ </configuration>
+ </execution>
+ <execution>
+ <id>Build HelloWorld jars</id>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <phase>process-test-classes</phase>
+ <configuration>
+ <executable>${project.basedir}/src/test/shell/makeHelloWorldJars.sh</executable>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
diff --git a/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/ClassPathPrinter.java b/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/ClassPathPrinter.java
new file mode 100644
index 0000000..7112d25
--- /dev/null
+++ b/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/ClassPathPrinter.java
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.vfs;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Collections;
+
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.impl.VFSClassLoader;
+
+public class ClassPathPrinter {
+
+ public interface Printer {
+ void print(String s);
+ }
+
+ public static void printClassPath(ClassLoader cl, boolean debug) {
+ printClassPath(cl, System.out::print, debug);
+ }
+
+ public static String getClassPath(ClassLoader cl, boolean debug) {
+ StringBuilder cp = new StringBuilder();
+ printClassPath(cl, cp::append, debug);
+ return cp.toString();
+ }
+
+ private static void printJar(Printer out, String jarPath, boolean debug, boolean sawFirst) {
+ if (debug) {
+ out.print("\t");
+ }
+ if (!debug && sawFirst) {
+ out.print(":");
+ }
+ out.print(jarPath);
+ if (debug) {
+ out.print("\n");
+ }
+ }
+
+ public static void printClassPath(ClassLoader cl, Printer out, boolean debug) {
+ try {
+ ArrayList<ClassLoader> classloaders = new ArrayList<>();
+
+ while (cl != null) {
+ classloaders.add(cl);
+ cl = cl.getParent();
+ }
+
+ Collections.reverse(classloaders);
+
+ int level = 0;
+
+ for (ClassLoader classLoader : classloaders) {
+
+ level++;
+
+ if (debug && level > 1) {
+ out.print("\n");
+ }
+ if (!debug && level < 2) {
+ continue;
+ }
+
+ boolean sawFirst = false;
+ if (classLoader.getClass().getName().startsWith("jdk.internal")) {
+ if (debug) {
+ out.print("Level " + level + ": " + classLoader.getClass().getName()
+ + " configuration not inspectable.\n");
+ }
+ } else if (classLoader instanceof URLClassLoader) {
+ if (debug) {
+ out.print("Level " + level + ": URL classpath, items are:\n");
+ }
+ for (URL u : ((URLClassLoader) classLoader).getURLs()) {
+ printJar(out, u.getFile(), debug, sawFirst);
+ sawFirst = true;
+ }
+ } else if (classLoader instanceof ReloadingVFSClassLoader) {
+ if (debug) {
+ out.print("Level " + level + ": ReloadingVFSClassLoader, classpath items are:\n");
+ }
+ @SuppressWarnings("resource")
+ ReloadingVFSClassLoader vcl = (ReloadingVFSClassLoader) classLoader;
+ ClassLoader delegate = vcl.getDelegateClassLoader();
+ if (delegate instanceof VFSClassLoaderWrapper) {
+ VFSClassLoaderWrapper wrapper = (VFSClassLoaderWrapper) delegate;
+ for (FileObject f : wrapper.getFileObjects()) {
+ printJar(out, f.getURL().getFile(), debug, sawFirst);
+ sawFirst = true;
+ }
+ }
+ } else if (classLoader instanceof VFSClassLoader) {
+ if (debug) {
+ out.print("Level " + level + ": VFSClassLoader, classpath items are:\n");
+ }
+ VFSClassLoader vcl = (VFSClassLoader) classLoader;
+ for (FileObject f : vcl.getFileObjects()) {
+ printJar(out, f.getURL().getFile(), debug, sawFirst);
+ sawFirst = true;
+ }
+ } else {
+ if (debug) {
+ out.print("Level " + level + ": Unknown classloader configuration "
+ + classLoader.getClass() + "\n");
+ }
+ }
+ }
+ out.print("\n");
+ } catch (Throwable t) {
+ throw new RuntimeException(t);
+ }
+ }
+
+}
diff --git a/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/ReloadingVFSClassLoader.java b/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/ReloadingVFSClassLoader.java
new file mode 100644
index 0000000..552076a
--- /dev/null
+++ b/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/ReloadingVFSClassLoader.java
@@ -0,0 +1,724 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.vfs;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URL;
+import java.util.Arrays;
+import java.util.Enumeration;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Consumer;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Stream;
+
+import org.apache.commons.vfs2.FileChangeEvent;
+import org.apache.commons.vfs2.FileListener;
+import org.apache.commons.vfs2.FileMonitor;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.impl.DefaultFileMonitor;
+import org.apache.commons.vfs2.impl.DefaultFileSystemManager;
+import org.apache.commons.vfs2.provider.hdfs.HdfsFileObject;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+
+/**
+ * <p>
+ * A {@code ClassLoader} implementation that watches for changes in any of the files/directories in
+ * the classpath. When a change is noticed, this classloader will then load the new classes in
+ * subsequent calls to loadClass. This classloader supports both the normal classloader
+ * pre-delegation model and a post-delegation model. To enable the post-delegation feature set the
+ * system property <b>vfs.class.loader.delegation</b> to "post".
+ *
+ * <p>
+ * This classloader uses the following system properties:
+ *
+ * <ol>
+ * <li><b>vfs.cache.dir</b> - for specifying the directory to use for the local VFS cache (default
+ * is the system property <b>java.io.tmpdir</b></li>
+ * <li><b>vfs.classpath.monitor.seconds</b> - for specifying the file system monitor (default:
+ * 5m)</li>
+ * <li><b>vfs.class.loader.classpath</b> - for specifying the class path</li>
+ * <li><b>vfs.class.loader.delegation</b> - valid values are "pre" and "post" (default: pre)</li>
+ * </ol>
+ *
+ * <p>
+ * This class will attempt to perform substitution on any environment variables found in the values.
+ * For example, the system property <b>vfs.cache.dir</b> can be set to <b>$HOME/cache</b>.
+ */
+public class ReloadingVFSClassLoader extends ClassLoader implements Closeable, FileListener {
+
+ public static final String VFS_CLASSPATH_MONITOR_INTERVAL = "vfs.classpath.monitor.seconds";
+ public static final String VFS_CACHE_DIR_PROPERTY = "vfs.cache.dir";
+ public static final String VFS_CLASSLOADER_CLASSPATH = "vfs.class.loader.classpath";
+ public static final String VFS_CLASSLOADER_DELEGATION = "vfs.class.loader.delegation";
+ public static final String VFS_CLASSLOADER_DEBUG = "vfs.class.loader.debug";
+
+ private static final String VFS_CACHE_DIR_DEFAULT = "java.io.tmpdir";
+
+ // set to 5 mins. The rationale behind this large time is to avoid a gazillion tservers all asking
+ // the name node for info too frequently.
+ private static final long DEFAULT_TIMEOUT = TimeUnit.MINUTES.toMillis(5);
+
+ private static boolean DEBUG = false;
+ private static String CLASSPATH = null;
+ private static Boolean PRE_DELEGATION = null;
+ private static Long MONITOR_INTERVAL = null;
+ private static boolean VM_INITIALIZED = false;
+
+ private volatile long maxWaitInterval = 60000;
+ private volatile long maxRetries = -1;
+ private volatile long sleepInterval = 1000;
+ private volatile boolean vfsInitializing = false;
+
+ private final ThreadPoolExecutor executor;
+ private final ClassLoader parent;
+ private final ReentrantReadWriteLock updateLock = new ReentrantReadWriteLock(true);
+ private final String name;
+ private final String classpath;
+ private final Boolean preDelegation;
+ private final long monitorInterval;
+ private DefaultFileMonitor monitor;
+ private FileObject[] files;
+ private VFSClassLoaderWrapper cl = null;
+ private DefaultFileSystemManager vfs = null;
+
+ static {
+ DEBUG = Boolean.parseBoolean(System.getProperty(VFS_CLASSLOADER_DEBUG, "false"));
+ CLASSPATH = getClassPathProperty();
+ PRE_DELEGATION = getPreDelegationModelProperty();
+ MONITOR_INTERVAL = getMonitorIntervalProperty();
+ }
+
+ private static void printDebug(String msg) {
+ if (!DEBUG)
+ return;
+ System.out
+ .println(String.format("%d ReloadingVFSClassLoader: %s", System.currentTimeMillis(), msg));
+ }
+
+ private static void printError(String msg) {
+ System.err
+ .println(String.format("%d ReloadingVFSClassLoader: %s", System.currentTimeMillis(), msg));
+ }
+
+ /**
+ * Get the classpath value from the environment and resolve embedded env vars
+ *
+ * @return classpath value
+ */
+ private static String getClassPathProperty() {
+ String cp = System.getProperty(VFS_CLASSLOADER_CLASSPATH);
+ if (null == cp || cp.isBlank()) {
+ printError(VFS_CLASSLOADER_CLASSPATH + " system property not set, using default of \"\"");
+ cp = "";
+ }
+ String result = replaceEnvVars(cp, System.getenv());
+ printDebug("Classpath set to: " + result);
+ return result;
+ }
+
+ /**
+ * Get the delegation model
+ *
+ * @return true if pre delegaion, false if post delegation
+ */
+ private static boolean getPreDelegationModelProperty() {
+ String delegation = System.getProperty(VFS_CLASSLOADER_DELEGATION);
+ boolean preDelegation = true;
+ if (null != delegation && delegation.equalsIgnoreCase("post")) {
+ preDelegation = false;
+ }
+ printDebug("ClassLoader configured for pre-delegation: " + preDelegation);
+ return preDelegation;
+ }
+
+ /**
+ * Get the directory for the VFS cache
+ *
+ * @return VFS cache directory
+ */
+ static String getVFSCacheDir() {
+ // Get configuration properties from the environment variables
+ String vfsCacheDir = System.getProperty(VFS_CACHE_DIR_PROPERTY);
+ if (null == vfsCacheDir || vfsCacheDir.isBlank()) {
+ printError(VFS_CACHE_DIR_PROPERTY + " system property not set, using default of "
+ + VFS_CACHE_DIR_DEFAULT);
+ vfsCacheDir = System.getProperty(VFS_CACHE_DIR_DEFAULT);
+ }
+ String cache = replaceEnvVars(vfsCacheDir, System.getenv());
+ printDebug("VFS Cache Dir set to: " + cache);
+ return cache;
+ }
+
+ /**
+ * Replace environment variables in the string with their actual value
+ */
+ public static String replaceEnvVars(String classpath, Map<String,String> env) {
+ Pattern envPat = Pattern.compile("\\$[A-Za-z][a-zA-Z0-9_]*");
+ Matcher envMatcher = envPat.matcher(classpath);
+ while (envMatcher.find(0)) {
+ // name comes after the '$'
+ String varName = envMatcher.group().substring(1);
+ String varValue = env.get(varName);
+ if (varValue == null) {
+ varValue = "";
+ }
+ classpath = (classpath.substring(0, envMatcher.start()) + varValue
+ + classpath.substring(envMatcher.end()));
+ envMatcher.reset(classpath);
+ }
+ return classpath;
+ }
+
+ /**
+ * Get the file system monitor interval
+ *
+ * @return monitor interval in ms
+ */
+ private static long getMonitorIntervalProperty() {
+ String interval = System.getProperty(VFS_CLASSPATH_MONITOR_INTERVAL);
+ if (null != interval && !interval.isBlank()) {
+ try {
+ return TimeUnit.SECONDS.toMillis(Long.parseLong(interval));
+ } catch (NumberFormatException e) {
+ printError(VFS_CLASSPATH_MONITOR_INTERVAL + " system property not set, using default of "
+ + DEFAULT_TIMEOUT);
+ return DEFAULT_TIMEOUT;
+ }
+ }
+ return DEFAULT_TIMEOUT;
+ }
+
+ /**
+ * This task replaces the delegate classloader with a new instance when the filesystem has
+ * changed. This will orphan the old classloader and the only references to the old classloader
+ * are from the objects that it loaded.
+ */
+ private final Runnable refresher = new Runnable() {
+ @Override
+ public void run() {
+ while (!executor.isTerminating()) {
+ try {
+ printDebug("Recreating delegate classloader due to filesystem change event");
+ updateDelegateClassloader();
+ return;
+ } catch (Exception e) {
+ e.printStackTrace();
+ try {
+ Thread.sleep(getMonitorInterval());
+ } catch (InterruptedException ie) {
+ ie.printStackTrace();
+ }
+ }
+ }
+ }
+ };
+
+ public ReloadingVFSClassLoader(ClassLoader parent) {
+ super(ReloadingVFSClassLoader.class.getSimpleName(), parent);
+ printDebug("Parent ClassLoader: " + parent.getClass().getName());
+ this.name = ReloadingVFSClassLoader.class.getSimpleName();
+ this.parent = parent;
+ this.classpath = CLASSPATH;
+ this.preDelegation = PRE_DELEGATION;
+ this.monitorInterval = MONITOR_INTERVAL;
+
+ BlockingQueue<Runnable> queue = new ArrayBlockingQueue<>(2);
+ ThreadFactory factory = r -> {
+ Thread t = new Thread(r);
+ t.setDaemon(true);
+ return t;
+ };
+ executor = new ThreadPoolExecutor(1, 1, 1, SECONDS, queue, factory);
+ }
+
+ protected DefaultFileSystemManager getFileSystem() {
+ if (null == this.vfs) {
+ if (DEBUG) {
+ VFSManager.enableDebug();
+ }
+ try {
+ this.vfs = VFSManager.generateVfs();
+ } catch (FileSystemException e) {
+ printError("Error creating FileSystem: " + e.getMessage());
+ e.printStackTrace();
+ }
+ printDebug("VFS File System created.");
+ }
+ return this.vfs;
+ }
+
+ protected String getClassPath() {
+ return this.classpath;
+ }
+
+ protected boolean isPreDelegationModel() {
+ return this.preDelegation;
+ }
+
+ protected long getMonitorInterval() {
+ return this.monitorInterval;
+ }
+
+ private synchronized FileMonitor getFileMonitor() {
+ if (null == this.monitor) {
+ this.monitor = new DefaultFileMonitor(this);
+ monitor.setDelay(getMonitorInterval());
+ monitor.setRecursive(false);
+ monitor.start();
+ printDebug("Monitor started with interval set to: " + monitor.getDelay());
+ }
+ return this.monitor;
+ }
+
+ private void addFileToMonitor(FileObject file) throws RuntimeException {
+ try {
+ getFileMonitor().addFile(file);
+ } catch (RuntimeException re) {
+ if (re.getMessage().contains("files-cache"))
+ printDebug("files-cache error adding " + file.toString() + " to VFS monitor. "
+ + "There is no implementation for files-cache in VFS2");
+ else
+ printDebug("Runtime error adding " + file.toString() + " to VFS monitor");
+
+ re.printStackTrace();
+
+ throw re;
+ }
+ }
+
+ private synchronized void updateDelegateClassloader() throws Exception {
+ try {
+ updateLock.writeLock().lock();
+ // Re-resolve the files on the classpath, things may have changed.
+ long retries = 0;
+ long currentSleepMillis = sleepInterval;
+ FileObject[] classpathFiles = VFSManager.resolve(getFileSystem(), this.getClassPath());
+ if (classpathFiles.length == 0) {
+ while (classpathFiles.length == 0 && retryPermitted(retries)) {
+ try {
+ printDebug("VFS path was empty. Waiting " + currentSleepMillis + " ms to retry");
+ Thread.sleep(currentSleepMillis);
+ classpathFiles = VFSManager.resolve(getFileSystem(), this.getClassPath());
+ retries++;
+ currentSleepMillis = Math.min(maxWaitInterval, currentSleepMillis + sleepInterval);
+ } catch (InterruptedException e) {
+ printError("VFS Retry Interruped");
+ e.printStackTrace();
+ throw new RuntimeException(e);
+ }
+ }
+ }
+ if (classpathFiles.length == 0) {
+ printError("ReloadingVFSClassLoader has no resources on classpath");
+ }
+ this.files = classpathFiles;
+ // There is a chance that the listener was removed from the top level directory or
+ // its children if they were deleted within some time window. Re-add files to be
+ // monitored. The Monitor will ignore files that are already/still being monitored.
+ // forEachCatchRTEs will capture a stream of thrown exceptions.
+ // and can collect them to list or reduce into one exception
+ forEachCatchRTEs(Arrays.stream(this.files), f -> {
+ addFileToMonitor(f);
+ printDebug("monitoring: " + f.toString());
+ });
+ // Create the new classloader delegate
+ printDebug("Rebuilding dynamic classloader using files: " + stringify(this.files));
+ VFSClassLoaderWrapper cl;
+ if (this.isPreDelegationModel()) {
+ // This is the normal classloader parent delegation model
+ cl = new VFSClassLoaderWrapper(this.files, getFileSystem(), parent);
+ } else {
+ // This delegates to the parent after we lookup locally first.
+ cl = new VFSClassLoaderWrapper(this.files, getFileSystem()) {
+ @Override
+ public synchronized Class<?> loadClass(String name, boolean resolve)
+ throws ClassNotFoundException {
+ Class<?> c = findLoadedClass(name);
+ if (c != null)
+ return c;
+ try {
+ // try finding this class here instead of parent
+ return findClass(name);
+ } catch (ClassNotFoundException e) {
+
+ }
+ return super.loadClass(name, resolve);
+ }
+ };
+ }
+ // An HDFS FileSystem and Configuration object were created for each unique HDFS namespace
+ // in the call to resolve above. The HDFS Client did us a favor and cached these objects
+ // so that the next time someone calls FileSystem.get(uri), they get the cached object.
+ // However, these objects were created not with the VFS classloader, but the
+ // classloader above it. We need to override the classloader on the Configuration objects.
+ // Ran into an issue were log recovery was being attempted and SequenceFile$Reader was
+ // trying to instantiate the key class via WritableName.getClass(String, Configuration)
+ printDebug("Setting ClassLoader on HDFS FileSystem objects");
+ for (FileObject fo : this.files) {
+ if (fo instanceof HdfsFileObject) {
+ String uri = fo.getName().getRootURI();
+ Configuration c = new Configuration(true);
+ c.set(FileSystem.FS_DEFAULT_NAME_KEY, uri);
+ try {
+ FileSystem fs = FileSystem.get(c);
+ fs.getConf().setClassLoader(cl);
+ } catch (IOException e) {
+ throw new RuntimeException("Error setting classloader on HDFS FileSystem object", e);
+ }
+ }
+ }
+
+ // Update the delegate reference to the new classloader
+ this.cl = cl;
+ printDebug("ReloadingVFSClassLoader set.");
+ } finally {
+ updateLock.writeLock().unlock();
+ }
+ }
+
+ /**
+ * Remove the file from the monitor
+ *
+ * @param file
+ * to remove
+ * @throws RuntimeException
+ * if error
+ */
+ private void removeFile(FileObject file) throws RuntimeException {
+ try {
+ getFileMonitor().removeFile(file);
+ } catch (RuntimeException re) {
+ printError("Error removing file from VFS cache: " + file.toString());
+ re.printStackTrace();
+ throw re;
+ }
+ }
+
+ @Override
+ public void fileCreated(FileChangeEvent event) throws Exception {
+ printDebug(event.getFileObject().getURL().toString() + " created, recreating classloader");
+ scheduleRefresh();
+ }
+
+ @Override
+ public void fileDeleted(FileChangeEvent event) throws Exception {
+ printDebug(event.getFileObject().getURL().toString() + " deleted, recreating classloader");
+ scheduleRefresh();
+ }
+
+ @Override
+ public void fileChanged(FileChangeEvent event) throws Exception {
+ printDebug(event.getFileObject().getURL().toString() + " changed, recreating classloader");
+ scheduleRefresh();
+ }
+
+ private void scheduleRefresh() {
+ try {
+ executor.execute(refresher);
+ } catch (RejectedExecutionException e) {
+ printDebug("Ignoring refresh request (already refreshing)");
+ }
+ }
+
+ @Override
+ public void close() {
+
+ forEachCatchRTEs(Stream.of(this.files), f -> {
+ removeFile(f);
+ printDebug("Closing, removing file from monitoring: " + f.toString());
+ });
+
+ this.executor.shutdownNow();
+ this.monitor.stop();
+ if (null != this.vfs)
+ VFSManager.returnVfs(this.vfs);
+ vfs = null;
+ }
+
+ public static <T> void forEachCatchRTEs(Stream<T> stream, Consumer<T> consumer) {
+ stream.flatMap(o -> {
+ try {
+ consumer.accept(o);
+ return null;
+ } catch (RuntimeException e) {
+ return Stream.of(e);
+ }
+ }).reduce((e1, e2) -> {
+ e1.addSuppressed(e2);
+ return e1;
+ }).ifPresent(e -> {
+ throw e;
+ });
+ }
+
+ private boolean retryPermitted(long retries) {
+ return (this.maxRetries < 0 || retries < this.maxRetries);
+ }
+
+ public String stringify(FileObject[] files) {
+ StringBuilder sb = new StringBuilder();
+ sb.append('[');
+ String delim = "";
+ for (FileObject file : files) {
+ sb.append(delim);
+ delim = ", ";
+ sb.append(file.getName());
+ }
+ sb.append(']');
+ return sb.toString();
+ }
+
+ /**
+ * Return a reference to the delegate classloader, create a new one if necessary
+ *
+ * @return reference to delegate classloader
+ */
+ synchronized ClassLoader getDelegateClassLoader() {
+ // We cannot create the VFS file system during VM initialization,
+ // we have to perform some lazy initialization here due to the fact
+ // that the logging libraries (and others) make use of the ServiceLoader
+ // and call ClassLoader.getSystemClassLoader() which you can't do until
+ // the VM is fully initialized.
+ if (!isVMInitialized() || vfsInitializing) {
+ return this.parent;
+ } else if (null == this.vfs) {
+ this.vfsInitializing = true;
+ printDebug("getDelegateClassLoader() initializing VFS.");
+ getFileSystem();
+ if (null == getFileSystem()) {
+ // Some error happened
+ throw new RuntimeException("Problem creating VFS file system");
+ }
+ printDebug("getDelegateClassLoader() VFS initialized.");
+ }
+ if (null == this.cl) {
+ try {
+ printDebug("Creating initial delegate class loader");
+ updateDelegateClassloader();
+ } catch (Exception e) {
+ e.printStackTrace();
+ throw new RuntimeException("Error creating initial delegate classloader", e);
+ }
+ }
+ if (this.vfsInitializing) {
+ this.vfsInitializing = false;
+ printDebug(ClassPathPrinter.getClassPath(this, true));
+ }
+ try {
+ updateLock.readLock().lock();
+ return this.cl;
+ } finally {
+ updateLock.readLock().unlock();
+ }
+ }
+
+ @Override
+ public Class<?> findClass(String name) throws ClassNotFoundException {
+ ClassLoader d = getDelegateClassLoader();
+ if (d instanceof VFSClassLoaderWrapper) {
+ return ((VFSClassLoaderWrapper) d).findClass(name);
+ } else {
+ return null;
+ }
+ }
+
+ @Override
+ public URL findResource(String name) {
+ ClassLoader d = getDelegateClassLoader();
+ if (d instanceof VFSClassLoaderWrapper) {
+ return ((VFSClassLoaderWrapper) d).findResource(name);
+ } else {
+ return null;
+ }
+ }
+
+ @Override
+ public Enumeration<URL> findResources(String name) throws IOException {
+ ClassLoader d = getDelegateClassLoader();
+ if (d instanceof VFSClassLoaderWrapper) {
+ return ((VFSClassLoaderWrapper) d).findResources(name);
+ } else {
+ return null;
+ }
+ }
+
+ @Override
+ public Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
+ ClassLoader d = getDelegateClassLoader();
+ if (d instanceof VFSClassLoaderWrapper) {
+ return ((VFSClassLoaderWrapper) d).loadClass(name, resolve);
+ } else {
+ return null;
+ }
+ }
+
+ @Override
+ public String getName() {
+ return name;
+ }
+
+ private boolean isVMInitialized() {
+ if (VM_INITIALIZED) {
+ return VM_INITIALIZED;
+ } else {
+ // We can't call VM.isBooted() directly, but we know from System.initPhase3() that
+ // when this classloader is set via 'java.system.class.loader' that it will be initialized,
+ // then set as the Thread context classloader, then the VM is fully initialized.
+ try {
+ printDebug(
+ "System ClassLoader: " + ClassLoader.getSystemClassLoader().getClass().getName());
+ VM_INITIALIZED = ClassLoader.getSystemClassLoader().equals(this);
+ } catch (IllegalStateException e) {
+ // VM is still initializing
+ VM_INITIALIZED = false;
+ }
+ printDebug("VM Initialized: " + VM_INITIALIZED);
+ return VM_INITIALIZED;
+ }
+ }
+
+ @Override
+ public Class<?> loadClass(String name) throws ClassNotFoundException {
+ return getDelegateClassLoader().loadClass(name);
+ }
+
+ @Override
+ public URL getResource(String name) {
+ return getDelegateClassLoader().getResource(name);
+ }
+
+ @Override
+ public Enumeration<URL> getResources(String name) throws IOException {
+ return getDelegateClassLoader().getResources(name);
+ }
+
+ @Override
+ public Stream<URL> resources(String name) {
+ return getDelegateClassLoader().resources(name);
+ }
+
+ @Override
+ public InputStream getResourceAsStream(String name) {
+ return getDelegateClassLoader().getResourceAsStream(name);
+ }
+
+ @Override
+ public void setDefaultAssertionStatus(boolean enabled) {
+ getDelegateClassLoader().setDefaultAssertionStatus(enabled);
+ }
+
+ @Override
+ public void setPackageAssertionStatus(String packageName, boolean enabled) {
+ getDelegateClassLoader().setPackageAssertionStatus(packageName, enabled);
+ }
+
+ @Override
+ public void setClassAssertionStatus(String className, boolean enabled) {
+ getDelegateClassLoader().setClassAssertionStatus(className, enabled);
+ }
+
+ @Override
+ public void clearAssertionStatus() {
+ getDelegateClassLoader().clearAssertionStatus();
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result + ((name == null) ? 0 : name.hashCode());
+ result = prime * result + ((parent.getName() == null) ? 0 : parent.getName().hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (this == obj)
+ return true;
+ if (obj == null)
+ return false;
+ if (getClass() != obj.getClass())
+ return false;
+ ReloadingVFSClassLoader other = (ReloadingVFSClassLoader) obj;
+ if (name == null) {
+ if (other.name != null)
+ return false;
+ } else if (!name.equals(other.name))
+ return false;
+ if (parent == null) {
+ if (other.parent != null)
+ return false;
+ } else if (!parent.getName().equals(other.parent.getName()))
+ return false;
+ return true;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder buf = new StringBuilder();
+
+ if (null != this.files) {
+ for (FileObject f : files) {
+ try {
+ buf.append("\t").append(f.getURL()).append("\n");
+ } catch (FileSystemException e) {
+ printError("Error getting URL for file: " + f.toString());
+ e.printStackTrace();
+ }
+ }
+ }
+ return buf.toString();
+ }
+
+ // VisibleForTesting intentionally not using annotation from Guava
+ // because it adds unwanted dependency
+ void setMaxRetries(long maxRetries) {
+ this.maxRetries = maxRetries;
+ }
+
+ // VisibleForTesting intentionally not using annotation from Guava
+ // because it adds unwanted dependency
+ void setVMInitializedForTests() {
+ VM_INITIALIZED = true;
+ }
+
+ // VisibleForTesting intentionally not using annotation from Guava
+ // because it adds unwanted dependency
+ void setVFSForTests(DefaultFileSystemManager vfs) {
+ this.vfs = vfs;
+ }
+
+ void enableDebugForTests() {
+ DEBUG = true;
+ }
+}
diff --git a/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/UniqueFileReplicator.java b/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/UniqueFileReplicator.java
new file mode 100644
index 0000000..abef05a
--- /dev/null
+++ b/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/UniqueFileReplicator.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.vfs;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSelector;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.provider.FileReplicator;
+import org.apache.commons.vfs2.provider.UriParser;
+import org.apache.commons.vfs2.provider.VfsComponent;
+import org.apache.commons.vfs2.provider.VfsComponentContext;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+public class UniqueFileReplicator implements VfsComponent, FileReplicator {
+
+ private static final char[] TMP_RESERVED_CHARS =
+ {'?', '/', '\\', ' ', '&', '"', '\'', '*', '#', ';', ':', '<', '>', '|'};
+
+ private File tempDir;
+ private VfsComponentContext context;
+ private List<File> tmpFiles = Collections.synchronizedList(new ArrayList<>());
+
+ public UniqueFileReplicator(File tempDir) {
+ this.tempDir = tempDir;
+ if (!tempDir.exists() && !tempDir.mkdirs())
+ System.out.println("Unexpected error creating directory: " + tempDir.getAbsolutePath());
+ }
+
+ @SuppressFBWarnings(value = "PATH_TRAVERSAL_IN",
+ justification = "input files are specified by admin, not unchecked user input")
+ @Override
+ public File replicateFile(FileObject srcFile, FileSelector selector) throws FileSystemException {
+ String baseName = srcFile.getName().getBaseName();
+
+ try {
+ String safeBasename = UriParser.encode(baseName, TMP_RESERVED_CHARS).replace('%', '_');
+ File file = File.createTempFile("vfsr_", "_" + safeBasename, tempDir);
+ file.deleteOnExit();
+
+ final FileObject destFile = context.toFileObject(file);
+ destFile.copyFrom(srcFile, selector);
+
+ return file;
+ } catch (IOException e) {
+ throw new FileSystemException(e);
+ }
+ }
+
+ @Override
+ public void setLogger(Log logger) {
+ // TODO Auto-generated method stub
+
+ }
+
+ @Override
+ public void setContext(VfsComponentContext context) {
+ this.context = context;
+ }
+
+ @Override
+ public void init() throws FileSystemException {
+
+ }
+
+ @Override
+ public void close() {
+ synchronized (tmpFiles) {
+ for (File tmpFile : tmpFiles) {
+ if (!tmpFile.delete())
+ System.out.println("File does not exist: " + tmpFile.getAbsolutePath());
+ }
+ }
+
+ if (tempDir.exists()) {
+ String[] list = tempDir.list();
+ int numChildren = list == null ? 0 : list.length;
+ if (numChildren == 0 && !tempDir.delete())
+ System.out.println("Cannot delete empty directory: " + tempDir.getAbsolutePath());
+ }
+ }
+}
diff --git a/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/VFSClassLoaderWrapper.java b/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/VFSClassLoaderWrapper.java
new file mode 100644
index 0000000..7f17684
--- /dev/null
+++ b/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/VFSClassLoaderWrapper.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.vfs;
+
+import java.io.IOException;
+import java.net.URL;
+import java.security.CodeSource;
+import java.security.PermissionCollection;
+import java.util.Enumeration;
+
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.commons.vfs2.impl.VFSClassLoader;
+
+/**
+ * This class exists to expose methods that are protected in the parent class so that we can use
+ * this in a delegate pattern
+ */
+public class VFSClassLoaderWrapper extends VFSClassLoader {
+
+ public VFSClassLoaderWrapper(FileObject file, FileSystemManager manager, ClassLoader parent)
+ throws FileSystemException {
+ super(file, manager, parent);
+ }
+
+ public VFSClassLoaderWrapper(FileObject file, FileSystemManager manager)
+ throws FileSystemException {
+ super(file, manager);
+ }
+
+ public VFSClassLoaderWrapper(FileObject[] files, FileSystemManager manager, ClassLoader parent)
+ throws FileSystemException {
+ super(files, manager, parent);
+ }
+
+ public VFSClassLoaderWrapper(FileObject[] files, FileSystemManager manager)
+ throws FileSystemException {
+ super(files, manager);
+ }
+
+ @Override
+ public Class<?> findClass(String name) throws ClassNotFoundException {
+ return super.findClass(name);
+ }
+
+ @Override
+ public PermissionCollection getPermissions(CodeSource cs) {
+ return super.getPermissions(cs);
+ }
+
+ @Override
+ public URL findResource(String name) {
+ return super.findResource(name);
+ }
+
+ @Override
+ public Enumeration<URL> findResources(String name) throws IOException {
+ return super.findResources(name);
+ }
+
+ @Override
+ public Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
+ return super.loadClass(name, resolve);
+ }
+
+}
diff --git a/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/VFSManager.java b/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/VFSManager.java
new file mode 100644
index 0000000..ba153d0
--- /dev/null
+++ b/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/VFSManager.java
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.vfs;
+
+import java.io.File;
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.lang.ref.WeakReference;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.vfs2.CacheStrategy;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.commons.vfs2.FileType;
+import org.apache.commons.vfs2.cache.SoftRefFilesCache;
+import org.apache.commons.vfs2.impl.DefaultFileSystemManager;
+import org.apache.commons.vfs2.impl.FileContentInfoFilenameFactory;
+import org.apache.commons.vfs2.provider.FileReplicator;
+import org.apache.commons.vfs2.provider.hdfs.HdfsFileProvider;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+public class VFSManager {
+
+ public static class AccumuloVFSManagerShutdownThread implements Runnable {
+
+ @Override
+ public void run() {
+ try {
+ VFSManager.close();
+ } catch (Exception e) {
+ // do nothing, we are shutting down anyway
+ }
+ }
+ }
+
+ private static List<WeakReference<DefaultFileSystemManager>> vfsInstances =
+ Collections.synchronizedList(new ArrayList<>());
+ private static volatile boolean DEBUG = false;
+
+ static void enableDebug() {
+ DEBUG = true;
+ }
+
+ static {
+ // Register the shutdown hook
+ Runtime.getRuntime().addShutdownHook(new Thread(new AccumuloVFSManagerShutdownThread()));
+ }
+
+ public static FileObject[] resolve(FileSystemManager vfs, String uris)
+ throws FileSystemException {
+ return resolve(vfs, uris, new ArrayList<>());
+ }
+
+ static FileObject[] resolve(FileSystemManager vfs, String uris,
+ ArrayList<FileObject> pathsToMonitor) throws FileSystemException {
+ if (uris == null) {
+ return new FileObject[0];
+ }
+
+ ArrayList<FileObject> classpath = new ArrayList<>();
+
+ pathsToMonitor.clear();
+
+ for (String path : uris.split(",")) {
+
+ path = path.trim();
+
+ if (path.equals("")) {
+ continue;
+ }
+
+ path = ReloadingVFSClassLoader.replaceEnvVars(path, System.getenv());
+
+ FileObject fo = vfs.resolveFile(path);
+
+ switch (fo.getType()) {
+ case FILE:
+ case FOLDER:
+ classpath.add(fo);
+ pathsToMonitor.add(fo);
+ break;
+ case IMAGINARY:
+ // assume its a pattern
+ String pattern = fo.getName().getBaseName();
+ if (fo.getParent() != null) {
+ // still monitor the parent
+ pathsToMonitor.add(fo.getParent());
+ if (fo.getParent().getType() == FileType.FOLDER) {
+ FileObject[] children = fo.getParent().getChildren();
+ for (FileObject child : children) {
+ if (child.getType() == FileType.FILE
+ && child.getName().getBaseName().matches(pattern)) {
+ classpath.add(child);
+ }
+ }
+ } else {
+ if (DEBUG)
+ System.out.println("classpath entry " + fo.getParent().toString() + " is "
+ + fo.getParent().getType().toString());
+ }
+ } else {
+ if (DEBUG)
+ System.out.println("ignoring classpath entry: " + fo.toString());
+ }
+ break;
+ default:
+ System.out.println("ignoring classpath entry: " + fo.toString());
+ break;
+ }
+
+ }
+
+ return classpath.toArray(new FileObject[classpath.size()]);
+ }
+
+ public static DefaultFileSystemManager generateVfs() throws FileSystemException {
+ DefaultFileSystemManager vfs = new DefaultFileSystemManager();
+ vfs.addProvider("res", new org.apache.commons.vfs2.provider.res.ResourceFileProvider());
+ vfs.addProvider("zip", new org.apache.commons.vfs2.provider.zip.ZipFileProvider());
+ vfs.addProvider("gz", new org.apache.commons.vfs2.provider.gzip.GzipFileProvider());
+ vfs.addProvider("ram", new org.apache.commons.vfs2.provider.ram.RamFileProvider());
+ vfs.addProvider("file", new org.apache.commons.vfs2.provider.local.DefaultLocalFileProvider());
+ vfs.addProvider("jar", new org.apache.commons.vfs2.provider.jar.JarFileProvider());
+ vfs.addProvider("http", new org.apache.commons.vfs2.provider.http.HttpFileProvider());
+ vfs.addProvider("https", new org.apache.commons.vfs2.provider.https.HttpsFileProvider());
+ vfs.addProvider("ftp", new org.apache.commons.vfs2.provider.ftp.FtpFileProvider());
+ vfs.addProvider("ftps", new org.apache.commons.vfs2.provider.ftps.FtpsFileProvider());
+ vfs.addProvider("war", new org.apache.commons.vfs2.provider.jar.JarFileProvider());
+ vfs.addProvider("par", new org.apache.commons.vfs2.provider.jar.JarFileProvider());
+ vfs.addProvider("ear", new org.apache.commons.vfs2.provider.jar.JarFileProvider());
+ vfs.addProvider("sar", new org.apache.commons.vfs2.provider.jar.JarFileProvider());
+ vfs.addProvider("ejb3", new org.apache.commons.vfs2.provider.jar.JarFileProvider());
+ vfs.addProvider("tmp", new org.apache.commons.vfs2.provider.temp.TemporaryFileProvider());
+ vfs.addProvider("tar", new org.apache.commons.vfs2.provider.tar.TarFileProvider());
+ vfs.addProvider("tbz2", new org.apache.commons.vfs2.provider.tar.TarFileProvider());
+ vfs.addProvider("tgz", new org.apache.commons.vfs2.provider.tar.TarFileProvider());
+ vfs.addProvider("bz2", new org.apache.commons.vfs2.provider.bzip2.Bzip2FileProvider());
+ vfs.addProvider("hdfs", new HdfsFileProvider());
+ vfs.addExtensionMap("jar", "jar");
+ vfs.addExtensionMap("zip", "zip");
+ vfs.addExtensionMap("gz", "gz");
+ vfs.addExtensionMap("tar", "tar");
+ vfs.addExtensionMap("tbz2", "tar");
+ vfs.addExtensionMap("tgz", "tar");
+ vfs.addExtensionMap("bz2", "bz2");
+ vfs.addMimeTypeMap("application/x-tar", "tar");
+ vfs.addMimeTypeMap("application/x-gzip", "gz");
+ vfs.addMimeTypeMap("application/zip", "zip");
+ vfs.setFileContentInfoFactory(new FileContentInfoFilenameFactory());
+ vfs.setFilesCache(new SoftRefFilesCache());
+ File cacheDir = computeTopCacheDir();
+ vfs.setReplicator(new UniqueFileReplicator(cacheDir));
+ vfs.setCacheStrategy(CacheStrategy.ON_RESOLVE);
+ vfs.init();
+ vfsInstances.add(new WeakReference<>(vfs));
+ return vfs;
+ }
+
+ @SuppressFBWarnings(value = "PATH_TRAVERSAL_IN",
+ justification = "tmpdir is controlled by admin, not unchecked user input")
+ private static File computeTopCacheDir() {
+ String cacheDirPath = ReloadingVFSClassLoader.getVFSCacheDir();
+ String procName = ManagementFactory.getRuntimeMXBean().getName();
+ return new File(cacheDirPath,
+ "accumulo-vfs-manager-cache-" + procName + "-" + System.getProperty("user.name", "nouser"));
+ }
+
+ public static void returnVfs(DefaultFileSystemManager vfs) {
+ if (DEBUG) {
+ System.out.println("Closing VFS instance.");
+ }
+ FileReplicator replicator;
+ try {
+ replicator = vfs.getReplicator();
+ if (replicator instanceof UniqueFileReplicator) {
+ ((UniqueFileReplicator) replicator).close();
+ }
+ } catch (FileSystemException e) {
+ System.err.println("Error occurred closing VFS instance: " + e.getMessage());
+ }
+ vfs.close();
+ }
+
+ public static void close() {
+ for (WeakReference<DefaultFileSystemManager> vfsInstance : vfsInstances) {
+ DefaultFileSystemManager ref = vfsInstance.get();
+ if (ref != null) {
+ returnVfs(ref);
+ }
+ }
+ try {
+ FileUtils.deleteDirectory(computeTopCacheDir());
+ } catch (IOException e) {
+ System.err.println("IOException deleting cache directory");
+ e.printStackTrace();
+ }
+ }
+}
diff --git a/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/context/ReloadingVFSContextClassLoaderFactory.java b/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/context/ReloadingVFSContextClassLoaderFactory.java
new file mode 100644
index 0000000..a4f93f9
--- /dev/null
+++ b/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/context/ReloadingVFSContextClassLoaderFactory.java
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.vfs.context;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.accumulo.classloader.vfs.ReloadingVFSClassLoader;
+import org.apache.accumulo.core.spi.common.ClassLoaderFactory;
+
+import com.google.gson.Gson;
+
+/**
+ * A ClassLoaderFactory implementation that uses a ReloadingVFSClassLoader per defined context.
+ * Configuration of this class is done with a JSON file whose location is defined by the system
+ * property <b>vfs.context.class.loader.config</b>. To use this ClassLoaderFactory you need to set
+ * the Accumulo configuration property <b>general.context.factory</b> to the fully qualified name of
+ * this class, create a configuration file that defines the supported contexts and their
+ * configuration, and set <b>vfs.context.class.loader.config</b> to the location of the
+ * configuration file.
+ *
+ * <p>
+ * Example configuration file:
+ *
+ * <pre>
+ * {
+ * "contexts": [
+ * {
+ * "name": "cx1",
+ * "config": {
+ * "classPath": "file:///tmp/foo",
+ * "postDelegate": true,
+ * "monitorIntervalMs": 30000
+ * }
+ * },
+ * {
+ * "name": "cx2",
+ * "config": {
+ * "classPath": "file:///tmp/bar",
+ * "postDelegate": false,
+ * "monitorIntervalMs": 30000
+ * }
+ * }
+ * ]
+ * }
+ * </pre>
+ */
+public class ReloadingVFSContextClassLoaderFactory implements ClassLoaderFactory {
+
+ public static class Contexts {
+ List<Context> contexts;
+
+ public List<Context> getContexts() {
+ return contexts;
+ }
+
+ public void setContexts(List<Context> contexts) {
+ this.contexts = contexts;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result + ((contexts == null) ? 0 : contexts.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (this == obj)
+ return true;
+ if (obj == null)
+ return false;
+ if (getClass() != obj.getClass())
+ return false;
+ Contexts other = (Contexts) obj;
+ if (contexts == null) {
+ if (other.contexts != null)
+ return false;
+ } else if (!contexts.equals(other.contexts))
+ return false;
+ return true;
+ }
+ }
+
+ public static class Context {
+ private String name;
+ private ContextConfig config;
+
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ public ContextConfig getConfig() {
+ return config;
+ }
+
+ public void setConfig(ContextConfig config) {
+ this.config = config;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result + ((config == null) ? 0 : config.hashCode());
+ result = prime * result + ((name == null) ? 0 : name.hashCode());
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (this == obj)
+ return true;
+ if (obj == null)
+ return false;
+ if (getClass() != obj.getClass())
+ return false;
+ Context other = (Context) obj;
+ if (config == null) {
+ if (other.config != null)
+ return false;
+ } else if (!config.equals(other.config))
+ return false;
+ if (name == null) {
+ if (other.name != null)
+ return false;
+ } else if (!name.equals(other.name))
+ return false;
+ return true;
+ }
+ }
+
+ public static class ContextConfig {
+ private String classPath;
+ private boolean postDelegate;
+ private long monitorIntervalMs;
+
+ public String getClassPath() {
+ return classPath;
+ }
+
+ public void setClassPath(String classPath) {
+ this.classPath = ReloadingVFSClassLoader.replaceEnvVars(classPath, System.getenv());
+ }
+
+ public boolean getPostDelegate() {
+ return postDelegate;
+ }
+
+ public void setPostDelegate(boolean postDelegate) {
+ this.postDelegate = postDelegate;
+ }
+
+ public long getMonitorIntervalMs() {
+ return monitorIntervalMs;
+ }
+
+ public void setMonitorIntervalMs(long monitorIntervalMs) {
+ this.monitorIntervalMs = monitorIntervalMs;
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result + ((classPath == null) ? 0 : classPath.hashCode());
+ result = prime * result + (int) (monitorIntervalMs ^ (monitorIntervalMs >>> 32));
+ result = prime * result + (postDelegate ? 1231 : 1237);
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (this == obj)
+ return true;
+ if (obj == null)
+ return false;
+ if (getClass() != obj.getClass())
+ return false;
+ ContextConfig other = (ContextConfig) obj;
+ if (classPath == null) {
+ if (other.classPath != null)
+ return false;
+ } else if (!classPath.equals(other.classPath))
+ return false;
+ if (monitorIntervalMs != other.monitorIntervalMs)
+ return false;
+ if (postDelegate != other.postDelegate)
+ return false;
+ return true;
+ }
+ }
+
+ public static final String CONFIG_LOCATION = "vfs.context.class.loader.config";
+ private static final Map<String,ReloadingVFSClassLoader> CONTEXTS = new HashMap<>();
+
+ protected String getConfigFileLocation() {
+ String loc = System.getProperty(CONFIG_LOCATION);
+ if (null == loc || loc.isBlank()) {
+ throw new RuntimeException(CONFIG_LOCATION
+ + " system property must be set to use ReloadingVFSContextClassLoaderFactory");
+ }
+ return loc;
+ }
+
+ @Override
+ public void initialize(ClassLoaderFactoryConfiguration conf) throws Exception {
+ // Properties
+ File f = new File(getConfigFileLocation());
+ if (!f.canRead()) {
+ throw new RuntimeException("Unable to read configuration file: " + f.getAbsolutePath());
+ }
+ Gson g = new Gson();
+ Contexts con = g.fromJson(Files.newBufferedReader(f.toPath()), Contexts.class);
+
+ con.getContexts().forEach(c -> {
+ CONTEXTS.put(c.getName(), new ReloadingVFSClassLoader(
+ ReloadingVFSContextClassLoaderFactory.class.getClassLoader()) {
+ @Override
+ protected String getClassPath() {
+ return c.getConfig().getClassPath();
+ }
+
+ @Override
+ protected boolean isPreDelegationModel() {
+ return !(c.getConfig().getPostDelegate());
+ }
+
+ @Override
+ protected long getMonitorInterval() {
+ return c.getConfig().getMonitorIntervalMs();
+ }
+ });
+ });
+ }
+
+ @Override
+ public ClassLoader getClassLoader(String contextName) throws IllegalArgumentException {
+ if (!CONTEXTS.containsKey(contextName)) {
+ throw new IllegalArgumentException(
+ "ReloadingVFSContextClassLoaderFactory not configured for context: " + contextName);
+ }
+ return CONTEXTS.get(contextName);
+ }
+
+}
diff --git a/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/AccumuloDFSBase.java b/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/AccumuloDFSBase.java
new file mode 100644
index 0000000..4c6c27d
--- /dev/null
+++ b/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/AccumuloDFSBase.java
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.vfs;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+
+import org.apache.accumulo.start.classloader.vfs.MiniDFSUtil;
+import org.apache.commons.vfs2.CacheStrategy;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.cache.DefaultFilesCache;
+import org.apache.commons.vfs2.cache.SoftRefFilesCache;
+import org.apache.commons.vfs2.impl.DefaultFileReplicator;
+import org.apache.commons.vfs2.impl.DefaultFileSystemManager;
+import org.apache.commons.vfs2.impl.FileContentInfoFilenameFactory;
+import org.apache.commons.vfs2.provider.hdfs.HdfsFileProvider;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+@SuppressFBWarnings(value = "PATH_TRAVERSAL_IN", justification = "paths not set by user input")
+public class AccumuloDFSBase {
+
+ protected static Configuration conf = null;
+ protected static DefaultFileSystemManager vfs = null;
+ protected static MiniDFSCluster cluster = null;
+
+ private static URI HDFS_URI;
+
+ protected static URI getHdfsUri() {
+ return HDFS_URI;
+ }
+
+ @BeforeClass
+ public static void miniDfsClusterSetup() {
+ System.setProperty("java.io.tmpdir", System.getProperty("user.dir") + "/target");
+
+ // Put the MiniDFSCluster directory in the target directory
+ System.setProperty("test.build.data", "target/build/test/data");
+
+ // Setup HDFS
+ conf = new Configuration();
+ conf.set("hadoop.security.token.service.use_ip", "true");
+
+ conf.set("dfs.datanode.data.dir.perm", MiniDFSUtil.computeDatanodeDirectoryPermission());
+ conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 1024 * 1024); // 1M blocksize
+
+ try {
+ cluster = new MiniDFSCluster.Builder(conf).build();
+ cluster.waitClusterUp();
+ // We can't assume that the hostname of "localhost" will still be "localhost" after
+ // starting up the NameNode. We may get mapped into a FQDN via settings in /etc/hosts.
+ HDFS_URI = cluster.getFileSystem().getUri();
+ } catch (IOException e) {
+ throw new RuntimeException("Error setting up mini cluster", e);
+ }
+
+ // Set up the VFS
+ vfs = new DefaultFileSystemManager();
+ try {
+ vfs.setFilesCache(new DefaultFilesCache());
+ vfs.addProvider("res", new org.apache.commons.vfs2.provider.res.ResourceFileProvider());
+ vfs.addProvider("zip", new org.apache.commons.vfs2.provider.zip.ZipFileProvider());
+ vfs.addProvider("gz", new org.apache.commons.vfs2.provider.gzip.GzipFileProvider());
+ vfs.addProvider("ram", new org.apache.commons.vfs2.provider.ram.RamFileProvider());
+ vfs.addProvider("file",
+ new org.apache.commons.vfs2.provider.local.DefaultLocalFileProvider());
+ vfs.addProvider("jar", new org.apache.commons.vfs2.provider.jar.JarFileProvider());
+ vfs.addProvider("http", new org.apache.commons.vfs2.provider.http.HttpFileProvider());
+ vfs.addProvider("https", new org.apache.commons.vfs2.provider.https.HttpsFileProvider());
+ vfs.addProvider("ftp", new org.apache.commons.vfs2.provider.ftp.FtpFileProvider());
+ vfs.addProvider("ftps", new org.apache.commons.vfs2.provider.ftps.FtpsFileProvider());
+ vfs.addProvider("war", new org.apache.commons.vfs2.provider.jar.JarFileProvider());
+ vfs.addProvider("par", new org.apache.commons.vfs2.provider.jar.JarFileProvider());
+ vfs.addProvider("ear", new org.apache.commons.vfs2.provider.jar.JarFileProvider());
+ vfs.addProvider("sar", new org.apache.commons.vfs2.provider.jar.JarFileProvider());
+ vfs.addProvider("ejb3", new org.apache.commons.vfs2.provider.jar.JarFileProvider());
+ vfs.addProvider("tmp", new org.apache.commons.vfs2.provider.temp.TemporaryFileProvider());
+ vfs.addProvider("tar", new org.apache.commons.vfs2.provider.tar.TarFileProvider());
+ vfs.addProvider("tbz2", new org.apache.commons.vfs2.provider.tar.TarFileProvider());
+ vfs.addProvider("tgz", new org.apache.commons.vfs2.provider.tar.TarFileProvider());
+ vfs.addProvider("bz2", new org.apache.commons.vfs2.provider.bzip2.Bzip2FileProvider());
+ vfs.addProvider("hdfs", new HdfsFileProvider());
+ vfs.addExtensionMap("jar", "jar");
+ vfs.addExtensionMap("zip", "zip");
+ vfs.addExtensionMap("gz", "gz");
+ vfs.addExtensionMap("tar", "tar");
+ vfs.addExtensionMap("tbz2", "tar");
+ vfs.addExtensionMap("tgz", "tar");
+ vfs.addExtensionMap("bz2", "bz2");
+ vfs.addMimeTypeMap("application/x-tar", "tar");
+ vfs.addMimeTypeMap("application/x-gzip", "gz");
+ vfs.addMimeTypeMap("application/zip", "zip");
+ vfs.setFileContentInfoFactory(new FileContentInfoFilenameFactory());
+ vfs.setFilesCache(new SoftRefFilesCache());
+ vfs.setReplicator(new DefaultFileReplicator(new File(System.getProperty("java.io.tmpdir"),
+ "accumulo-vfs-cache-" + System.getProperty("user.name", "nouser"))));
+ vfs.setCacheStrategy(CacheStrategy.ON_RESOLVE);
+ vfs.init();
+ } catch (FileSystemException e) {
+ throw new RuntimeException("Error setting up VFS", e);
+ }
+
+ }
+
+ @AfterClass
+ public static void tearDownMiniDfsCluster() {
+ if (null != cluster) {
+ cluster.shutdown();
+ }
+ }
+
+}
diff --git a/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/ClassPathPrinterTest.java b/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/ClassPathPrinterTest.java
new file mode 100644
index 0000000..6845b2a
--- /dev/null
+++ b/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/ClassPathPrinterTest.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.vfs;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.io.File;
+import java.net.MalformedURLException;
+
+import org.apache.commons.vfs2.impl.DefaultFileSystemManager;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+@SuppressFBWarnings(value = "PATH_TRAVERSAL_IN", justification = "paths not set by user input")
+public class ClassPathPrinterTest {
+
+ @Rule
+ public TemporaryFolder folder1 =
+ new TemporaryFolder(new File(System.getProperty("user.dir") + "/target"));
+
+ private final ClassLoader parent = ClassPathPrinterTest.class.getClassLoader();
+
+ private static void assertPattern(String output, String pattern, boolean shouldMatch) {
+ if (shouldMatch) {
+ assertTrue("Pattern " + pattern + " did not match output: " + output,
+ output.matches(pattern));
+ } else {
+ assertFalse("Pattern " + pattern + " should not match output: " + output,
+ output.matches(pattern));
+ }
+ }
+
+ @Test
+ public void testPrintClassPath() throws Exception {
+ File conf = folder1.newFile("accumulo.properties");
+ DefaultFileSystemManager vfs = VFSManager.generateVfs();
+
+ ReloadingVFSClassLoader cl = new ReloadingVFSClassLoader(parent) {
+ @Override
+ protected String getClassPath() {
+ try {
+ return conf.toURI().toURL().toString();
+ } catch (MalformedURLException e) {
+ throw new RuntimeException("URL problem", e);
+ }
+ }
+
+ @Override
+ protected DefaultFileSystemManager getFileSystem() {
+ return vfs;
+ }
+ };
+ cl.setVMInitializedForTests();
+ cl.setVFSForTests(vfs);
+
+ assertPattern(ClassPathPrinter.getClassPath(cl, true), "(?s).*\\s+.*\\n$", true);
+ assertTrue(ClassPathPrinter.getClassPath(cl, true)
+ .contains("Level 3: ReloadingVFSClassLoader, classpath items are"));
+ assertTrue(ClassPathPrinter.getClassPath(cl, true).length()
+ > ClassPathPrinter.getClassPath(cl, false).length());
+ assertPattern(ClassPathPrinter.getClassPath(cl, false), "(?s).*\\s+.*\\n$", false);
+ assertFalse(ClassPathPrinter.getClassPath(cl, false)
+ .contains("Level 3: ReloadingVFSClassLoader, classpath items are"));
+ }
+}
diff --git a/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/ReloadingVFSClassLoaderTest.java b/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/ReloadingVFSClassLoaderTest.java
new file mode 100644
index 0000000..739b5ac
--- /dev/null
+++ b/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/ReloadingVFSClassLoaderTest.java
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.vfs;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertNotSame;
+import static org.junit.Assert.assertTrue;
+
+import java.io.File;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.impl.DefaultFileSystemManager;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+@SuppressFBWarnings(value = "PATH_TRAVERSAL_IN", justification = "paths not set by user input")
+public class ReloadingVFSClassLoaderTest {
+
+ @Rule
+ public TemporaryFolder folder1 =
+ new TemporaryFolder(new File(System.getProperty("user.dir") + "/target"));
+ String folderPath;
+ private DefaultFileSystemManager vfs;
+
+ @Before
+ public void setup() throws Exception {
+ System.setProperty(ReloadingVFSClassLoader.VFS_CLASSPATH_MONITOR_INTERVAL, "1");
+ vfs = VFSManager.generateVfs();
+
+ folderPath = folder1.getRoot().toURI() + ".*";
+
+ FileUtils.copyURLToFile(this.getClass().getResource("/HelloWorld.jar"),
+ folder1.newFile("HelloWorld.jar"));
+ }
+
+ FileObject[] createFileSystems(FileObject[] fos) throws FileSystemException {
+ FileObject[] rfos = new FileObject[fos.length];
+ for (int i = 0; i < fos.length; i++) {
+ if (vfs.canCreateFileSystem(fos[i])) {
+ rfos[i] = vfs.createFileSystem(fos[i]);
+ } else {
+ rfos[i] = fos[i];
+ }
+ }
+
+ return rfos;
+ }
+
+ @Test
+ public void testConstructor() throws Exception {
+ FileObject testDir = vfs.resolveFile(folder1.getRoot().toURI().toString());
+ FileObject[] dirContents = testDir.getChildren();
+
+ ReloadingVFSClassLoader arvcl =
+ new ReloadingVFSClassLoader(ClassLoader.getSystemClassLoader()) {
+ @Override
+ protected String getClassPath() {
+ return folderPath;
+ }
+
+ @Override
+ protected DefaultFileSystemManager getFileSystem() {
+ return vfs;
+ }
+ };
+ arvcl.setVMInitializedForTests();
+ arvcl.setVFSForTests(vfs);
+
+ FileObject[] files = ((VFSClassLoaderWrapper) arvcl.getDelegateClassLoader()).getFileObjects();
+ assertArrayEquals(createFileSystems(dirContents), files);
+
+ arvcl.close();
+ }
+
+ @Test
+ public void testReloading() throws Exception {
+ FileObject testDir = vfs.resolveFile(folder1.getRoot().toURI().toString());
+ FileObject[] dirContents = testDir.getChildren();
+
+ ReloadingVFSClassLoader arvcl =
+ new ReloadingVFSClassLoader(ClassLoader.getSystemClassLoader()) {
+ @Override
+ protected String getClassPath() {
+ return folderPath;
+ }
+
+ @Override
+ protected long getMonitorInterval() {
+ return 500l;
+ }
+
+ @Override
+ protected DefaultFileSystemManager getFileSystem() {
+ return vfs;
+ }
+ };
+ arvcl.setVMInitializedForTests();
+ arvcl.setVFSForTests(vfs);
+
+ FileObject[] files = ((VFSClassLoaderWrapper) arvcl.getDelegateClassLoader()).getFileObjects();
+ assertArrayEquals(createFileSystems(dirContents), files);
+
+ // set retry settings sufficiently low that not everything is reloaded in the first round
+ arvcl.setMaxRetries(1);
+
+ Class<?> clazz1 = arvcl.loadClass("test.HelloWorld");
+ Object o1 = clazz1.getDeclaredConstructor().newInstance();
+ assertEquals("Hello World!", o1.toString());
+
+ // Check that the class is the same before the update
+ Class<?> clazz1_5 = arvcl.loadClass("test.HelloWorld");
+ assertEquals(clazz1, clazz1_5);
+
+ assertTrue(new File(folder1.getRoot(), "HelloWorld.jar").delete());
+
+ Thread.sleep(1000);
+
+ // Update the class
+ FileUtils.copyURLToFile(this.getClass().getResource("/HelloWorld.jar"),
+ folder1.newFile("HelloWorld2.jar"));
+
+ // Wait for the monitor to notice
+ Thread.sleep(1000);
+
+ Class<?> clazz2 = arvcl.loadClass("test.HelloWorld");
+ Object o2 = clazz2.getDeclaredConstructor().newInstance();
+ assertEquals("Hello World!", o2.toString());
+
+ // This is false because they are loaded by a different classloader
+ assertNotEquals(clazz1, clazz2);
+ assertNotEquals(o1, o2);
+
+ arvcl.close();
+ }
+
+ @Test
+ public void testReloadingWithLongerTimeout() throws Exception {
+ FileObject testDir = vfs.resolveFile(folder1.getRoot().toURI().toString());
+ FileObject[] dirContents = testDir.getChildren();
+
+ ReloadingVFSClassLoader arvcl =
+ new ReloadingVFSClassLoader(ClassLoader.getSystemClassLoader()) {
+ @Override
+ protected String getClassPath() {
+ return folderPath;
+ }
+
+ @Override
+ protected long getMonitorInterval() {
+ return 1000l;
+ }
+
+ @Override
+ protected DefaultFileSystemManager getFileSystem() {
+ return vfs;
+ }
+ };
+ arvcl.setVMInitializedForTests();
+ arvcl.setVFSForTests(vfs);
+
+ FileObject[] files = ((VFSClassLoaderWrapper) arvcl.getDelegateClassLoader()).getFileObjects();
+ assertArrayEquals(createFileSystems(dirContents), files);
+
+ // set retry settings sufficiently high such that reloading happens in the first rounds
+ arvcl.setMaxRetries(3);
+
+ Class<?> clazz1 = arvcl.loadClass("test.HelloWorld");
+ Object o1 = clazz1.getDeclaredConstructor().newInstance();
+ assertEquals("Hello World!", o1.toString());
+
+ // Check that the class is the same before the update
+ Class<?> clazz1_5 = arvcl.loadClass("test.HelloWorld");
+ assertEquals(clazz1, clazz1_5);
+
+ assertTrue(new File(folder1.getRoot(), "HelloWorld.jar").delete());
+
+ Thread.sleep(3000);
+
+ // Update the class
+ FileUtils.copyURLToFile(this.getClass().getResource("/HelloWorld.jar"),
+ folder1.newFile("HelloWorld2.jar"));
+
+ // Wait for the monitor to notice
+ Thread.sleep(3000);
+
+ Class<?> clazz2 = arvcl.loadClass("test.HelloWorld");
+ Object o2 = clazz2.getDeclaredConstructor().newInstance();
+ assertEquals("Hello World!", o2.toString());
+
+ // This is false because even though it's the same class, it's loaded from a different jar
+ // this is a change in behavior from previous versions of vfs2 where it would load the same
+ // class from different jars as though it was from the first jar
+ assertNotEquals(clazz1, clazz2);
+ assertNotSame(o1, o2);
+ assertEquals(clazz1.getName(), clazz2.getName());
+ assertEquals(o1.toString(), o2.toString());
+
+ arvcl.close();
+ }
+
+}
diff --git a/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/VfsClassLoaderTest.java b/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/VfsClassLoaderTest.java
new file mode 100644
index 0000000..4778872
--- /dev/null
+++ b/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/VfsClassLoaderTest.java
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.vfs;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.net.URL;
+
+import org.apache.commons.vfs2.FileChangeEvent;
+import org.apache.commons.vfs2.FileListener;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.impl.DefaultFileMonitor;
+import org.apache.commons.vfs2.impl.VFSClassLoader;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+public class VfsClassLoaderTest extends AccumuloDFSBase {
+
+ private static final Path TEST_DIR = new Path(getHdfsUri() + "/test-dir");
+
+ private FileSystem hdfs = null;
+ private VFSClassLoader cl = null;
+
+ @Before
+ public void setup() throws Exception {
+
+ this.hdfs = cluster.getFileSystem();
+ this.hdfs.mkdirs(TEST_DIR);
+
+ // Copy jar file to TEST_DIR
+ URL jarPath = this.getClass().getResource("/HelloWorld.jar");
+ Path src = new Path(jarPath.toURI().toString());
+ Path dst = new Path(TEST_DIR, src.getName());
+ this.hdfs.copyFromLocalFile(src, dst);
+
+ FileObject testDir = vfs.resolveFile(TEST_DIR.toUri().toString());
+ FileObject[] dirContents = testDir.getChildren();
+
+ // Point the VFSClassLoader to all of the objects in TEST_DIR
+ this.cl = new VFSClassLoader(dirContents, vfs);
+ }
+
+ @Test
+ public void testGetClass() throws Exception {
+ Class<?> helloWorldClass = this.cl.loadClass("test.HelloWorld");
+ Object o = helloWorldClass.getDeclaredConstructor().newInstance();
+ assertEquals("Hello World!", o.toString());
+ }
+
+ @Test
+ public void testFileMonitor() throws Exception {
+ MyFileMonitor listener = new MyFileMonitor();
+ DefaultFileMonitor monitor = new DefaultFileMonitor(listener);
+ monitor.setRecursive(true);
+ FileObject testDir = vfs.resolveFile(TEST_DIR.toUri().toString());
+ monitor.addFile(testDir);
+ monitor.start();
+
+ // Copy jar file to a new file name
+ URL jarPath = this.getClass().getResource("/HelloWorld.jar");
+ Path src = new Path(jarPath.toURI().toString());
+ Path dst = new Path(TEST_DIR, "HelloWorld2.jar");
+ this.hdfs.copyFromLocalFile(src, dst);
+
+ // VFS-487 significantly wait to avoid failure
+ Thread.sleep(7000);
+ assertTrue(listener.isFileCreated());
+
+ // Update the jar
+ jarPath = this.getClass().getResource("/HelloWorld.jar");
+ src = new Path(jarPath.toURI().toString());
+ dst = new Path(TEST_DIR, "HelloWorld2.jar");
+ this.hdfs.copyFromLocalFile(src, dst);
+
+ // VFS-487 significantly wait to avoid failure
+ Thread.sleep(7000);
+ assertTrue(listener.isFileChanged());
+
+ this.hdfs.delete(dst, false);
+ // VFS-487 significantly wait to avoid failure
+ Thread.sleep(7000);
+ assertTrue(listener.isFileDeleted());
+
+ monitor.stop();
+
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ this.hdfs.delete(TEST_DIR, true);
+ }
+
+ public static class MyFileMonitor implements FileListener {
+
+ private boolean fileChanged = false;
+ private boolean fileDeleted = false;
+ private boolean fileCreated = false;
+
+ @Override
+ public void fileCreated(FileChangeEvent event) throws Exception {
+ // System.out.println(event.getFile() + " created");
+ this.fileCreated = true;
+ }
+
+ @Override
+ public void fileDeleted(FileChangeEvent event) throws Exception {
+ // System.out.println(event.getFile() + " deleted");
+ this.fileDeleted = true;
+ }
+
+ @Override
+ public void fileChanged(FileChangeEvent event) throws Exception {
+ // System.out.println(event.getFile() + " changed");
+ this.fileChanged = true;
+ }
+
+ public boolean isFileChanged() {
+ return fileChanged;
+ }
+
+ public boolean isFileDeleted() {
+ return fileDeleted;
+ }
+
+ public boolean isFileCreated() {
+ return fileCreated;
+ }
+
+ }
+}
diff --git a/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/context/ReloadingVFSContextClassLoaderFactoryTest.java b/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/context/ReloadingVFSContextClassLoaderFactoryTest.java
new file mode 100644
index 0000000..4be5c46
--- /dev/null
+++ b/modules/vfs-class-loader/src/test/java/org/apache/accumulo/classloader/vfs/context/ReloadingVFSContextClassLoaderFactoryTest.java
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.vfs.context;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static java.nio.file.StandardOpenOption.WRITE;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.accumulo.classloader.vfs.context.ReloadingVFSContextClassLoaderFactory.Context;
+import org.apache.accumulo.classloader.vfs.context.ReloadingVFSContextClassLoaderFactory.ContextConfig;
+import org.apache.accumulo.classloader.vfs.context.ReloadingVFSContextClassLoaderFactory.Contexts;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import com.google.gson.Gson;
+
+public class ReloadingVFSContextClassLoaderFactoryTest {
+
+ @Rule
+ public TemporaryFolder temp = new TemporaryFolder();
+
+ private static final Contexts c = new Contexts();
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ ContextConfig cc1 = new ContextConfig();
+ cc1.setClassPath("file:///tmp/foo");
+ cc1.setPostDelegate(true);
+ cc1.setMonitorIntervalMs(30000);
+ Context c1 = new Context();
+ c1.setName("cx1");
+ c1.setConfig(cc1);
+
+ ContextConfig cc2 = new ContextConfig();
+ cc2.setClassPath("file:///tmp/bar");
+ cc2.setPostDelegate(false);
+ cc2.setMonitorIntervalMs(30000);
+ Context c2 = new Context();
+ c2.setName("cx2");
+ c2.setConfig(cc2);
+
+ List<Context> list = new ArrayList<>();
+ list.add(c1);
+ list.add(c2);
+ c.setContexts(list);
+ }
+
+ @Test
+ public void testDeSer() throws Exception {
+ Gson g = new Gson().newBuilder().setPrettyPrinting().create();
+ String contexts = g.toJson(c);
+ System.out.println(contexts);
+
+ Gson g2 = new Gson();
+ Contexts actual = g2.fromJson(contexts, Contexts.class);
+
+ assertEquals(c, actual);
+
+ }
+
+ @Test
+ public void testCreation() throws Exception {
+ File f = temp.newFile();
+ f.deleteOnExit();
+ Gson g = new Gson();
+ String contexts = g.toJson(c);
+ try (BufferedWriter writer = Files.newBufferedWriter(f.toPath(), UTF_8, WRITE)) {
+ writer.write(contexts);
+ }
+ ReloadingVFSContextClassLoaderFactory cl = new ReloadingVFSContextClassLoaderFactory() {
+ @Override
+ protected String getConfigFileLocation() {
+ return f.getAbsolutePath();
+ }
+ };
+ cl.initialize(null);
+ try {
+ cl.getClassLoader("c1");
+ fail("Expected illegal argument exception");
+ } catch (IllegalArgumentException e) {
+ // works
+ }
+ cl.getClassLoader("cx1");
+ cl.getClassLoader("cx2");
+
+ }
+
+}
diff --git a/modules/vfs-class-loader/src/test/java/test/HelloWorldTemplate b/modules/vfs-class-loader/src/test/java/test/HelloWorldTemplate
new file mode 100644
index 0000000..b1a6a73
--- /dev/null
+++ b/modules/vfs-class-loader/src/test/java/test/HelloWorldTemplate
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package test;
+
+public class HelloWorld {
+
+ @Override
+ public String toString() {
+ return "%%";
+ }
+}
diff --git a/modules/vfs-class-loader/src/test/java/test/Test.java b/modules/vfs-class-loader/src/test/java/test/Test.java
new file mode 100644
index 0000000..a93aded
--- /dev/null
+++ b/modules/vfs-class-loader/src/test/java/test/Test.java
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package test;
+
+public interface Test {
+
+ String hello();
+
+ int add();
+
+}
diff --git a/modules/vfs-class-loader/src/test/java/test/TestTemplate b/modules/vfs-class-loader/src/test/java/test/TestTemplate
new file mode 100644
index 0000000..ab5e217
--- /dev/null
+++ b/modules/vfs-class-loader/src/test/java/test/TestTemplate
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package test;
+
+public class TestObject implements Test {
+
+ int i = 0;
+
+ @Override
+ public String hello() {
+ return "Hello from testX";
+ }
+
+ @Override
+ public int add() {
+ i += 1;
+ return i;
+ }
+
+}
diff --git a/modules/vfs-class-loader/src/test/resources/log4j2-test.properties b/modules/vfs-class-loader/src/test/resources/log4j2-test.properties
new file mode 100644
index 0000000..6dcf0c5
--- /dev/null
+++ b/modules/vfs-class-loader/src/test/resources/log4j2-test.properties
@@ -0,0 +1,35 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+status = info
+dest = err
+name = AccumuloVFSClassLoaderTestLoggingProperties
+
+appender.console.type = Console
+appender.console.name = STDOUT
+appender.console.target = SYSTEM_OUT
+appender.console.layout.type = PatternLayout
+appender.console.layout.pattern = %d{ISO8601} [%-8c{2}] %-5p: %m%n
+
+logger.01.name = org.apache.accumulo.server.util.TabletIterator
+logger.01.level = error
+
+rootLogger.level = info
+rootLogger.appenderRef.console.ref = STDOUT
+
diff --git a/modules/vfs-class-loader/src/test/shell/makeHelloWorldJars.sh b/modules/vfs-class-loader/src/test/shell/makeHelloWorldJars.sh
new file mode 100755
index 0000000..04289ec
--- /dev/null
+++ b/modules/vfs-class-loader/src/test/shell/makeHelloWorldJars.sh
@@ -0,0 +1,33 @@
+#! /usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+if [ -z "$JAVA_HOME" ]; then
+ echo "JAVA_HOME is not set. Java is required to proceed"
+ exit 1
+fi
+mkdir -p target/generated-sources/HelloWorld/test
+sed "s/%%/Hello World\!/" < src/test/java/test/HelloWorldTemplate > target/generated-sources/HelloWorld/test/HelloWorld.java
+$JAVA_HOME/bin/javac target/generated-sources/HelloWorld/test/HelloWorld.java -d target/generated-sources/HelloWorld
+$JAVA_HOME/bin/jar -cf target/test-classes/HelloWorld.jar -C target/generated-sources/HelloWorld test/HelloWorld.class
+
+mkdir -p target/generated-sources/HalloWelt/test
+sed "s/%%/Hallo Welt/" < src/test/java/test/HelloWorldTemplate > target/generated-sources/HalloWelt/test/HelloWorld.java
+$JAVA_HOME/bin/javac target/generated-sources/HalloWelt/test/HelloWorld.java -d target/generated-sources/HalloWelt
+$JAVA_HOME/bin/jar -cf target/test-classes/HelloWorld2.jar -C target/generated-sources/HalloWelt test/HelloWorld.class
diff --git a/modules/vfs-class-loader/src/test/shell/makeTestJars.sh b/modules/vfs-class-loader/src/test/shell/makeTestJars.sh
new file mode 100755
index 0000000..fc86fb7
--- /dev/null
+++ b/modules/vfs-class-loader/src/test/shell/makeTestJars.sh
@@ -0,0 +1,32 @@
+#! /usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+if [ -z "$JAVA_HOME" ]; then
+ echo "JAVA_HOME is not set. Java is required to proceed"
+ exit 1
+fi
+
+for x in A B C
+do
+ mkdir -p target/generated-sources/$x/test target/test-classes/ClassLoaderTest$x
+ sed "s/testX/test$x/" < src/test/java/test/TestTemplate > target/generated-sources/$x/test/TestObject.java
+ $JAVA_HOME/bin/javac -cp target/test-classes target/generated-sources/$x/test/TestObject.java -d target/generated-sources/$x
+ $JAVA_HOME/bin/jar -cf target/test-classes/ClassLoaderTest$x/Test.jar -C target/generated-sources/$x test/TestObject.class
+done
diff --git a/pom.xml b/pom.xml
index 61e4e2a..21ca3e3 100644
--- a/pom.xml
+++ b/pom.xml
@@ -25,6 +25,7 @@
<groupId>org.apache.accumulo</groupId>
<artifactId>classloader-extras</artifactId>
<version>1.0.0-SNAPSHOT</version>
+ <packaging>pom</packaging>
<name>Classloader Extras</name>
<description>Classloader Extras provided by the Apache Accumulo project</description>
<inceptionYear>2020</inceptionYear>
@@ -66,6 +67,9 @@
<archive>https://lists.apache.org/list.html?notifications@accumulo.apache.org</archive>
</mailingList>
</mailingLists>
+ <modules>
+ <module>modules/vfs-class-loader</module>
+ </modules>
<scm>
<connection>scm:git:https://gitbox.apache.org/repos/asf/accumulo-classloaders.git</connection>
<developerConnection>scm:git:https://gitbox.apache.org/repos/asf/accumulo-classloaders.git</developerConnection>
@@ -94,14 +98,11 @@
</properties>
<dependencies>
<!-- spotbugs-annotations provides SuppressFBWarnings annotation -->
- <!--
<dependency>
<groupId>com.github.spotbugs</groupId>
<artifactId>spotbugs-annotations</artifactId>
<version>4.1.2</version>
- <optional>true</optional>
</dependency>
- -->
</dependencies>
<build>
<pluginManagement>
[accumulo-classloaders] 03/03: Make changes to get the initial code
buildable
Posted by ct...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ctubbsii pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-classloaders.git
commit 99f918713b3af3be28bb25af41a822c58e48b561
Author: Christopher Tubbs <ct...@apache.org>
AuthorDate: Fri Sep 25 23:21:20 2020 -0400
Make changes to get the initial code buildable
---
modules/classloaderfactory-substitute/.gitignore | 32 ++++++++++++
modules/classloaderfactory-substitute/pom.xml | 39 ++++++++++++++
.../core/spi/common/ClassLoaderFactory.java | 61 ++++++++++++++++++++++
modules/vfs-class-loader/WIP.txt | 17 ++++++
modules/vfs-class-loader/pom.xml | 32 +++++++++---
.../classloader/vfs/ReloadingVFSClassLoader.java | 4 +-
pom.xml | 1 +
7 files changed, 176 insertions(+), 10 deletions(-)
diff --git a/modules/classloaderfactory-substitute/.gitignore b/modules/classloaderfactory-substitute/.gitignore
new file mode 100644
index 0000000..3d5bdae
--- /dev/null
+++ b/modules/classloaderfactory-substitute/.gitignore
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Maven ignores
+/target/
+
+# IDE ignores
+/.settings/
+/.project
+/.classpath
+/.pydevproject
+/.idea
+/*.iml
+/*.ipr
+/*.iws
+/nbproject/
+/nbactions.xml
+/nb-configuration.xml
+.vscode/
+.factorypath
diff --git a/modules/classloaderfactory-substitute/pom.xml b/modules/classloaderfactory-substitute/pom.xml
new file mode 100644
index 0000000..d5475fc
--- /dev/null
+++ b/modules/classloaderfactory-substitute/pom.xml
@@ -0,0 +1,39 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.accumulo</groupId>
+ <artifactId>classloader-extras</artifactId>
+ <version>1.0.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+ <artifactId>classloaderfactory-substitute</artifactId>
+ <name>ClassloaderFactory substitute</name>
+ <description>This exists temporarily, until a version of Accumulo is published containing the required ClassloaderFactory interface</description>
+ <properties>
+ <eclipseFormatterStyle>../../contrib/Eclipse-Accumulo-Codestyle.xml</eclipseFormatterStyle>
+ <maven.deploy.skip>true</maven.deploy.skip>
+ <!-- temporarily skip analyzing dependencies until things are fixed, in order to build -->
+ <mdep.analyze.skip>true</mdep.analyze.skip>
+ </properties>
+</project>
diff --git a/modules/classloaderfactory-substitute/src/main/java/org/apache/accumulo/core/spi/common/ClassLoaderFactory.java b/modules/classloaderfactory-substitute/src/main/java/org/apache/accumulo/core/spi/common/ClassLoaderFactory.java
new file mode 100644
index 0000000..7fb1d59
--- /dev/null
+++ b/modules/classloaderfactory-substitute/src/main/java/org/apache/accumulo/core/spi/common/ClassLoaderFactory.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.spi.common;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.Map.Entry;
+
+/**
+ * The ClassLoaderFactory is defined by the property general.context.factory. The factory
+ * implementation is configured externally to Accumulo and will return a ClassLoader for a given
+ * contextName.
+ *
+ */
+public interface ClassLoaderFactory {
+
+ static class ClassLoaderFactoryConfiguration {
+
+ public Iterator<Entry<String,String>> get() {
+ return Collections.emptyIterator();
+ }
+ }
+
+ /**
+ * Initialize the ClassLoaderFactory. Implementations may need a reference to the configuration so
+ * that it can clean up contexts that are no longer being used.
+ *
+ * @param conf
+ * Accumulo configuration properties
+ * @throws Exception
+ * if error initializing ClassLoaderFactory
+ */
+ void initialize(ClassLoaderFactoryConfiguration conf) throws Exception;
+
+ /**
+ *
+ * @param contextName
+ * name of classloader context
+ * @return classloader configured for the context
+ * @throws IllegalArgumentException
+ * if contextName is not supported
+ */
+ ClassLoader getClassLoader(String contextName) throws IllegalArgumentException;
+
+}
diff --git a/modules/vfs-class-loader/WIP.txt b/modules/vfs-class-loader/WIP.txt
index 7f92d22..09edcb9 100644
--- a/modules/vfs-class-loader/WIP.txt
+++ b/modules/vfs-class-loader/WIP.txt
@@ -1,3 +1,20 @@
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+
1. Running Accumulo with new VFS ClassLoader as SystemClassLoader
a. Add accumulo/test/target/TestJar-Iterators.jar to HDFS directory
diff --git a/modules/vfs-class-loader/pom.xml b/modules/vfs-class-loader/pom.xml
index e727d2f..981c258 100644
--- a/modules/vfs-class-loader/pom.xml
+++ b/modules/vfs-class-loader/pom.xml
@@ -31,12 +31,27 @@
<name>VFS Reloading ClassLoader</name>
<properties>
<eclipseFormatterStyle>../../contrib/Eclipse-Accumulo-Codestyle.xml</eclipseFormatterStyle>
+ <!-- temporarily skip analyzing dependencies until things are fixed, in order to build -->
+ <mdep.analyze.skip>true</mdep.analyze.skip>
+ <!-- temporarily skip spotbugs until things are fixed, in order to build -->
+ <spotbugs.skip>true</spotbugs.skip>
</properties>
<dependencies>
<dependency>
+ <groupId>com.google.protobuf</groupId>
+ <artifactId>protobuf-java</artifactId>
+ <version>3.7.1</version>
+ </dependency>
+ <dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-vfs2</artifactId>
<version>2.6.0</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs-client</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
@@ -63,15 +78,9 @@
</dependency>
<dependency>
<groupId>org.apache.accumulo</groupId>
- <artifactId>accumulo-core</artifactId>
- <version>2.1.0-SNAPSHOT</version>
+ <artifactId>classloaderfactory-substitute</artifactId>
+ <version>${project.version}</version>
<scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.accumulo</groupId>
- <artifactId>accumulo-start</artifactId>
- </exclusion>
- </exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
@@ -86,6 +95,12 @@
<scope>provided</scope>
</dependency>
<dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>28.2-jre</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
@@ -170,6 +185,7 @@
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
+ <version>1.6.0</version>
<executions>
<execution>
<id>Build Test jars</id>
diff --git a/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/ReloadingVFSClassLoader.java b/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/ReloadingVFSClassLoader.java
index 552076a..755f0d8 100644
--- a/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/ReloadingVFSClassLoader.java
+++ b/modules/vfs-class-loader/src/main/java/org/apache/accumulo/classloader/vfs/ReloadingVFSClassLoader.java
@@ -408,7 +408,7 @@ public class ReloadingVFSClassLoader extends ClassLoader implements Closeable, F
/**
* Remove the file from the monitor
- *
+ *
* @param file
* to remove
* @throws RuntimeException
@@ -500,7 +500,7 @@ public class ReloadingVFSClassLoader extends ClassLoader implements Closeable, F
/**
* Return a reference to the delegate classloader, create a new one if necessary
- *
+ *
* @return reference to delegate classloader
*/
synchronized ClassLoader getDelegateClassLoader() {
diff --git a/pom.xml b/pom.xml
index 21ca3e3..f91a08b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -68,6 +68,7 @@
</mailingList>
</mailingLists>
<modules>
+ <module>modules/classloaderfactory-substitute</module>
<module>modules/vfs-class-loader</module>
</modules>
<scm>
[accumulo-classloaders] 02/03: Capturing testing document
Posted by ct...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ctubbsii pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-classloaders.git
commit 5566e52143592cc5cfe06ce1b92161cf6f443f3c
Author: Dave Marion <dl...@apache.org>
AuthorDate: Fri Sep 25 21:14:12 2020 +0000
Capturing testing document
---
modules/vfs-class-loader/WIP.txt | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/modules/vfs-class-loader/WIP.txt b/modules/vfs-class-loader/WIP.txt
new file mode 100644
index 0000000..7f92d22
--- /dev/null
+++ b/modules/vfs-class-loader/WIP.txt
@@ -0,0 +1,15 @@
+1. Running Accumulo with new VFS ClassLoader as SystemClassLoader
+
+ a. Add accumulo/test/target/TestJar-Iterators.jar to HDFS directory
+ hadoop fs -mkdir /iterators
+ hadoop fs -put TestJar-Iterators.jar /iterators/.
+ b. Add vfs-reloading-classloader-0.0.1-SNAPSHOT.jar to CLASSPATH
+ c. Add "-Djava.system.class.loader=org.apache.accumulo.classloader.vfs.ReloadingVFSClassLoader" to JAVA_OPTS
+ d. Add "-Dvfs.class.loader.classpath=hdfs://localhost:9000/iterators/TestJar-Iterators.jar" to JAVA_OPTS
+ e. Add "-Dvfs.class.loader.debug=true" to JAVA_OPTS
+ f. Create table , insert some test data
+ createtable test
+ insert a b c this_is_a_test
+ scan
+ setiter -class org.apache.accumulo.test.functional.ValueReversingIterator -scan -t test -name reverse -p 100
+