[DISCUSS] Classloader change proposals

[Christopher <ct...@apache.org>]

Hi Accumulo Devs,

Lately, Dave Marion (Apache ID: dlmarion) has been working on
prototyping some new class loader concepts for Accumulo that he and I
have discussed, and I wanted to pitch the idea here for consideration
for the project.

# Background:

Accumulo currently has two classloaders that are instantiated at
startup, and which can be used to bootstrap Accumulo dependencies (at
least, those not needed for the classloader code itself). This allows
us to use the `general.classpaths`[1] and
`general.dynamic.classpaths`[2] properties, as well as the per-context
classloaders (`general.vfs.*`[3] and `table.classpath.context`[4]) for
things like iterator class isolation. Since 2.0.0, we have deprecated
`general.classpaths` and `general.dynamic.classpaths`, the former
supplanted by the better use of the `CLASSPATH` environment variable
(along with much improved scripts in 2.0.0), and the latter being
replaceable by a user-provided class loader using the built-in Java
property, `java.system.class.loader`[5], at their discretion.

# The Problem:

The main problem with the current code is: complexity. Accumulo is
already complex enough without needing to be in the business of
developing and supporting complex custom class loading features,
especially when users have viable alternatives that can be better
supported by independent, dedicated projects. Furthermore, these
custom class loaders also have a dependency on commons-vfs2, which has
been the source of numerous problems and bugs that we have needed to
deal with, and which affect Accumulo, even though they are not
necessarily bugs in Accumulo itself. This also brings in a lot of
optional dependencies that aren't needed by users who don't rely on
these features.

# The Requirements:

In spite of these problems, I believe we still want to enable the use
cases that our classloaders are currently enabling.

Specifically,
1) the ability to have separate contexts for iterator class isolation
(A/B testing of iterators, updating iterators in a live system, etc.),
and
2) the ability for users to bootstrap their class path from some other
distributed storage than local disk.

# The Proposal:

1. Create a new reloading vfs class loader, with similar functionality
as our current two-classloaders that do the reloading and provide vfs
features, that can be easily used as a system class loader, if the
user chooses to, and deprecate (for removal in 3.0) the built-in
implementations. This class loader could not only be used with
Accumulo, but it could also be used by any other project that chooses
to use it, because it will not have much, if any, dependencies beyond
commons-vfs2, and will certainly not depend on Accumulo. Creating this
separate class loader provides us a path forward to simplify Accumulo
by removing these features from Accumulo directly (the properties are
already deprecated), and enabling it to be maintained independently.
2. Create a new class loader factory property in Accumulo, with
corresponding SPI interface, for users to provide their own
implementation of a class loader factory, that can map a per-table
"context" to a ClassLoader of the implementation's choosing.

The result of doing these two things will allow us to more flexibly
support user class loading needs, without being directly responsible
for class loading implementations inside Accumulo's core code. All the
same functionality that is available today will continue to exist, but
will be configured differently. The resulting code in Accumulo will be
dramatically simpler, as we would no longer have any complex class
loading implementations in our code base, and we would no longer have
any direct dependency on commons-vfs2, which has been problematic.
Independent implementations may use commons-vfs2, or something else,
but will be more easily testable and maintainable as independent
projects that are pluggable in Accumulo.

Dave has already been working on prototyping these proposed changes,
and it is looking very feasible.

We are now ready to:
1. get feedback on the overall proposal, and
2. decide on where to maintain the separate class loader.

For where to maintain, the options seem to be: A) try to donate to
commons-vfs2 OR B) maintain as a new repository,
accumulo-vfs-classloader.

Note that we have not yet proposed the idea of a user-facing,
configurable, reloading vfs classloader to the commons-vfs2
developers. We wanted to get our own community's feedback on this
first.

Please discuss.

Thanks,
Christopher (in collaboration with Dave)

[1]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
[2]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
[3]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
[4]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
[5]: https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93

Re: [DISCUSS] Classloader change proposals

[Christopher <ct...@apache.org>]

Hi Ivan,

Thanks for the feedback. I've attempted to respond inline with some
points I may not have explained well in the original post.

On Thu, Sep 17, 2020 at 3:18 PM ivan bella <iv...@ivan.bella.name> wrote:
>
>      I like the idea of simplifying the dependencies that Accumulo has.  As long as we can still have the ability to monitor a file system (e.g. hdfs) and reload the jars from a specified classpath when it has changed. This is needed for loading jars for our deployed application and managing patches, but then a second class loader would load the accumulo jars from disk. If the reloading classloader is the system classloader, will it still be able to fall back to the java classloader that uses CLASSPATH? Also I missed where the reloading vfs classloader would actually live?
>

Using a system class loader instead of the baked-in, there should be
no change in the use cases supported... only how it is configured.
Setting it as a system class loader would make it "sit on top of" the
default system class loader that Java uses, and it could be
implemented to either check the parent first, or attempt loading
itself before falling back to the parent.

As for where it would actually live... that was one of the things I
specifically named in my original post as requesting feedback on. The
options we have considered are: trying to get it pushed into
commons-vfs, or just creating a new repository / subproject under
Accumulo (just like our accumulo-maven-plugin or accumulo-proxy
repos).

>      I believe we will still end up having a stack of classloaders as we do today so I am not sure this is really simpler. It appears we are simply replacing the Vfs classloader with a reloading classloader that can then delegate to the vfs classloader. Perhaps I am missing something here.

The code in Accumulo would be simpler. We wouldn't need to maintain
the 5 separate class loaders in Accumulo today:
ls start/src/main/**/*lass*oader*.java
And, we wouldn't need to immediately launch a separate thread to run
our main methods under a different class loader, or remember to do
anything special when loading pluggable classes set in the config
(unless they are loaded in a context). The context class loader would
be simpler, because we'd maintain only a simple reference
implementation for that.

In the separate repo, the new reloading vfs class loader could be
simpler, too... as it could focus on a narrow set of easy-to-configure
features, rather than trying to intermingle or handle special cases to
bootstrap Accumulo code... all that bootstrapping would be handled by
Java's own feature on its own.

The dependency tree (and potential dependency conflicts) would be
simpler as well, if the feature is not used (since it would be
optional). Of course, this wouldn't apply to those who *do* use the
feature, but it means the blame for problems using the feature should
be easier to identify (and possibly correct).

Accumulo build testing would also be simpler, as we'd have fewer
complications setting up our test cases (and a few fewer tests to
run).

>
>
> > On September 14, 2020 5:30 PM Christopher <ct...@apache.org> wrote:
> >
> >
> > Hi Accumulo Devs,
> >
> > Lately, Dave Marion (Apache ID: dlmarion) has been working on
> > prototyping some new class loader concepts for Accumulo that he and I
> > have discussed, and I wanted to pitch the idea here for consideration
> > for the project.
> >
> > # Background:
> >
> > Accumulo currently has two classloaders that are instantiated at
> > startup, and which can be used to bootstrap Accumulo dependencies (at
> > least, those not needed for the classloader code itself). This allows
> > us to use the `general.classpaths`[1] and
> > `general.dynamic.classpaths`[2] properties, as well as the per-context
> > classloaders (`general.vfs.*`[3] and `table.classpath.context`[4]) for
> > things like iterator class isolation. Since 2.0.0, we have deprecated
> > `general.classpaths` and `general.dynamic.classpaths`, the former
> > supplanted by the better use of the `CLASSPATH` environment variable
> > (along with much improved scripts in 2.0.0), and the latter being
> > replaceable by a user-provided class loader using the built-in Java
> > property, `java.system.class.loader`[5], at their discretion.
> >
> > # The Problem:
> >
> > The main problem with the current code is: complexity. Accumulo is
> > already complex enough without needing to be in the business of
> > developing and supporting complex custom class loading features,
> > especially when users have viable alternatives that can be better
> > supported by independent, dedicated projects. Furthermore, these
> > custom class loaders also have a dependency on commons-vfs2, which has
> > been the source of numerous problems and bugs that we have needed to
> > deal with, and which affect Accumulo, even though they are not
> > necessarily bugs in Accumulo itself. This also brings in a lot of
> > optional dependencies that aren't needed by users who don't rely on
> > these features.
> >
> > # The Requirements:
> >
> > In spite of these problems, I believe we still want to enable the use
> > cases that our classloaders are currently enabling.
> >
> > Specifically,
> > 1) the ability to have separate contexts for iterator class isolation
> > (A/B testing of iterators, updating iterators in a live system, etc.),
> > and
> > 2) the ability for users to bootstrap their class path from some other
> > distributed storage than local disk.
> >
> > # The Proposal:
> >
> > 1. Create a new reloading vfs class loader, with similar functionality
> > as our current two-classloaders that do the reloading and provide vfs
> > features, that can be easily used as a system class loader, if the
> > user chooses to, and deprecate (for removal in 3.0) the built-in
> > implementations. This class loader could not only be used with
> > Accumulo, but it could also be used by any other project that chooses
> > to use it, because it will not have much, if any, dependencies beyond
> > commons-vfs2, and will certainly not depend on Accumulo. Creating this
> > separate class loader provides us a path forward to simplify Accumulo
> > by removing these features from Accumulo directly (the properties are
> > already deprecated), and enabling it to be maintained independently.
> > 2. Create a new class loader factory property in Accumulo, with
> > corresponding SPI interface, for users to provide their own
> > implementation of a class loader factory, that can map a per-table
> > "context" to a ClassLoader of the implementation's choosing.
> >
> > The result of doing these two things will allow us to more flexibly
> > support user class loading needs, without being directly responsible
> > for class loading implementations inside Accumulo's core code. All the
> > same functionality that is available today will continue to exist, but
> > will be configured differently. The resulting code in Accumulo will be
> > dramatically simpler, as we would no longer have any complex class
> > loading implementations in our code base, and we would no longer have
> > any direct dependency on commons-vfs2, which has been problematic.
> > Independent implementations may use commons-vfs2, or something else,
> > but will be more easily testable and maintainable as independent
> > projects that are pluggable in Accumulo.
> >
> > Dave has already been working on prototyping these proposed changes,
> > and it is looking very feasible.
> >
> > We are now ready to:
> > 1. get feedback on the overall proposal, and
> > 2. decide on where to maintain the separate class loader.
> >
> > For where to maintain, the options seem to be: A) try to donate to
> > commons-vfs2 OR B) maintain as a new repository,
> > accumulo-vfs-classloader.
> >
> > Note that we have not yet proposed the idea of a user-facing,
> > configurable, reloading vfs classloader to the commons-vfs2
> > developers. We wanted to get our own community's feedback on this
> > first.
> >
> > Please discuss.
> >
> > Thanks,
> > Christopher (in collaboration with Dave)
> >
> > [1]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
> > [2]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
> > [3]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
> > [4]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
> > [5]: https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93

Re: [DISCUSS] Classloader change proposals

[Dave Marion <dm...@gmail.com>]

  The new classloader will maintain the same functionality that you have
with the current classloader. At the point where the current classloader is
removed from the accumulo core code (version 3.0) Accumulo by default will
use the standard JVM classloading mechanism. If someone wants to use the
new classloader (location TBD) they would set the
`java.system.class.loader` property to the fully qualified class name of
the new classloader. When set, the JVM instantiates the specified
classloader as a child of the app classloader (the one that loads from
java.class.path) and then sets the new classloader as the system
classloader. See [1] for more information.

  The context classloading mechanism functionality is also maintained, but
the configuration has been modified a bit. There has been discussion about
the location for the new classloader with the fallback being that it's an
Accumulo subproject.

  I should be able to put up a PR soon with the proposed changes.

[1]
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/ClassLoader.html#getSystemClassLoader()

On Thu, Sep 17, 2020 at 3:18 PM ivan bella <iv...@ivan.bella.name> wrote:

>      I like the idea of simplifying the dependencies that Accumulo has.
> As long as we can still have the ability to monitor a file system (e.g.
> hdfs) and reload the jars from a specified classpath when it has changed.
> This is needed for loading jars for our deployed application and managing
> patches, but then a second class loader would load the accumulo jars from
> disk. If the reloading classloader is the system classloader, will it still
> be able to fall back to the java classloader that uses CLASSPATH? Also I
> missed where the reloading vfs classloader would actually live?
>
>      I believe we will still end up having a stack of classloaders as we
> do today so I am not sure this is really simpler. It appears we are simply
> replacing the Vfs classloader with a reloading classloader that can then
> delegate to the vfs classloader. Perhaps I am missing something here.
>
>
> > On September 14, 2020 5:30 PM Christopher <ct...@apache.org> wrote:
> >
> >
> > Hi Accumulo Devs,
> >
> > Lately, Dave Marion (Apache ID: dlmarion) has been working on
> > prototyping some new class loader concepts for Accumulo that he and I
> > have discussed, and I wanted to pitch the idea here for consideration
> > for the project.
> >
> > # Background:
> >
> > Accumulo currently has two classloaders that are instantiated at
> > startup, and which can be used to bootstrap Accumulo dependencies (at
> > least, those not needed for the classloader code itself). This allows
> > us to use the `general.classpaths`[1] and
> > `general.dynamic.classpaths`[2] properties, as well as the per-context
> > classloaders (`general.vfs.*`[3] and `table.classpath.context`[4]) for
> > things like iterator class isolation. Since 2.0.0, we have deprecated
> > `general.classpaths` and `general.dynamic.classpaths`, the former
> > supplanted by the better use of the `CLASSPATH` environment variable
> > (along with much improved scripts in 2.0.0), and the latter being
> > replaceable by a user-provided class loader using the built-in Java
> > property, `java.system.class.loader`[5], at their discretion.
> >
> > # The Problem:
> >
> > The main problem with the current code is: complexity. Accumulo is
> > already complex enough without needing to be in the business of
> > developing and supporting complex custom class loading features,
> > especially when users have viable alternatives that can be better
> > supported by independent, dedicated projects. Furthermore, these
> > custom class loaders also have a dependency on commons-vfs2, which has
> > been the source of numerous problems and bugs that we have needed to
> > deal with, and which affect Accumulo, even though they are not
> > necessarily bugs in Accumulo itself. This also brings in a lot of
> > optional dependencies that aren't needed by users who don't rely on
> > these features.
> >
> > # The Requirements:
> >
> > In spite of these problems, I believe we still want to enable the use
> > cases that our classloaders are currently enabling.
> >
> > Specifically,
> > 1) the ability to have separate contexts for iterator class isolation
> > (A/B testing of iterators, updating iterators in a live system, etc.),
> > and
> > 2) the ability for users to bootstrap their class path from some other
> > distributed storage than local disk.
> >
> > # The Proposal:
> >
> > 1. Create a new reloading vfs class loader, with similar functionality
> > as our current two-classloaders that do the reloading and provide vfs
> > features, that can be easily used as a system class loader, if the
> > user chooses to, and deprecate (for removal in 3.0) the built-in
> > implementations. This class loader could not only be used with
> > Accumulo, but it could also be used by any other project that chooses
> > to use it, because it will not have much, if any, dependencies beyond
> > commons-vfs2, and will certainly not depend on Accumulo. Creating this
> > separate class loader provides us a path forward to simplify Accumulo
> > by removing these features from Accumulo directly (the properties are
> > already deprecated), and enabling it to be maintained independently.
> > 2. Create a new class loader factory property in Accumulo, with
> > corresponding SPI interface, for users to provide their own
> > implementation of a class loader factory, that can map a per-table
> > "context" to a ClassLoader of the implementation's choosing.
> >
> > The result of doing these two things will allow us to more flexibly
> > support user class loading needs, without being directly responsible
> > for class loading implementations inside Accumulo's core code. All the
> > same functionality that is available today will continue to exist, but
> > will be configured differently. The resulting code in Accumulo will be
> > dramatically simpler, as we would no longer have any complex class
> > loading implementations in our code base, and we would no longer have
> > any direct dependency on commons-vfs2, which has been problematic.
> > Independent implementations may use commons-vfs2, or something else,
> > but will be more easily testable and maintainable as independent
> > projects that are pluggable in Accumulo.
> >
> > Dave has already been working on prototyping these proposed changes,
> > and it is looking very feasible.
> >
> > We are now ready to:
> > 1. get feedback on the overall proposal, and
> > 2. decide on where to maintain the separate class loader.
> >
> > For where to maintain, the options seem to be: A) try to donate to
> > commons-vfs2 OR B) maintain as a new repository,
> > accumulo-vfs-classloader.
> >
> > Note that we have not yet proposed the idea of a user-facing,
> > configurable, reloading vfs classloader to the commons-vfs2
> > developers. We wanted to get our own community's feedback on this
> > first.
> >
> > Please discuss.
> >
> > Thanks,
> > Christopher (in collaboration with Dave)
> >
> > [1]:
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
> > [2]:
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
> > [3]:
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
> > [4]:
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
> > [5]:
> https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93
>

Re: [DISCUSS] Classloader change proposals

[ivan bella <iv...@ivan.bella.name>]

     I like the idea of simplifying the dependencies that Accumulo has.  As long as we can still have the ability to monitor a file system (e.g. hdfs) and reload the jars from a specified classpath when it has changed. This is needed for loading jars for our deployed application and managing patches, but then a second class loader would load the accumulo jars from disk. If the reloading classloader is the system classloader, will it still be able to fall back to the java classloader that uses CLASSPATH? Also I missed where the reloading vfs classloader would actually live?

     I believe we will still end up having a stack of classloaders as we do today so I am not sure this is really simpler. It appears we are simply replacing the Vfs classloader with a reloading classloader that can then delegate to the vfs classloader. Perhaps I am missing something here.


> On September 14, 2020 5:30 PM Christopher <ct...@apache.org> wrote:
> 
> 
> Hi Accumulo Devs,
> 
> Lately, Dave Marion (Apache ID: dlmarion) has been working on
> prototyping some new class loader concepts for Accumulo that he and I
> have discussed, and I wanted to pitch the idea here for consideration
> for the project.
> 
> # Background:
> 
> Accumulo currently has two classloaders that are instantiated at
> startup, and which can be used to bootstrap Accumulo dependencies (at
> least, those not needed for the classloader code itself). This allows
> us to use the `general.classpaths`[1] and
> `general.dynamic.classpaths`[2] properties, as well as the per-context
> classloaders (`general.vfs.*`[3] and `table.classpath.context`[4]) for
> things like iterator class isolation. Since 2.0.0, we have deprecated
> `general.classpaths` and `general.dynamic.classpaths`, the former
> supplanted by the better use of the `CLASSPATH` environment variable
> (along with much improved scripts in 2.0.0), and the latter being
> replaceable by a user-provided class loader using the built-in Java
> property, `java.system.class.loader`[5], at their discretion.
> 
> # The Problem:
> 
> The main problem with the current code is: complexity. Accumulo is
> already complex enough without needing to be in the business of
> developing and supporting complex custom class loading features,
> especially when users have viable alternatives that can be better
> supported by independent, dedicated projects. Furthermore, these
> custom class loaders also have a dependency on commons-vfs2, which has
> been the source of numerous problems and bugs that we have needed to
> deal with, and which affect Accumulo, even though they are not
> necessarily bugs in Accumulo itself. This also brings in a lot of
> optional dependencies that aren't needed by users who don't rely on
> these features.
> 
> # The Requirements:
> 
> In spite of these problems, I believe we still want to enable the use
> cases that our classloaders are currently enabling.
> 
> Specifically,
> 1) the ability to have separate contexts for iterator class isolation
> (A/B testing of iterators, updating iterators in a live system, etc.),
> and
> 2) the ability for users to bootstrap their class path from some other
> distributed storage than local disk.
> 
> # The Proposal:
> 
> 1. Create a new reloading vfs class loader, with similar functionality
> as our current two-classloaders that do the reloading and provide vfs
> features, that can be easily used as a system class loader, if the
> user chooses to, and deprecate (for removal in 3.0) the built-in
> implementations. This class loader could not only be used with
> Accumulo, but it could also be used by any other project that chooses
> to use it, because it will not have much, if any, dependencies beyond
> commons-vfs2, and will certainly not depend on Accumulo. Creating this
> separate class loader provides us a path forward to simplify Accumulo
> by removing these features from Accumulo directly (the properties are
> already deprecated), and enabling it to be maintained independently.
> 2. Create a new class loader factory property in Accumulo, with
> corresponding SPI interface, for users to provide their own
> implementation of a class loader factory, that can map a per-table
> "context" to a ClassLoader of the implementation's choosing.
> 
> The result of doing these two things will allow us to more flexibly
> support user class loading needs, without being directly responsible
> for class loading implementations inside Accumulo's core code. All the
> same functionality that is available today will continue to exist, but
> will be configured differently. The resulting code in Accumulo will be
> dramatically simpler, as we would no longer have any complex class
> loading implementations in our code base, and we would no longer have
> any direct dependency on commons-vfs2, which has been problematic.
> Independent implementations may use commons-vfs2, or something else,
> but will be more easily testable and maintainable as independent
> projects that are pluggable in Accumulo.
> 
> Dave has already been working on prototyping these proposed changes,
> and it is looking very feasible.
> 
> We are now ready to:
> 1. get feedback on the overall proposal, and
> 2. decide on where to maintain the separate class loader.
> 
> For where to maintain, the options seem to be: A) try to donate to
> commons-vfs2 OR B) maintain as a new repository,
> accumulo-vfs-classloader.
> 
> Note that we have not yet proposed the idea of a user-facing,
> configurable, reloading vfs classloader to the commons-vfs2
> developers. We wanted to get our own community's feedback on this
> first.
> 
> Please discuss.
> 
> Thanks,
> Christopher (in collaboration with Dave)
> 
> [1]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
> [2]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
> [3]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
> [4]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
> [5]: https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93

Re: [DISCUSS] Classloader change proposals

[Christopher <ct...@apache.org>]

I cannot think of a good use case where somebody would want dynamic
class reloading from jars in the startup context. That seems risky and
prone to bugs. Other than a few user-pluggable components, such as
balancer, volume chooser, etc., most of the startup context contains
stuff that isn't safe to reload (ZK jars, Hadoop jars, and all their
dependencies, for example). The biggest use case for reloading seems
to be with the per-table contexts, and that doesn't require the system
class loader to support reloading.

We *could* make the startup context pluggable with its own
classloader, rather than rely on the built-in Java system class loader
mechanism, and that startup context classloader could be one that
supports reloading. However, this would also have its own problems,
like where would it get its config from when we haven't yet
bootstrapped enough of the application to set all the Accumulo code
for config handling, logging, etc. yet.

Given the available options, I'm inclined to just drop the reloading
behavior from the separate class loader intended for use as the system
class loader (but still leave reloading as a feature of the per-table
context classloading). If somebody really needs the reloading behavior
in the startup context, they can implement it later, as either a PR,
or as a completely separate launcher/wrapper or system class loader,
or people can run Accumulo in an application server. All of these
options seem far outside the scope of "Accumulo" work, though. I think
it'd be enough if we just made the basic non-reloading system class
loader available as an option (along with the reloading per-table
class loading).

On Mon, Oct 5, 2020 at 2:44 PM Dave Marion <dm...@gmail.com> wrote:
>
> I have tested the new classloader as the `java.system.class.loader`, and it
> works allowing me to load classes from HDFS. The reloading feature works
> too, except that the ClassLoader.loadClass() logic short circuits the
> loading of the new classes with the same name by calling findLoadedClass
> (which returns a cached Class object). I don't believe that we will be able
> to retain the reloading feature that we currently have with
> `general.dynamic.classpaths` (lib/ext) in this new classloader
> implementation. I have not tested it yet, but I believe that the table
> context class loader reloading mechanism will work.
>
> I know that the table context classloader feature is being used. Does
> anyone know of an instance where general.dynamic.classpath is still in use
> and losing the reloading feature would be unacceptable?
>
> On Thu, Sep 24, 2020 at 10:43 AM Christopher <ct...@apache.org> wrote:
>
> > The INFRA ticket is now closed, and the default branch has been updated.
> > I also prepopulated the repo with some issue templates and GitHub
> > Actions build configurations, as well as LICENSE/NOTICE and a pom.xml
> > file to start it off.
> >
> > On Wed, Sep 23, 2020 at 10:31 AM Christopher <ct...@apache.org> wrote:
> > >
> > > I created the repo, but waiting on
> > > https://issues.apache.org/jira/browse/INFRA-20884 to fix the GitBox
> > > default branch so that the GitHub issues can be enabled.
> > >
> > > It should be okay to add the initial code to it, though.
> > >
> > > On Tue, Sep 22, 2020 at 1:40 PM Dave Marion <dm...@gmail.com> wrote:
> > > >
> > > > Sounds good.
> > > >
> > > > On Tue, Sep 22, 2020, 1:37 PM Christopher <ct...@apache.org> wrote:
> > > >
> > > > > Based on the conversation and direction of this, I think it probably
> > > > > makes the most sense to just create a new repository under Accumulo
> > > > > for it to live.
> > > > >
> > > > > How about 'accumulo-classloaders' for the repo name? If this is okay,
> > > > > I can create it later today or tomorrow.
> > > > > This is just a repo name. This is just a place to collaborate on
> > > > > classloader code without constraining the scope of the repo's
> > contents
> > > > > too much. The actual package names and artifactId can be different,
> > if
> > > > > we want.
> > > > >
> > > > > On Tue, Sep 22, 2020 at 9:44 AM Dave Marion <dm...@gmail.com>
> > wrote:
> > > > > >
> > > > > > [1] contains the initial set of changes to the Accumulo code base
> > that
> > > > > > defines the new context class loader configuration and deprecates
> > the
> > > > > > existing VFS ClassLoader objects. [2] contains the new
> > > > > > ReloadingVFSClassLoader that can be used as the system classloader
> > and a
> > > > > > ClassLoaderFactory implementation for configuring contexts for
> > tables and
> > > > > > scans. Both build successfully and I plan on doing some local
> > testing
> > > > > next.
> > > > > > Feedback on the design and the code is welcome.
> > > > > >
> > > > > > [1] https://github.com/dlmarion/accumulo/pull/2
> > > > > > [2] https://github.com/dlmarion/vfs-reloading-classloader
> > > > > >
> > > > > > On Fri, Sep 18, 2020 at 12:19 PM Dave Marion <dm...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > >  I tend to agree with Marc on this. If we need to push out a fix
> > for
> > > > > the
> > > > > > > new classloader, then we can do it as needed and not have to
> > rely on
> > > > > > > another group of people to come to consensus on a release. Of
> > course,
> > > > > we
> > > > > > > could maintain a fork of it in that case, but then what's the
> > point? It
> > > > > > > appears that VFS does have some recent activity [1], but mostly
> > by one
> > > > > > > person. I'm thinking that we should create a subproject for it
> > and
> > > > > notify
> > > > > > > the commons-vfs project of its existence. If they want to copy
> > it and
> > > > > > > include it in their project, then they can do that.
> > > > > > >
> > > > > > > [1]
> > https://gitbox.apache.org/repos/asf?p=commons-vfs.git;a=shortlog
> > > > > > >
> > > > > > > On Wed, Sep 16, 2020 at 4:48 PM Christopher <ctubbsii@apache.org
> > >
> > > > > wrote:
> > > > > > >
> > > > > > >> Only Marc joined, and we didn't talk about anything that isn't
> > > > > already in
> > > > > > >> this proposal, except he did mention how difficult it might be
> > to try
> > > > > to
> > > > > > >> maintain the class loader in commons-vfs2, rather than as our
> > own
> > > > > small
> > > > > > >> subproject, which is relatively easy.
> > > > > > >>
> > > > > > >> On Wed, Sep 16, 2020, 16:11 Dave Marion <dm...@gmail.com>
> > wrote:
> > > > > > >>
> > > > > > >> > Did anyone join the call? Any notes?
> > > > > > >> >
> > > > > > >> > On Wed, Sep 16, 2020 at 12:44 PM Christopher <
> > ctubbsii@apache.org>
> > > > > > >> wrote:
> > > > > > >> >
> > > > > > >> > > I just want to remind everybody that I'm available in Slack
> > now to
> > > > > > >> > > discuss this in the ongoing video call I created in the
> > #accumulo
> > > > > > >> > > room, if you want to join.
> > > > > > >> > >
> > > > > > >> > > On Mon, Sep 14, 2020 at 10:41 PM Christopher <
> > ctubbsii@apache.org
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> > > >
> > > > > > >> > > > Also, if anybody is interested in a live video
> > conversation to
> > > > > > >> discuss
> > > > > > >> > > > this interactively, I intend to be on Slack on Wednesday
> > > > > afternoon
> > > > > > >> > > > (EDT) starting around noon.
> > > > > > >> > > >
> > > > > > >> > > > On Mon, Sep 14, 2020 at 5:30 PM Christopher <
> > > > > ctubbsii@apache.org>
> > > > > > >> > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > Hi Accumulo Devs,
> > > > > > >> > > > >
> > > > > > >> > > > > Lately, Dave Marion (Apache ID: dlmarion) has been
> > working on
> > > > > > >> > > > > prototyping some new class loader concepts for Accumulo
> > that
> > > > > he
> > > > > > >> and I
> > > > > > >> > > > > have discussed, and I wanted to pitch the idea here for
> > > > > > >> consideration
> > > > > > >> > > > > for the project.
> > > > > > >> > > > >
> > > > > > >> > > > > # Background:
> > > > > > >> > > > >
> > > > > > >> > > > > Accumulo currently has two classloaders that are
> > instantiated
> > > > > at
> > > > > > >> > > > > startup, and which can be used to bootstrap Accumulo
> > > > > dependencies
> > > > > > >> (at
> > > > > > >> > > > > least, those not needed for the classloader code
> > itself). This
> > > > > > >> allows
> > > > > > >> > > > > us to use the `general.classpaths`[1] and
> > > > > > >> > > > > `general.dynamic.classpaths`[2] properties, as well as
> > the
> > > > > > >> > per-context
> > > > > > >> > > > > classloaders (`general.vfs.*`[3] and
> > > > > `table.classpath.context`[4])
> > > > > > >> > for
> > > > > > >> > > > > things like iterator class isolation. Since 2.0.0, we
> > have
> > > > > > >> deprecated
> > > > > > >> > > > > `general.classpaths` and `general.dynamic.classpaths`,
> > the
> > > > > former
> > > > > > >> > > > > supplanted by the better use of the `CLASSPATH`
> > environment
> > > > > > >> variable
> > > > > > >> > > > > (along with much improved scripts in 2.0.0), and the
> > latter
> > > > > being
> > > > > > >> > > > > replaceable by a user-provided class loader using the
> > built-in
> > > > > > >> Java
> > > > > > >> > > > > property, `java.system.class.loader`[5], at their
> > discretion.
> > > > > > >> > > > >
> > > > > > >> > > > > # The Problem:
> > > > > > >> > > > >
> > > > > > >> > > > > The main problem with the current code is: complexity.
> > > > > Accumulo is
> > > > > > >> > > > > already complex enough without needing to be in the
> > business
> > > > > of
> > > > > > >> > > > > developing and supporting complex custom class loading
> > > > > features,
> > > > > > >> > > > > especially when users have viable alternatives that can
> > be
> > > > > better
> > > > > > >> > > > > supported by independent, dedicated projects.
> > Furthermore,
> > > > > these
> > > > > > >> > > > > custom class loaders also have a dependency on
> > commons-vfs2,
> > > > > which
> > > > > > >> > has
> > > > > > >> > > > > been the source of numerous problems and bugs that we
> > have
> > > > > needed
> > > > > > >> to
> > > > > > >> > > > > deal with, and which affect Accumulo, even though they
> > are not
> > > > > > >> > > > > necessarily bugs in Accumulo itself. This also brings
> > in a
> > > > > lot of
> > > > > > >> > > > > optional dependencies that aren't needed by users who
> > don't
> > > > > rely
> > > > > > >> on
> > > > > > >> > > > > these features.
> > > > > > >> > > > >
> > > > > > >> > > > > # The Requirements:
> > > > > > >> > > > >
> > > > > > >> > > > > In spite of these problems, I believe we still want to
> > enable
> > > > > the
> > > > > > >> use
> > > > > > >> > > > > cases that our classloaders are currently enabling.
> > > > > > >> > > > >
> > > > > > >> > > > > Specifically,
> > > > > > >> > > > > 1) the ability to have separate contexts for iterator
> > class
> > > > > > >> isolation
> > > > > > >> > > > > (A/B testing of iterators, updating iterators in a live
> > > > > system,
> > > > > > >> > etc.),
> > > > > > >> > > > > and
> > > > > > >> > > > > 2) the ability for users to bootstrap their class path
> > from
> > > > > some
> > > > > > >> > other
> > > > > > >> > > > > distributed storage than local disk.
> > > > > > >> > > > >
> > > > > > >> > > > > # The Proposal:
> > > > > > >> > > > >
> > > > > > >> > > > > 1. Create a new reloading vfs class loader, with similar
> > > > > > >> > functionality
> > > > > > >> > > > > as our current two-classloaders that do the reloading
> > and
> > > > > provide
> > > > > > >> vfs
> > > > > > >> > > > > features, that can be easily used as a system class
> > loader,
> > > > > if the
> > > > > > >> > > > > user chooses to, and deprecate (for removal in 3.0) the
> > > > > built-in
> > > > > > >> > > > > implementations. This class loader could not only be
> > used with
> > > > > > >> > > > > Accumulo, but it could also be used by any other
> > project that
> > > > > > >> chooses
> > > > > > >> > > > > to use it, because it will not have much, if any,
> > dependencies
> > > > > > >> beyond
> > > > > > >> > > > > commons-vfs2, and will certainly not depend on Accumulo.
> > > > > Creating
> > > > > > >> > this
> > > > > > >> > > > > separate class loader provides us a path forward to
> > simplify
> > > > > > >> Accumulo
> > > > > > >> > > > > by removing these features from Accumulo directly (the
> > > > > properties
> > > > > > >> are
> > > > > > >> > > > > already deprecated), and enabling it to be maintained
> > > > > > >> independently.
> > > > > > >> > > > > 2. Create a new class loader factory property in
> > Accumulo,
> > > > > with
> > > > > > >> > > > > corresponding SPI interface, for users to provide their
> > own
> > > > > > >> > > > > implementation of a class loader factory, that can map a
> > > > > per-table
> > > > > > >> > > > > "context" to a ClassLoader of the implementation's
> > choosing.
> > > > > > >> > > > >
> > > > > > >> > > > > The result of doing these two things will allow us to
> > more
> > > > > > >> flexibly
> > > > > > >> > > > > support user class loading needs, without being directly
> > > > > > >> responsible
> > > > > > >> > > > > for class loading implementations inside Accumulo's core
> > > > > code. All
> > > > > > >> > the
> > > > > > >> > > > > same functionality that is available today will
> > continue to
> > > > > exist,
> > > > > > >> > but
> > > > > > >> > > > > will be configured differently. The resulting code in
> > Accumulo
> > > > > > >> will
> > > > > > >> > be
> > > > > > >> > > > > dramatically simpler, as we would no longer have any
> > complex
> > > > > class
> > > > > > >> > > > > loading implementations in our code base, and we would
> > no
> > > > > longer
> > > > > > >> have
> > > > > > >> > > > > any direct dependency on commons-vfs2, which has been
> > > > > problematic.
> > > > > > >> > > > > Independent implementations may use commons-vfs2, or
> > something
> > > > > > >> else,
> > > > > > >> > > > > but will be more easily testable and maintainable as
> > > > > independent
> > > > > > >> > > > > projects that are pluggable in Accumulo.
> > > > > > >> > > > >
> > > > > > >> > > > > Dave has already been working on prototyping these
> > proposed
> > > > > > >> changes,
> > > > > > >> > > > > and it is looking very feasible.
> > > > > > >> > > > >
> > > > > > >> > > > > We are now ready to:
> > > > > > >> > > > > 1. get feedback on the overall proposal, and
> > > > > > >> > > > > 2. decide on where to maintain the separate class
> > loader.
> > > > > > >> > > > >
> > > > > > >> > > > > For where to maintain, the options seem to be: A) try to
> > > > > donate to
> > > > > > >> > > > > commons-vfs2 OR B) maintain as a new repository,
> > > > > > >> > > > > accumulo-vfs-classloader.
> > > > > > >> > > > >
> > > > > > >> > > > > Note that we have not yet proposed the idea of a
> > user-facing,
> > > > > > >> > > > > configurable, reloading vfs classloader to the
> > commons-vfs2
> > > > > > >> > > > > developers. We wanted to get our own community's
> > feedback on
> > > > > this
> > > > > > >> > > > > first.
> > > > > > >> > > > >
> > > > > > >> > > > > Please discuss.
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > Christopher (in collaboration with Dave)
> > > > > > >> > > > >
> > > > > > >> > > > > [1]:
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
> > > > > > >> > > > > [2]:
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
> > > > > > >> > > > > [3]:
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
> > > > > > >> > > > > [4]:
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
> > > > > > >> > > > > [5]:
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > >
> >

Re: [DISCUSS] Classloader change proposals

[Dave Marion <dm...@gmail.com>]

I have tested the new classloader as the `java.system.class.loader`, and it
works allowing me to load classes from HDFS. The reloading feature works
too, except that the ClassLoader.loadClass() logic short circuits the
loading of the new classes with the same name by calling findLoadedClass
(which returns a cached Class object). I don't believe that we will be able
to retain the reloading feature that we currently have with
`general.dynamic.classpaths` (lib/ext) in this new classloader
implementation. I have not tested it yet, but I believe that the table
context class loader reloading mechanism will work.

I know that the table context classloader feature is being used. Does
anyone know of an instance where general.dynamic.classpath is still in use
and losing the reloading feature would be unacceptable?

On Thu, Sep 24, 2020 at 10:43 AM Christopher <ct...@apache.org> wrote:

> The INFRA ticket is now closed, and the default branch has been updated.
> I also prepopulated the repo with some issue templates and GitHub
> Actions build configurations, as well as LICENSE/NOTICE and a pom.xml
> file to start it off.
>
> On Wed, Sep 23, 2020 at 10:31 AM Christopher <ct...@apache.org> wrote:
> >
> > I created the repo, but waiting on
> > https://issues.apache.org/jira/browse/INFRA-20884 to fix the GitBox
> > default branch so that the GitHub issues can be enabled.
> >
> > It should be okay to add the initial code to it, though.
> >
> > On Tue, Sep 22, 2020 at 1:40 PM Dave Marion <dm...@gmail.com> wrote:
> > >
> > > Sounds good.
> > >
> > > On Tue, Sep 22, 2020, 1:37 PM Christopher <ct...@apache.org> wrote:
> > >
> > > > Based on the conversation and direction of this, I think it probably
> > > > makes the most sense to just create a new repository under Accumulo
> > > > for it to live.
> > > >
> > > > How about 'accumulo-classloaders' for the repo name? If this is okay,
> > > > I can create it later today or tomorrow.
> > > > This is just a repo name. This is just a place to collaborate on
> > > > classloader code without constraining the scope of the repo's
> contents
> > > > too much. The actual package names and artifactId can be different,
> if
> > > > we want.
> > > >
> > > > On Tue, Sep 22, 2020 at 9:44 AM Dave Marion <dm...@gmail.com>
> wrote:
> > > > >
> > > > > [1] contains the initial set of changes to the Accumulo code base
> that
> > > > > defines the new context class loader configuration and deprecates
> the
> > > > > existing VFS ClassLoader objects. [2] contains the new
> > > > > ReloadingVFSClassLoader that can be used as the system classloader
> and a
> > > > > ClassLoaderFactory implementation for configuring contexts for
> tables and
> > > > > scans. Both build successfully and I plan on doing some local
> testing
> > > > next.
> > > > > Feedback on the design and the code is welcome.
> > > > >
> > > > > [1] https://github.com/dlmarion/accumulo/pull/2
> > > > > [2] https://github.com/dlmarion/vfs-reloading-classloader
> > > > >
> > > > > On Fri, Sep 18, 2020 at 12:19 PM Dave Marion <dm...@gmail.com>
> > > > wrote:
> > > > >
> > > > > >
> > > > > >  I tend to agree with Marc on this. If we need to push out a fix
> for
> > > > the
> > > > > > new classloader, then we can do it as needed and not have to
> rely on
> > > > > > another group of people to come to consensus on a release. Of
> course,
> > > > we
> > > > > > could maintain a fork of it in that case, but then what's the
> point? It
> > > > > > appears that VFS does have some recent activity [1], but mostly
> by one
> > > > > > person. I'm thinking that we should create a subproject for it
> and
> > > > notify
> > > > > > the commons-vfs project of its existence. If they want to copy
> it and
> > > > > > include it in their project, then they can do that.
> > > > > >
> > > > > > [1]
> https://gitbox.apache.org/repos/asf?p=commons-vfs.git;a=shortlog
> > > > > >
> > > > > > On Wed, Sep 16, 2020 at 4:48 PM Christopher <ctubbsii@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > >> Only Marc joined, and we didn't talk about anything that isn't
> > > > already in
> > > > > >> this proposal, except he did mention how difficult it might be
> to try
> > > > to
> > > > > >> maintain the class loader in commons-vfs2, rather than as our
> own
> > > > small
> > > > > >> subproject, which is relatively easy.
> > > > > >>
> > > > > >> On Wed, Sep 16, 2020, 16:11 Dave Marion <dm...@gmail.com>
> wrote:
> > > > > >>
> > > > > >> > Did anyone join the call? Any notes?
> > > > > >> >
> > > > > >> > On Wed, Sep 16, 2020 at 12:44 PM Christopher <
> ctubbsii@apache.org>
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > I just want to remind everybody that I'm available in Slack
> now to
> > > > > >> > > discuss this in the ongoing video call I created in the
> #accumulo
> > > > > >> > > room, if you want to join.
> > > > > >> > >
> > > > > >> > > On Mon, Sep 14, 2020 at 10:41 PM Christopher <
> ctubbsii@apache.org
> > > > >
> > > > > >> > wrote:
> > > > > >> > > >
> > > > > >> > > > Also, if anybody is interested in a live video
> conversation to
> > > > > >> discuss
> > > > > >> > > > this interactively, I intend to be on Slack on Wednesday
> > > > afternoon
> > > > > >> > > > (EDT) starting around noon.
> > > > > >> > > >
> > > > > >> > > > On Mon, Sep 14, 2020 at 5:30 PM Christopher <
> > > > ctubbsii@apache.org>
> > > > > >> > wrote:
> > > > > >> > > > >
> > > > > >> > > > > Hi Accumulo Devs,
> > > > > >> > > > >
> > > > > >> > > > > Lately, Dave Marion (Apache ID: dlmarion) has been
> working on
> > > > > >> > > > > prototyping some new class loader concepts for Accumulo
> that
> > > > he
> > > > > >> and I
> > > > > >> > > > > have discussed, and I wanted to pitch the idea here for
> > > > > >> consideration
> > > > > >> > > > > for the project.
> > > > > >> > > > >
> > > > > >> > > > > # Background:
> > > > > >> > > > >
> > > > > >> > > > > Accumulo currently has two classloaders that are
> instantiated
> > > > at
> > > > > >> > > > > startup, and which can be used to bootstrap Accumulo
> > > > dependencies
> > > > > >> (at
> > > > > >> > > > > least, those not needed for the classloader code
> itself). This
> > > > > >> allows
> > > > > >> > > > > us to use the `general.classpaths`[1] and
> > > > > >> > > > > `general.dynamic.classpaths`[2] properties, as well as
> the
> > > > > >> > per-context
> > > > > >> > > > > classloaders (`general.vfs.*`[3] and
> > > > `table.classpath.context`[4])
> > > > > >> > for
> > > > > >> > > > > things like iterator class isolation. Since 2.0.0, we
> have
> > > > > >> deprecated
> > > > > >> > > > > `general.classpaths` and `general.dynamic.classpaths`,
> the
> > > > former
> > > > > >> > > > > supplanted by the better use of the `CLASSPATH`
> environment
> > > > > >> variable
> > > > > >> > > > > (along with much improved scripts in 2.0.0), and the
> latter
> > > > being
> > > > > >> > > > > replaceable by a user-provided class loader using the
> built-in
> > > > > >> Java
> > > > > >> > > > > property, `java.system.class.loader`[5], at their
> discretion.
> > > > > >> > > > >
> > > > > >> > > > > # The Problem:
> > > > > >> > > > >
> > > > > >> > > > > The main problem with the current code is: complexity.
> > > > Accumulo is
> > > > > >> > > > > already complex enough without needing to be in the
> business
> > > > of
> > > > > >> > > > > developing and supporting complex custom class loading
> > > > features,
> > > > > >> > > > > especially when users have viable alternatives that can
> be
> > > > better
> > > > > >> > > > > supported by independent, dedicated projects.
> Furthermore,
> > > > these
> > > > > >> > > > > custom class loaders also have a dependency on
> commons-vfs2,
> > > > which
> > > > > >> > has
> > > > > >> > > > > been the source of numerous problems and bugs that we
> have
> > > > needed
> > > > > >> to
> > > > > >> > > > > deal with, and which affect Accumulo, even though they
> are not
> > > > > >> > > > > necessarily bugs in Accumulo itself. This also brings
> in a
> > > > lot of
> > > > > >> > > > > optional dependencies that aren't needed by users who
> don't
> > > > rely
> > > > > >> on
> > > > > >> > > > > these features.
> > > > > >> > > > >
> > > > > >> > > > > # The Requirements:
> > > > > >> > > > >
> > > > > >> > > > > In spite of these problems, I believe we still want to
> enable
> > > > the
> > > > > >> use
> > > > > >> > > > > cases that our classloaders are currently enabling.
> > > > > >> > > > >
> > > > > >> > > > > Specifically,
> > > > > >> > > > > 1) the ability to have separate contexts for iterator
> class
> > > > > >> isolation
> > > > > >> > > > > (A/B testing of iterators, updating iterators in a live
> > > > system,
> > > > > >> > etc.),
> > > > > >> > > > > and
> > > > > >> > > > > 2) the ability for users to bootstrap their class path
> from
> > > > some
> > > > > >> > other
> > > > > >> > > > > distributed storage than local disk.
> > > > > >> > > > >
> > > > > >> > > > > # The Proposal:
> > > > > >> > > > >
> > > > > >> > > > > 1. Create a new reloading vfs class loader, with similar
> > > > > >> > functionality
> > > > > >> > > > > as our current two-classloaders that do the reloading
> and
> > > > provide
> > > > > >> vfs
> > > > > >> > > > > features, that can be easily used as a system class
> loader,
> > > > if the
> > > > > >> > > > > user chooses to, and deprecate (for removal in 3.0) the
> > > > built-in
> > > > > >> > > > > implementations. This class loader could not only be
> used with
> > > > > >> > > > > Accumulo, but it could also be used by any other
> project that
> > > > > >> chooses
> > > > > >> > > > > to use it, because it will not have much, if any,
> dependencies
> > > > > >> beyond
> > > > > >> > > > > commons-vfs2, and will certainly not depend on Accumulo.
> > > > Creating
> > > > > >> > this
> > > > > >> > > > > separate class loader provides us a path forward to
> simplify
> > > > > >> Accumulo
> > > > > >> > > > > by removing these features from Accumulo directly (the
> > > > properties
> > > > > >> are
> > > > > >> > > > > already deprecated), and enabling it to be maintained
> > > > > >> independently.
> > > > > >> > > > > 2. Create a new class loader factory property in
> Accumulo,
> > > > with
> > > > > >> > > > > corresponding SPI interface, for users to provide their
> own
> > > > > >> > > > > implementation of a class loader factory, that can map a
> > > > per-table
> > > > > >> > > > > "context" to a ClassLoader of the implementation's
> choosing.
> > > > > >> > > > >
> > > > > >> > > > > The result of doing these two things will allow us to
> more
> > > > > >> flexibly
> > > > > >> > > > > support user class loading needs, without being directly
> > > > > >> responsible
> > > > > >> > > > > for class loading implementations inside Accumulo's core
> > > > code. All
> > > > > >> > the
> > > > > >> > > > > same functionality that is available today will
> continue to
> > > > exist,
> > > > > >> > but
> > > > > >> > > > > will be configured differently. The resulting code in
> Accumulo
> > > > > >> will
> > > > > >> > be
> > > > > >> > > > > dramatically simpler, as we would no longer have any
> complex
> > > > class
> > > > > >> > > > > loading implementations in our code base, and we would
> no
> > > > longer
> > > > > >> have
> > > > > >> > > > > any direct dependency on commons-vfs2, which has been
> > > > problematic.
> > > > > >> > > > > Independent implementations may use commons-vfs2, or
> something
> > > > > >> else,
> > > > > >> > > > > but will be more easily testable and maintainable as
> > > > independent
> > > > > >> > > > > projects that are pluggable in Accumulo.
> > > > > >> > > > >
> > > > > >> > > > > Dave has already been working on prototyping these
> proposed
> > > > > >> changes,
> > > > > >> > > > > and it is looking very feasible.
> > > > > >> > > > >
> > > > > >> > > > > We are now ready to:
> > > > > >> > > > > 1. get feedback on the overall proposal, and
> > > > > >> > > > > 2. decide on where to maintain the separate class
> loader.
> > > > > >> > > > >
> > > > > >> > > > > For where to maintain, the options seem to be: A) try to
> > > > donate to
> > > > > >> > > > > commons-vfs2 OR B) maintain as a new repository,
> > > > > >> > > > > accumulo-vfs-classloader.
> > > > > >> > > > >
> > > > > >> > > > > Note that we have not yet proposed the idea of a
> user-facing,
> > > > > >> > > > > configurable, reloading vfs classloader to the
> commons-vfs2
> > > > > >> > > > > developers. We wanted to get our own community's
> feedback on
> > > > this
> > > > > >> > > > > first.
> > > > > >> > > > >
> > > > > >> > > > > Please discuss.
> > > > > >> > > > >
> > > > > >> > > > > Thanks,
> > > > > >> > > > > Christopher (in collaboration with Dave)
> > > > > >> > > > >
> > > > > >> > > > > [1]:
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
> > > > > >> > > > > [2]:
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
> > > > > >> > > > > [3]:
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
> > > > > >> > > > > [4]:
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
> > > > > >> > > > > [5]:
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > >
>

Re: [DISCUSS] Classloader change proposals

[Christopher <ct...@apache.org>]

The INFRA ticket is now closed, and the default branch has been updated.
I also prepopulated the repo with some issue templates and GitHub
Actions build configurations, as well as LICENSE/NOTICE and a pom.xml
file to start it off.

On Wed, Sep 23, 2020 at 10:31 AM Christopher <ct...@apache.org> wrote:
>
> I created the repo, but waiting on
> https://issues.apache.org/jira/browse/INFRA-20884 to fix the GitBox
> default branch so that the GitHub issues can be enabled.
>
> It should be okay to add the initial code to it, though.
>
> On Tue, Sep 22, 2020 at 1:40 PM Dave Marion <dm...@gmail.com> wrote:
> >
> > Sounds good.
> >
> > On Tue, Sep 22, 2020, 1:37 PM Christopher <ct...@apache.org> wrote:
> >
> > > Based on the conversation and direction of this, I think it probably
> > > makes the most sense to just create a new repository under Accumulo
> > > for it to live.
> > >
> > > How about 'accumulo-classloaders' for the repo name? If this is okay,
> > > I can create it later today or tomorrow.
> > > This is just a repo name. This is just a place to collaborate on
> > > classloader code without constraining the scope of the repo's contents
> > > too much. The actual package names and artifactId can be different, if
> > > we want.
> > >
> > > On Tue, Sep 22, 2020 at 9:44 AM Dave Marion <dm...@gmail.com> wrote:
> > > >
> > > > [1] contains the initial set of changes to the Accumulo code base that
> > > > defines the new context class loader configuration and deprecates the
> > > > existing VFS ClassLoader objects. [2] contains the new
> > > > ReloadingVFSClassLoader that can be used as the system classloader and a
> > > > ClassLoaderFactory implementation for configuring contexts for tables and
> > > > scans. Both build successfully and I plan on doing some local testing
> > > next.
> > > > Feedback on the design and the code is welcome.
> > > >
> > > > [1] https://github.com/dlmarion/accumulo/pull/2
> > > > [2] https://github.com/dlmarion/vfs-reloading-classloader
> > > >
> > > > On Fri, Sep 18, 2020 at 12:19 PM Dave Marion <dm...@gmail.com>
> > > wrote:
> > > >
> > > > >
> > > > >  I tend to agree with Marc on this. If we need to push out a fix for
> > > the
> > > > > new classloader, then we can do it as needed and not have to rely on
> > > > > another group of people to come to consensus on a release. Of course,
> > > we
> > > > > could maintain a fork of it in that case, but then what's the point? It
> > > > > appears that VFS does have some recent activity [1], but mostly by one
> > > > > person. I'm thinking that we should create a subproject for it and
> > > notify
> > > > > the commons-vfs project of its existence. If they want to copy it and
> > > > > include it in their project, then they can do that.
> > > > >
> > > > > [1] https://gitbox.apache.org/repos/asf?p=commons-vfs.git;a=shortlog
> > > > >
> > > > > On Wed, Sep 16, 2020 at 4:48 PM Christopher <ct...@apache.org>
> > > wrote:
> > > > >
> > > > >> Only Marc joined, and we didn't talk about anything that isn't
> > > already in
> > > > >> this proposal, except he did mention how difficult it might be to try
> > > to
> > > > >> maintain the class loader in commons-vfs2, rather than as our own
> > > small
> > > > >> subproject, which is relatively easy.
> > > > >>
> > > > >> On Wed, Sep 16, 2020, 16:11 Dave Marion <dm...@gmail.com> wrote:
> > > > >>
> > > > >> > Did anyone join the call? Any notes?
> > > > >> >
> > > > >> > On Wed, Sep 16, 2020 at 12:44 PM Christopher <ct...@apache.org>
> > > > >> wrote:
> > > > >> >
> > > > >> > > I just want to remind everybody that I'm available in Slack now to
> > > > >> > > discuss this in the ongoing video call I created in the #accumulo
> > > > >> > > room, if you want to join.
> > > > >> > >
> > > > >> > > On Mon, Sep 14, 2020 at 10:41 PM Christopher <ctubbsii@apache.org
> > > >
> > > > >> > wrote:
> > > > >> > > >
> > > > >> > > > Also, if anybody is interested in a live video conversation to
> > > > >> discuss
> > > > >> > > > this interactively, I intend to be on Slack on Wednesday
> > > afternoon
> > > > >> > > > (EDT) starting around noon.
> > > > >> > > >
> > > > >> > > > On Mon, Sep 14, 2020 at 5:30 PM Christopher <
> > > ctubbsii@apache.org>
> > > > >> > wrote:
> > > > >> > > > >
> > > > >> > > > > Hi Accumulo Devs,
> > > > >> > > > >
> > > > >> > > > > Lately, Dave Marion (Apache ID: dlmarion) has been working on
> > > > >> > > > > prototyping some new class loader concepts for Accumulo that
> > > he
> > > > >> and I
> > > > >> > > > > have discussed, and I wanted to pitch the idea here for
> > > > >> consideration
> > > > >> > > > > for the project.
> > > > >> > > > >
> > > > >> > > > > # Background:
> > > > >> > > > >
> > > > >> > > > > Accumulo currently has two classloaders that are instantiated
> > > at
> > > > >> > > > > startup, and which can be used to bootstrap Accumulo
> > > dependencies
> > > > >> (at
> > > > >> > > > > least, those not needed for the classloader code itself). This
> > > > >> allows
> > > > >> > > > > us to use the `general.classpaths`[1] and
> > > > >> > > > > `general.dynamic.classpaths`[2] properties, as well as the
> > > > >> > per-context
> > > > >> > > > > classloaders (`general.vfs.*`[3] and
> > > `table.classpath.context`[4])
> > > > >> > for
> > > > >> > > > > things like iterator class isolation. Since 2.0.0, we have
> > > > >> deprecated
> > > > >> > > > > `general.classpaths` and `general.dynamic.classpaths`, the
> > > former
> > > > >> > > > > supplanted by the better use of the `CLASSPATH` environment
> > > > >> variable
> > > > >> > > > > (along with much improved scripts in 2.0.0), and the latter
> > > being
> > > > >> > > > > replaceable by a user-provided class loader using the built-in
> > > > >> Java
> > > > >> > > > > property, `java.system.class.loader`[5], at their discretion.
> > > > >> > > > >
> > > > >> > > > > # The Problem:
> > > > >> > > > >
> > > > >> > > > > The main problem with the current code is: complexity.
> > > Accumulo is
> > > > >> > > > > already complex enough without needing to be in the business
> > > of
> > > > >> > > > > developing and supporting complex custom class loading
> > > features,
> > > > >> > > > > especially when users have viable alternatives that can be
> > > better
> > > > >> > > > > supported by independent, dedicated projects. Furthermore,
> > > these
> > > > >> > > > > custom class loaders also have a dependency on commons-vfs2,
> > > which
> > > > >> > has
> > > > >> > > > > been the source of numerous problems and bugs that we have
> > > needed
> > > > >> to
> > > > >> > > > > deal with, and which affect Accumulo, even though they are not
> > > > >> > > > > necessarily bugs in Accumulo itself. This also brings in a
> > > lot of
> > > > >> > > > > optional dependencies that aren't needed by users who don't
> > > rely
> > > > >> on
> > > > >> > > > > these features.
> > > > >> > > > >
> > > > >> > > > > # The Requirements:
> > > > >> > > > >
> > > > >> > > > > In spite of these problems, I believe we still want to enable
> > > the
> > > > >> use
> > > > >> > > > > cases that our classloaders are currently enabling.
> > > > >> > > > >
> > > > >> > > > > Specifically,
> > > > >> > > > > 1) the ability to have separate contexts for iterator class
> > > > >> isolation
> > > > >> > > > > (A/B testing of iterators, updating iterators in a live
> > > system,
> > > > >> > etc.),
> > > > >> > > > > and
> > > > >> > > > > 2) the ability for users to bootstrap their class path from
> > > some
> > > > >> > other
> > > > >> > > > > distributed storage than local disk.
> > > > >> > > > >
> > > > >> > > > > # The Proposal:
> > > > >> > > > >
> > > > >> > > > > 1. Create a new reloading vfs class loader, with similar
> > > > >> > functionality
> > > > >> > > > > as our current two-classloaders that do the reloading and
> > > provide
> > > > >> vfs
> > > > >> > > > > features, that can be easily used as a system class loader,
> > > if the
> > > > >> > > > > user chooses to, and deprecate (for removal in 3.0) the
> > > built-in
> > > > >> > > > > implementations. This class loader could not only be used with
> > > > >> > > > > Accumulo, but it could also be used by any other project that
> > > > >> chooses
> > > > >> > > > > to use it, because it will not have much, if any, dependencies
> > > > >> beyond
> > > > >> > > > > commons-vfs2, and will certainly not depend on Accumulo.
> > > Creating
> > > > >> > this
> > > > >> > > > > separate class loader provides us a path forward to simplify
> > > > >> Accumulo
> > > > >> > > > > by removing these features from Accumulo directly (the
> > > properties
> > > > >> are
> > > > >> > > > > already deprecated), and enabling it to be maintained
> > > > >> independently.
> > > > >> > > > > 2. Create a new class loader factory property in Accumulo,
> > > with
> > > > >> > > > > corresponding SPI interface, for users to provide their own
> > > > >> > > > > implementation of a class loader factory, that can map a
> > > per-table
> > > > >> > > > > "context" to a ClassLoader of the implementation's choosing.
> > > > >> > > > >
> > > > >> > > > > The result of doing these two things will allow us to more
> > > > >> flexibly
> > > > >> > > > > support user class loading needs, without being directly
> > > > >> responsible
> > > > >> > > > > for class loading implementations inside Accumulo's core
> > > code. All
> > > > >> > the
> > > > >> > > > > same functionality that is available today will continue to
> > > exist,
> > > > >> > but
> > > > >> > > > > will be configured differently. The resulting code in Accumulo
> > > > >> will
> > > > >> > be
> > > > >> > > > > dramatically simpler, as we would no longer have any complex
> > > class
> > > > >> > > > > loading implementations in our code base, and we would no
> > > longer
> > > > >> have
> > > > >> > > > > any direct dependency on commons-vfs2, which has been
> > > problematic.
> > > > >> > > > > Independent implementations may use commons-vfs2, or something
> > > > >> else,
> > > > >> > > > > but will be more easily testable and maintainable as
> > > independent
> > > > >> > > > > projects that are pluggable in Accumulo.
> > > > >> > > > >
> > > > >> > > > > Dave has already been working on prototyping these proposed
> > > > >> changes,
> > > > >> > > > > and it is looking very feasible.
> > > > >> > > > >
> > > > >> > > > > We are now ready to:
> > > > >> > > > > 1. get feedback on the overall proposal, and
> > > > >> > > > > 2. decide on where to maintain the separate class loader.
> > > > >> > > > >
> > > > >> > > > > For where to maintain, the options seem to be: A) try to
> > > donate to
> > > > >> > > > > commons-vfs2 OR B) maintain as a new repository,
> > > > >> > > > > accumulo-vfs-classloader.
> > > > >> > > > >
> > > > >> > > > > Note that we have not yet proposed the idea of a user-facing,
> > > > >> > > > > configurable, reloading vfs classloader to the commons-vfs2
> > > > >> > > > > developers. We wanted to get our own community's feedback on
> > > this
> > > > >> > > > > first.
> > > > >> > > > >
> > > > >> > > > > Please discuss.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Christopher (in collaboration with Dave)
> > > > >> > > > >
> > > > >> > > > > [1]:
> > > > >> > >
> > > > >> >
> > > > >>
> > > https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
> > > > >> > > > > [2]:
> > > > >> > >
> > > > >> >
> > > > >>
> > > https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
> > > > >> > > > > [3]:
> > > > >> > >
> > > > >> >
> > > > >>
> > > https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
> > > > >> > > > > [4]:
> > > > >> > >
> > > > >> >
> > > > >>
> > > https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
> > > > >> > > > > [5]:
> > > > >> > >
> > > > >> >
> > > > >>
> > > https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > >

Re: [DISCUSS] Classloader change proposals

[Christopher <ct...@apache.org>]

I created the repo, but waiting on
https://issues.apache.org/jira/browse/INFRA-20884 to fix the GitBox
default branch so that the GitHub issues can be enabled.

It should be okay to add the initial code to it, though.

On Tue, Sep 22, 2020 at 1:40 PM Dave Marion <dm...@gmail.com> wrote:
>
> Sounds good.
>
> On Tue, Sep 22, 2020, 1:37 PM Christopher <ct...@apache.org> wrote:
>
> > Based on the conversation and direction of this, I think it probably
> > makes the most sense to just create a new repository under Accumulo
> > for it to live.
> >
> > How about 'accumulo-classloaders' for the repo name? If this is okay,
> > I can create it later today or tomorrow.
> > This is just a repo name. This is just a place to collaborate on
> > classloader code without constraining the scope of the repo's contents
> > too much. The actual package names and artifactId can be different, if
> > we want.
> >
> > On Tue, Sep 22, 2020 at 9:44 AM Dave Marion <dm...@gmail.com> wrote:
> > >
> > > [1] contains the initial set of changes to the Accumulo code base that
> > > defines the new context class loader configuration and deprecates the
> > > existing VFS ClassLoader objects. [2] contains the new
> > > ReloadingVFSClassLoader that can be used as the system classloader and a
> > > ClassLoaderFactory implementation for configuring contexts for tables and
> > > scans. Both build successfully and I plan on doing some local testing
> > next.
> > > Feedback on the design and the code is welcome.
> > >
> > > [1] https://github.com/dlmarion/accumulo/pull/2
> > > [2] https://github.com/dlmarion/vfs-reloading-classloader
> > >
> > > On Fri, Sep 18, 2020 at 12:19 PM Dave Marion <dm...@gmail.com>
> > wrote:
> > >
> > > >
> > > >  I tend to agree with Marc on this. If we need to push out a fix for
> > the
> > > > new classloader, then we can do it as needed and not have to rely on
> > > > another group of people to come to consensus on a release. Of course,
> > we
> > > > could maintain a fork of it in that case, but then what's the point? It
> > > > appears that VFS does have some recent activity [1], but mostly by one
> > > > person. I'm thinking that we should create a subproject for it and
> > notify
> > > > the commons-vfs project of its existence. If they want to copy it and
> > > > include it in their project, then they can do that.
> > > >
> > > > [1] https://gitbox.apache.org/repos/asf?p=commons-vfs.git;a=shortlog
> > > >
> > > > On Wed, Sep 16, 2020 at 4:48 PM Christopher <ct...@apache.org>
> > wrote:
> > > >
> > > >> Only Marc joined, and we didn't talk about anything that isn't
> > already in
> > > >> this proposal, except he did mention how difficult it might be to try
> > to
> > > >> maintain the class loader in commons-vfs2, rather than as our own
> > small
> > > >> subproject, which is relatively easy.
> > > >>
> > > >> On Wed, Sep 16, 2020, 16:11 Dave Marion <dm...@gmail.com> wrote:
> > > >>
> > > >> > Did anyone join the call? Any notes?
> > > >> >
> > > >> > On Wed, Sep 16, 2020 at 12:44 PM Christopher <ct...@apache.org>
> > > >> wrote:
> > > >> >
> > > >> > > I just want to remind everybody that I'm available in Slack now to
> > > >> > > discuss this in the ongoing video call I created in the #accumulo
> > > >> > > room, if you want to join.
> > > >> > >
> > > >> > > On Mon, Sep 14, 2020 at 10:41 PM Christopher <ctubbsii@apache.org
> > >
> > > >> > wrote:
> > > >> > > >
> > > >> > > > Also, if anybody is interested in a live video conversation to
> > > >> discuss
> > > >> > > > this interactively, I intend to be on Slack on Wednesday
> > afternoon
> > > >> > > > (EDT) starting around noon.
> > > >> > > >
> > > >> > > > On Mon, Sep 14, 2020 at 5:30 PM Christopher <
> > ctubbsii@apache.org>
> > > >> > wrote:
> > > >> > > > >
> > > >> > > > > Hi Accumulo Devs,
> > > >> > > > >
> > > >> > > > > Lately, Dave Marion (Apache ID: dlmarion) has been working on
> > > >> > > > > prototyping some new class loader concepts for Accumulo that
> > he
> > > >> and I
> > > >> > > > > have discussed, and I wanted to pitch the idea here for
> > > >> consideration
> > > >> > > > > for the project.
> > > >> > > > >
> > > >> > > > > # Background:
> > > >> > > > >
> > > >> > > > > Accumulo currently has two classloaders that are instantiated
> > at
> > > >> > > > > startup, and which can be used to bootstrap Accumulo
> > dependencies
> > > >> (at
> > > >> > > > > least, those not needed for the classloader code itself). This
> > > >> allows
> > > >> > > > > us to use the `general.classpaths`[1] and
> > > >> > > > > `general.dynamic.classpaths`[2] properties, as well as the
> > > >> > per-context
> > > >> > > > > classloaders (`general.vfs.*`[3] and
> > `table.classpath.context`[4])
> > > >> > for
> > > >> > > > > things like iterator class isolation. Since 2.0.0, we have
> > > >> deprecated
> > > >> > > > > `general.classpaths` and `general.dynamic.classpaths`, the
> > former
> > > >> > > > > supplanted by the better use of the `CLASSPATH` environment
> > > >> variable
> > > >> > > > > (along with much improved scripts in 2.0.0), and the latter
> > being
> > > >> > > > > replaceable by a user-provided class loader using the built-in
> > > >> Java
> > > >> > > > > property, `java.system.class.loader`[5], at their discretion.
> > > >> > > > >
> > > >> > > > > # The Problem:
> > > >> > > > >
> > > >> > > > > The main problem with the current code is: complexity.
> > Accumulo is
> > > >> > > > > already complex enough without needing to be in the business
> > of
> > > >> > > > > developing and supporting complex custom class loading
> > features,
> > > >> > > > > especially when users have viable alternatives that can be
> > better
> > > >> > > > > supported by independent, dedicated projects. Furthermore,
> > these
> > > >> > > > > custom class loaders also have a dependency on commons-vfs2,
> > which
> > > >> > has
> > > >> > > > > been the source of numerous problems and bugs that we have
> > needed
> > > >> to
> > > >> > > > > deal with, and which affect Accumulo, even though they are not
> > > >> > > > > necessarily bugs in Accumulo itself. This also brings in a
> > lot of
> > > >> > > > > optional dependencies that aren't needed by users who don't
> > rely
> > > >> on
> > > >> > > > > these features.
> > > >> > > > >
> > > >> > > > > # The Requirements:
> > > >> > > > >
> > > >> > > > > In spite of these problems, I believe we still want to enable
> > the
> > > >> use
> > > >> > > > > cases that our classloaders are currently enabling.
> > > >> > > > >
> > > >> > > > > Specifically,
> > > >> > > > > 1) the ability to have separate contexts for iterator class
> > > >> isolation
> > > >> > > > > (A/B testing of iterators, updating iterators in a live
> > system,
> > > >> > etc.),
> > > >> > > > > and
> > > >> > > > > 2) the ability for users to bootstrap their class path from
> > some
> > > >> > other
> > > >> > > > > distributed storage than local disk.
> > > >> > > > >
> > > >> > > > > # The Proposal:
> > > >> > > > >
> > > >> > > > > 1. Create a new reloading vfs class loader, with similar
> > > >> > functionality
> > > >> > > > > as our current two-classloaders that do the reloading and
> > provide
> > > >> vfs
> > > >> > > > > features, that can be easily used as a system class loader,
> > if the
> > > >> > > > > user chooses to, and deprecate (for removal in 3.0) the
> > built-in
> > > >> > > > > implementations. This class loader could not only be used with
> > > >> > > > > Accumulo, but it could also be used by any other project that
> > > >> chooses
> > > >> > > > > to use it, because it will not have much, if any, dependencies
> > > >> beyond
> > > >> > > > > commons-vfs2, and will certainly not depend on Accumulo.
> > Creating
> > > >> > this
> > > >> > > > > separate class loader provides us a path forward to simplify
> > > >> Accumulo
> > > >> > > > > by removing these features from Accumulo directly (the
> > properties
> > > >> are
> > > >> > > > > already deprecated), and enabling it to be maintained
> > > >> independently.
> > > >> > > > > 2. Create a new class loader factory property in Accumulo,
> > with
> > > >> > > > > corresponding SPI interface, for users to provide their own
> > > >> > > > > implementation of a class loader factory, that can map a
> > per-table
> > > >> > > > > "context" to a ClassLoader of the implementation's choosing.
> > > >> > > > >
> > > >> > > > > The result of doing these two things will allow us to more
> > > >> flexibly
> > > >> > > > > support user class loading needs, without being directly
> > > >> responsible
> > > >> > > > > for class loading implementations inside Accumulo's core
> > code. All
> > > >> > the
> > > >> > > > > same functionality that is available today will continue to
> > exist,
> > > >> > but
> > > >> > > > > will be configured differently. The resulting code in Accumulo
> > > >> will
> > > >> > be
> > > >> > > > > dramatically simpler, as we would no longer have any complex
> > class
> > > >> > > > > loading implementations in our code base, and we would no
> > longer
> > > >> have
> > > >> > > > > any direct dependency on commons-vfs2, which has been
> > problematic.
> > > >> > > > > Independent implementations may use commons-vfs2, or something
> > > >> else,
> > > >> > > > > but will be more easily testable and maintainable as
> > independent
> > > >> > > > > projects that are pluggable in Accumulo.
> > > >> > > > >
> > > >> > > > > Dave has already been working on prototyping these proposed
> > > >> changes,
> > > >> > > > > and it is looking very feasible.
> > > >> > > > >
> > > >> > > > > We are now ready to:
> > > >> > > > > 1. get feedback on the overall proposal, and
> > > >> > > > > 2. decide on where to maintain the separate class loader.
> > > >> > > > >
> > > >> > > > > For where to maintain, the options seem to be: A) try to
> > donate to
> > > >> > > > > commons-vfs2 OR B) maintain as a new repository,
> > > >> > > > > accumulo-vfs-classloader.
> > > >> > > > >
> > > >> > > > > Note that we have not yet proposed the idea of a user-facing,
> > > >> > > > > configurable, reloading vfs classloader to the commons-vfs2
> > > >> > > > > developers. We wanted to get our own community's feedback on
> > this
> > > >> > > > > first.
> > > >> > > > >
> > > >> > > > > Please discuss.
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > Christopher (in collaboration with Dave)
> > > >> > > > >
> > > >> > > > > [1]:
> > > >> > >
> > > >> >
> > > >>
> > https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
> > > >> > > > > [2]:
> > > >> > >
> > > >> >
> > > >>
> > https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
> > > >> > > > > [3]:
> > > >> > >
> > > >> >
> > > >>
> > https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
> > > >> > > > > [4]:
> > > >> > >
> > > >> >
> > > >>
> > https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
> > > >> > > > > [5]:
> > > >> > >
> > > >> >
> > > >>
> > https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93
> > > >> > >
> > > >> >
> > > >>
> > > >
> >

Re: [DISCUSS] Classloader change proposals

[Dave Marion <dm...@gmail.com>]

Sounds good.

On Tue, Sep 22, 2020, 1:37 PM Christopher <ct...@apache.org> wrote:

> Based on the conversation and direction of this, I think it probably
> makes the most sense to just create a new repository under Accumulo
> for it to live.
>
> How about 'accumulo-classloaders' for the repo name? If this is okay,
> I can create it later today or tomorrow.
> This is just a repo name. This is just a place to collaborate on
> classloader code without constraining the scope of the repo's contents
> too much. The actual package names and artifactId can be different, if
> we want.
>
> On Tue, Sep 22, 2020 at 9:44 AM Dave Marion <dm...@gmail.com> wrote:
> >
> > [1] contains the initial set of changes to the Accumulo code base that
> > defines the new context class loader configuration and deprecates the
> > existing VFS ClassLoader objects. [2] contains the new
> > ReloadingVFSClassLoader that can be used as the system classloader and a
> > ClassLoaderFactory implementation for configuring contexts for tables and
> > scans. Both build successfully and I plan on doing some local testing
> next.
> > Feedback on the design and the code is welcome.
> >
> > [1] https://github.com/dlmarion/accumulo/pull/2
> > [2] https://github.com/dlmarion/vfs-reloading-classloader
> >
> > On Fri, Sep 18, 2020 at 12:19 PM Dave Marion <dm...@gmail.com>
> wrote:
> >
> > >
> > >  I tend to agree with Marc on this. If we need to push out a fix for
> the
> > > new classloader, then we can do it as needed and not have to rely on
> > > another group of people to come to consensus on a release. Of course,
> we
> > > could maintain a fork of it in that case, but then what's the point? It
> > > appears that VFS does have some recent activity [1], but mostly by one
> > > person. I'm thinking that we should create a subproject for it and
> notify
> > > the commons-vfs project of its existence. If they want to copy it and
> > > include it in their project, then they can do that.
> > >
> > > [1] https://gitbox.apache.org/repos/asf?p=commons-vfs.git;a=shortlog
> > >
> > > On Wed, Sep 16, 2020 at 4:48 PM Christopher <ct...@apache.org>
> wrote:
> > >
> > >> Only Marc joined, and we didn't talk about anything that isn't
> already in
> > >> this proposal, except he did mention how difficult it might be to try
> to
> > >> maintain the class loader in commons-vfs2, rather than as our own
> small
> > >> subproject, which is relatively easy.
> > >>
> > >> On Wed, Sep 16, 2020, 16:11 Dave Marion <dm...@gmail.com> wrote:
> > >>
> > >> > Did anyone join the call? Any notes?
> > >> >
> > >> > On Wed, Sep 16, 2020 at 12:44 PM Christopher <ct...@apache.org>
> > >> wrote:
> > >> >
> > >> > > I just want to remind everybody that I'm available in Slack now to
> > >> > > discuss this in the ongoing video call I created in the #accumulo
> > >> > > room, if you want to join.
> > >> > >
> > >> > > On Mon, Sep 14, 2020 at 10:41 PM Christopher <ctubbsii@apache.org
> >
> > >> > wrote:
> > >> > > >
> > >> > > > Also, if anybody is interested in a live video conversation to
> > >> discuss
> > >> > > > this interactively, I intend to be on Slack on Wednesday
> afternoon
> > >> > > > (EDT) starting around noon.
> > >> > > >
> > >> > > > On Mon, Sep 14, 2020 at 5:30 PM Christopher <
> ctubbsii@apache.org>
> > >> > wrote:
> > >> > > > >
> > >> > > > > Hi Accumulo Devs,
> > >> > > > >
> > >> > > > > Lately, Dave Marion (Apache ID: dlmarion) has been working on
> > >> > > > > prototyping some new class loader concepts for Accumulo that
> he
> > >> and I
> > >> > > > > have discussed, and I wanted to pitch the idea here for
> > >> consideration
> > >> > > > > for the project.
> > >> > > > >
> > >> > > > > # Background:
> > >> > > > >
> > >> > > > > Accumulo currently has two classloaders that are instantiated
> at
> > >> > > > > startup, and which can be used to bootstrap Accumulo
> dependencies
> > >> (at
> > >> > > > > least, those not needed for the classloader code itself). This
> > >> allows
> > >> > > > > us to use the `general.classpaths`[1] and
> > >> > > > > `general.dynamic.classpaths`[2] properties, as well as the
> > >> > per-context
> > >> > > > > classloaders (`general.vfs.*`[3] and
> `table.classpath.context`[4])
> > >> > for
> > >> > > > > things like iterator class isolation. Since 2.0.0, we have
> > >> deprecated
> > >> > > > > `general.classpaths` and `general.dynamic.classpaths`, the
> former
> > >> > > > > supplanted by the better use of the `CLASSPATH` environment
> > >> variable
> > >> > > > > (along with much improved scripts in 2.0.0), and the latter
> being
> > >> > > > > replaceable by a user-provided class loader using the built-in
> > >> Java
> > >> > > > > property, `java.system.class.loader`[5], at their discretion.
> > >> > > > >
> > >> > > > > # The Problem:
> > >> > > > >
> > >> > > > > The main problem with the current code is: complexity.
> Accumulo is
> > >> > > > > already complex enough without needing to be in the business
> of
> > >> > > > > developing and supporting complex custom class loading
> features,
> > >> > > > > especially when users have viable alternatives that can be
> better
> > >> > > > > supported by independent, dedicated projects. Furthermore,
> these
> > >> > > > > custom class loaders also have a dependency on commons-vfs2,
> which
> > >> > has
> > >> > > > > been the source of numerous problems and bugs that we have
> needed
> > >> to
> > >> > > > > deal with, and which affect Accumulo, even though they are not
> > >> > > > > necessarily bugs in Accumulo itself. This also brings in a
> lot of
> > >> > > > > optional dependencies that aren't needed by users who don't
> rely
> > >> on
> > >> > > > > these features.
> > >> > > > >
> > >> > > > > # The Requirements:
> > >> > > > >
> > >> > > > > In spite of these problems, I believe we still want to enable
> the
> > >> use
> > >> > > > > cases that our classloaders are currently enabling.
> > >> > > > >
> > >> > > > > Specifically,
> > >> > > > > 1) the ability to have separate contexts for iterator class
> > >> isolation
> > >> > > > > (A/B testing of iterators, updating iterators in a live
> system,
> > >> > etc.),
> > >> > > > > and
> > >> > > > > 2) the ability for users to bootstrap their class path from
> some
> > >> > other
> > >> > > > > distributed storage than local disk.
> > >> > > > >
> > >> > > > > # The Proposal:
> > >> > > > >
> > >> > > > > 1. Create a new reloading vfs class loader, with similar
> > >> > functionality
> > >> > > > > as our current two-classloaders that do the reloading and
> provide
> > >> vfs
> > >> > > > > features, that can be easily used as a system class loader,
> if the
> > >> > > > > user chooses to, and deprecate (for removal in 3.0) the
> built-in
> > >> > > > > implementations. This class loader could not only be used with
> > >> > > > > Accumulo, but it could also be used by any other project that
> > >> chooses
> > >> > > > > to use it, because it will not have much, if any, dependencies
> > >> beyond
> > >> > > > > commons-vfs2, and will certainly not depend on Accumulo.
> Creating
> > >> > this
> > >> > > > > separate class loader provides us a path forward to simplify
> > >> Accumulo
> > >> > > > > by removing these features from Accumulo directly (the
> properties
> > >> are
> > >> > > > > already deprecated), and enabling it to be maintained
> > >> independently.
> > >> > > > > 2. Create a new class loader factory property in Accumulo,
> with
> > >> > > > > corresponding SPI interface, for users to provide their own
> > >> > > > > implementation of a class loader factory, that can map a
> per-table
> > >> > > > > "context" to a ClassLoader of the implementation's choosing.
> > >> > > > >
> > >> > > > > The result of doing these two things will allow us to more
> > >> flexibly
> > >> > > > > support user class loading needs, without being directly
> > >> responsible
> > >> > > > > for class loading implementations inside Accumulo's core
> code. All
> > >> > the
> > >> > > > > same functionality that is available today will continue to
> exist,
> > >> > but
> > >> > > > > will be configured differently. The resulting code in Accumulo
> > >> will
> > >> > be
> > >> > > > > dramatically simpler, as we would no longer have any complex
> class
> > >> > > > > loading implementations in our code base, and we would no
> longer
> > >> have
> > >> > > > > any direct dependency on commons-vfs2, which has been
> problematic.
> > >> > > > > Independent implementations may use commons-vfs2, or something
> > >> else,
> > >> > > > > but will be more easily testable and maintainable as
> independent
> > >> > > > > projects that are pluggable in Accumulo.
> > >> > > > >
> > >> > > > > Dave has already been working on prototyping these proposed
> > >> changes,
> > >> > > > > and it is looking very feasible.
> > >> > > > >
> > >> > > > > We are now ready to:
> > >> > > > > 1. get feedback on the overall proposal, and
> > >> > > > > 2. decide on where to maintain the separate class loader.
> > >> > > > >
> > >> > > > > For where to maintain, the options seem to be: A) try to
> donate to
> > >> > > > > commons-vfs2 OR B) maintain as a new repository,
> > >> > > > > accumulo-vfs-classloader.
> > >> > > > >
> > >> > > > > Note that we have not yet proposed the idea of a user-facing,
> > >> > > > > configurable, reloading vfs classloader to the commons-vfs2
> > >> > > > > developers. We wanted to get our own community's feedback on
> this
> > >> > > > > first.
> > >> > > > >
> > >> > > > > Please discuss.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Christopher (in collaboration with Dave)
> > >> > > > >
> > >> > > > > [1]:
> > >> > >
> > >> >
> > >>
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
> > >> > > > > [2]:
> > >> > >
> > >> >
> > >>
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
> > >> > > > > [3]:
> > >> > >
> > >> >
> > >>
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
> > >> > > > > [4]:
> > >> > >
> > >> >
> > >>
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
> > >> > > > > [5]:
> > >> > >
> > >> >
> > >>
> https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93
> > >> > >
> > >> >
> > >>
> > >
>

Re: [DISCUSS] Classloader change proposals

[Christopher <ct...@apache.org>]

Based on the conversation and direction of this, I think it probably
makes the most sense to just create a new repository under Accumulo
for it to live.

How about 'accumulo-classloaders' for the repo name? If this is okay,
I can create it later today or tomorrow.
This is just a repo name. This is just a place to collaborate on
classloader code without constraining the scope of the repo's contents
too much. The actual package names and artifactId can be different, if
we want.

On Tue, Sep 22, 2020 at 9:44 AM Dave Marion <dm...@gmail.com> wrote:
>
> [1] contains the initial set of changes to the Accumulo code base that
> defines the new context class loader configuration and deprecates the
> existing VFS ClassLoader objects. [2] contains the new
> ReloadingVFSClassLoader that can be used as the system classloader and a
> ClassLoaderFactory implementation for configuring contexts for tables and
> scans. Both build successfully and I plan on doing some local testing next.
> Feedback on the design and the code is welcome.
>
> [1] https://github.com/dlmarion/accumulo/pull/2
> [2] https://github.com/dlmarion/vfs-reloading-classloader
>
> On Fri, Sep 18, 2020 at 12:19 PM Dave Marion <dm...@gmail.com> wrote:
>
> >
> >  I tend to agree with Marc on this. If we need to push out a fix for the
> > new classloader, then we can do it as needed and not have to rely on
> > another group of people to come to consensus on a release. Of course, we
> > could maintain a fork of it in that case, but then what's the point? It
> > appears that VFS does have some recent activity [1], but mostly by one
> > person. I'm thinking that we should create a subproject for it and notify
> > the commons-vfs project of its existence. If they want to copy it and
> > include it in their project, then they can do that.
> >
> > [1] https://gitbox.apache.org/repos/asf?p=commons-vfs.git;a=shortlog
> >
> > On Wed, Sep 16, 2020 at 4:48 PM Christopher <ct...@apache.org> wrote:
> >
> >> Only Marc joined, and we didn't talk about anything that isn't already in
> >> this proposal, except he did mention how difficult it might be to try to
> >> maintain the class loader in commons-vfs2, rather than as our own small
> >> subproject, which is relatively easy.
> >>
> >> On Wed, Sep 16, 2020, 16:11 Dave Marion <dm...@gmail.com> wrote:
> >>
> >> > Did anyone join the call? Any notes?
> >> >
> >> > On Wed, Sep 16, 2020 at 12:44 PM Christopher <ct...@apache.org>
> >> wrote:
> >> >
> >> > > I just want to remind everybody that I'm available in Slack now to
> >> > > discuss this in the ongoing video call I created in the #accumulo
> >> > > room, if you want to join.
> >> > >
> >> > > On Mon, Sep 14, 2020 at 10:41 PM Christopher <ct...@apache.org>
> >> > wrote:
> >> > > >
> >> > > > Also, if anybody is interested in a live video conversation to
> >> discuss
> >> > > > this interactively, I intend to be on Slack on Wednesday afternoon
> >> > > > (EDT) starting around noon.
> >> > > >
> >> > > > On Mon, Sep 14, 2020 at 5:30 PM Christopher <ct...@apache.org>
> >> > wrote:
> >> > > > >
> >> > > > > Hi Accumulo Devs,
> >> > > > >
> >> > > > > Lately, Dave Marion (Apache ID: dlmarion) has been working on
> >> > > > > prototyping some new class loader concepts for Accumulo that he
> >> and I
> >> > > > > have discussed, and I wanted to pitch the idea here for
> >> consideration
> >> > > > > for the project.
> >> > > > >
> >> > > > > # Background:
> >> > > > >
> >> > > > > Accumulo currently has two classloaders that are instantiated at
> >> > > > > startup, and which can be used to bootstrap Accumulo dependencies
> >> (at
> >> > > > > least, those not needed for the classloader code itself). This
> >> allows
> >> > > > > us to use the `general.classpaths`[1] and
> >> > > > > `general.dynamic.classpaths`[2] properties, as well as the
> >> > per-context
> >> > > > > classloaders (`general.vfs.*`[3] and `table.classpath.context`[4])
> >> > for
> >> > > > > things like iterator class isolation. Since 2.0.0, we have
> >> deprecated
> >> > > > > `general.classpaths` and `general.dynamic.classpaths`, the former
> >> > > > > supplanted by the better use of the `CLASSPATH` environment
> >> variable
> >> > > > > (along with much improved scripts in 2.0.0), and the latter being
> >> > > > > replaceable by a user-provided class loader using the built-in
> >> Java
> >> > > > > property, `java.system.class.loader`[5], at their discretion.
> >> > > > >
> >> > > > > # The Problem:
> >> > > > >
> >> > > > > The main problem with the current code is: complexity. Accumulo is
> >> > > > > already complex enough without needing to be in the business of
> >> > > > > developing and supporting complex custom class loading features,
> >> > > > > especially when users have viable alternatives that can be better
> >> > > > > supported by independent, dedicated projects. Furthermore, these
> >> > > > > custom class loaders also have a dependency on commons-vfs2, which
> >> > has
> >> > > > > been the source of numerous problems and bugs that we have needed
> >> to
> >> > > > > deal with, and which affect Accumulo, even though they are not
> >> > > > > necessarily bugs in Accumulo itself. This also brings in a lot of
> >> > > > > optional dependencies that aren't needed by users who don't rely
> >> on
> >> > > > > these features.
> >> > > > >
> >> > > > > # The Requirements:
> >> > > > >
> >> > > > > In spite of these problems, I believe we still want to enable the
> >> use
> >> > > > > cases that our classloaders are currently enabling.
> >> > > > >
> >> > > > > Specifically,
> >> > > > > 1) the ability to have separate contexts for iterator class
> >> isolation
> >> > > > > (A/B testing of iterators, updating iterators in a live system,
> >> > etc.),
> >> > > > > and
> >> > > > > 2) the ability for users to bootstrap their class path from some
> >> > other
> >> > > > > distributed storage than local disk.
> >> > > > >
> >> > > > > # The Proposal:
> >> > > > >
> >> > > > > 1. Create a new reloading vfs class loader, with similar
> >> > functionality
> >> > > > > as our current two-classloaders that do the reloading and provide
> >> vfs
> >> > > > > features, that can be easily used as a system class loader, if the
> >> > > > > user chooses to, and deprecate (for removal in 3.0) the built-in
> >> > > > > implementations. This class loader could not only be used with
> >> > > > > Accumulo, but it could also be used by any other project that
> >> chooses
> >> > > > > to use it, because it will not have much, if any, dependencies
> >> beyond
> >> > > > > commons-vfs2, and will certainly not depend on Accumulo. Creating
> >> > this
> >> > > > > separate class loader provides us a path forward to simplify
> >> Accumulo
> >> > > > > by removing these features from Accumulo directly (the properties
> >> are
> >> > > > > already deprecated), and enabling it to be maintained
> >> independently.
> >> > > > > 2. Create a new class loader factory property in Accumulo, with
> >> > > > > corresponding SPI interface, for users to provide their own
> >> > > > > implementation of a class loader factory, that can map a per-table
> >> > > > > "context" to a ClassLoader of the implementation's choosing.
> >> > > > >
> >> > > > > The result of doing these two things will allow us to more
> >> flexibly
> >> > > > > support user class loading needs, without being directly
> >> responsible
> >> > > > > for class loading implementations inside Accumulo's core code. All
> >> > the
> >> > > > > same functionality that is available today will continue to exist,
> >> > but
> >> > > > > will be configured differently. The resulting code in Accumulo
> >> will
> >> > be
> >> > > > > dramatically simpler, as we would no longer have any complex class
> >> > > > > loading implementations in our code base, and we would no longer
> >> have
> >> > > > > any direct dependency on commons-vfs2, which has been problematic.
> >> > > > > Independent implementations may use commons-vfs2, or something
> >> else,
> >> > > > > but will be more easily testable and maintainable as independent
> >> > > > > projects that are pluggable in Accumulo.
> >> > > > >
> >> > > > > Dave has already been working on prototyping these proposed
> >> changes,
> >> > > > > and it is looking very feasible.
> >> > > > >
> >> > > > > We are now ready to:
> >> > > > > 1. get feedback on the overall proposal, and
> >> > > > > 2. decide on where to maintain the separate class loader.
> >> > > > >
> >> > > > > For where to maintain, the options seem to be: A) try to donate to
> >> > > > > commons-vfs2 OR B) maintain as a new repository,
> >> > > > > accumulo-vfs-classloader.
> >> > > > >
> >> > > > > Note that we have not yet proposed the idea of a user-facing,
> >> > > > > configurable, reloading vfs classloader to the commons-vfs2
> >> > > > > developers. We wanted to get our own community's feedback on this
> >> > > > > first.
> >> > > > >
> >> > > > > Please discuss.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Christopher (in collaboration with Dave)
> >> > > > >
> >> > > > > [1]:
> >> > >
> >> >
> >> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
> >> > > > > [2]:
> >> > >
> >> >
> >> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
> >> > > > > [3]:
> >> > >
> >> >
> >> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
> >> > > > > [4]:
> >> > >
> >> >
> >> https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
> >> > > > > [5]:
> >> > >
> >> >
> >> https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93
> >> > >
> >> >
> >>
> >

Re: [DISCUSS] Classloader change proposals

[Dave Marion <dm...@gmail.com>]

[1] contains the initial set of changes to the Accumulo code base that
defines the new context class loader configuration and deprecates the
existing VFS ClassLoader objects. [2] contains the new
ReloadingVFSClassLoader that can be used as the system classloader and a
ClassLoaderFactory implementation for configuring contexts for tables and
scans. Both build successfully and I plan on doing some local testing next.
Feedback on the design and the code is welcome.

[1] https://github.com/dlmarion/accumulo/pull/2
[2] https://github.com/dlmarion/vfs-reloading-classloader

On Fri, Sep 18, 2020 at 12:19 PM Dave Marion <dm...@gmail.com> wrote:

>
>  I tend to agree with Marc on this. If we need to push out a fix for the
> new classloader, then we can do it as needed and not have to rely on
> another group of people to come to consensus on a release. Of course, we
> could maintain a fork of it in that case, but then what's the point? It
> appears that VFS does have some recent activity [1], but mostly by one
> person. I'm thinking that we should create a subproject for it and notify
> the commons-vfs project of its existence. If they want to copy it and
> include it in their project, then they can do that.
>
> [1] https://gitbox.apache.org/repos/asf?p=commons-vfs.git;a=shortlog
>
> On Wed, Sep 16, 2020 at 4:48 PM Christopher <ct...@apache.org> wrote:
>
>> Only Marc joined, and we didn't talk about anything that isn't already in
>> this proposal, except he did mention how difficult it might be to try to
>> maintain the class loader in commons-vfs2, rather than as our own small
>> subproject, which is relatively easy.
>>
>> On Wed, Sep 16, 2020, 16:11 Dave Marion <dm...@gmail.com> wrote:
>>
>> > Did anyone join the call? Any notes?
>> >
>> > On Wed, Sep 16, 2020 at 12:44 PM Christopher <ct...@apache.org>
>> wrote:
>> >
>> > > I just want to remind everybody that I'm available in Slack now to
>> > > discuss this in the ongoing video call I created in the #accumulo
>> > > room, if you want to join.
>> > >
>> > > On Mon, Sep 14, 2020 at 10:41 PM Christopher <ct...@apache.org>
>> > wrote:
>> > > >
>> > > > Also, if anybody is interested in a live video conversation to
>> discuss
>> > > > this interactively, I intend to be on Slack on Wednesday afternoon
>> > > > (EDT) starting around noon.
>> > > >
>> > > > On Mon, Sep 14, 2020 at 5:30 PM Christopher <ct...@apache.org>
>> > wrote:
>> > > > >
>> > > > > Hi Accumulo Devs,
>> > > > >
>> > > > > Lately, Dave Marion (Apache ID: dlmarion) has been working on
>> > > > > prototyping some new class loader concepts for Accumulo that he
>> and I
>> > > > > have discussed, and I wanted to pitch the idea here for
>> consideration
>> > > > > for the project.
>> > > > >
>> > > > > # Background:
>> > > > >
>> > > > > Accumulo currently has two classloaders that are instantiated at
>> > > > > startup, and which can be used to bootstrap Accumulo dependencies
>> (at
>> > > > > least, those not needed for the classloader code itself). This
>> allows
>> > > > > us to use the `general.classpaths`[1] and
>> > > > > `general.dynamic.classpaths`[2] properties, as well as the
>> > per-context
>> > > > > classloaders (`general.vfs.*`[3] and `table.classpath.context`[4])
>> > for
>> > > > > things like iterator class isolation. Since 2.0.0, we have
>> deprecated
>> > > > > `general.classpaths` and `general.dynamic.classpaths`, the former
>> > > > > supplanted by the better use of the `CLASSPATH` environment
>> variable
>> > > > > (along with much improved scripts in 2.0.0), and the latter being
>> > > > > replaceable by a user-provided class loader using the built-in
>> Java
>> > > > > property, `java.system.class.loader`[5], at their discretion.
>> > > > >
>> > > > > # The Problem:
>> > > > >
>> > > > > The main problem with the current code is: complexity. Accumulo is
>> > > > > already complex enough without needing to be in the business of
>> > > > > developing and supporting complex custom class loading features,
>> > > > > especially when users have viable alternatives that can be better
>> > > > > supported by independent, dedicated projects. Furthermore, these
>> > > > > custom class loaders also have a dependency on commons-vfs2, which
>> > has
>> > > > > been the source of numerous problems and bugs that we have needed
>> to
>> > > > > deal with, and which affect Accumulo, even though they are not
>> > > > > necessarily bugs in Accumulo itself. This also brings in a lot of
>> > > > > optional dependencies that aren't needed by users who don't rely
>> on
>> > > > > these features.
>> > > > >
>> > > > > # The Requirements:
>> > > > >
>> > > > > In spite of these problems, I believe we still want to enable the
>> use
>> > > > > cases that our classloaders are currently enabling.
>> > > > >
>> > > > > Specifically,
>> > > > > 1) the ability to have separate contexts for iterator class
>> isolation
>> > > > > (A/B testing of iterators, updating iterators in a live system,
>> > etc.),
>> > > > > and
>> > > > > 2) the ability for users to bootstrap their class path from some
>> > other
>> > > > > distributed storage than local disk.
>> > > > >
>> > > > > # The Proposal:
>> > > > >
>> > > > > 1. Create a new reloading vfs class loader, with similar
>> > functionality
>> > > > > as our current two-classloaders that do the reloading and provide
>> vfs
>> > > > > features, that can be easily used as a system class loader, if the
>> > > > > user chooses to, and deprecate (for removal in 3.0) the built-in
>> > > > > implementations. This class loader could not only be used with
>> > > > > Accumulo, but it could also be used by any other project that
>> chooses
>> > > > > to use it, because it will not have much, if any, dependencies
>> beyond
>> > > > > commons-vfs2, and will certainly not depend on Accumulo. Creating
>> > this
>> > > > > separate class loader provides us a path forward to simplify
>> Accumulo
>> > > > > by removing these features from Accumulo directly (the properties
>> are
>> > > > > already deprecated), and enabling it to be maintained
>> independently.
>> > > > > 2. Create a new class loader factory property in Accumulo, with
>> > > > > corresponding SPI interface, for users to provide their own
>> > > > > implementation of a class loader factory, that can map a per-table
>> > > > > "context" to a ClassLoader of the implementation's choosing.
>> > > > >
>> > > > > The result of doing these two things will allow us to more
>> flexibly
>> > > > > support user class loading needs, without being directly
>> responsible
>> > > > > for class loading implementations inside Accumulo's core code. All
>> > the
>> > > > > same functionality that is available today will continue to exist,
>> > but
>> > > > > will be configured differently. The resulting code in Accumulo
>> will
>> > be
>> > > > > dramatically simpler, as we would no longer have any complex class
>> > > > > loading implementations in our code base, and we would no longer
>> have
>> > > > > any direct dependency on commons-vfs2, which has been problematic.
>> > > > > Independent implementations may use commons-vfs2, or something
>> else,
>> > > > > but will be more easily testable and maintainable as independent
>> > > > > projects that are pluggable in Accumulo.
>> > > > >
>> > > > > Dave has already been working on prototyping these proposed
>> changes,
>> > > > > and it is looking very feasible.
>> > > > >
>> > > > > We are now ready to:
>> > > > > 1. get feedback on the overall proposal, and
>> > > > > 2. decide on where to maintain the separate class loader.
>> > > > >
>> > > > > For where to maintain, the options seem to be: A) try to donate to
>> > > > > commons-vfs2 OR B) maintain as a new repository,
>> > > > > accumulo-vfs-classloader.
>> > > > >
>> > > > > Note that we have not yet proposed the idea of a user-facing,
>> > > > > configurable, reloading vfs classloader to the commons-vfs2
>> > > > > developers. We wanted to get our own community's feedback on this
>> > > > > first.
>> > > > >
>> > > > > Please discuss.
>> > > > >
>> > > > > Thanks,
>> > > > > Christopher (in collaboration with Dave)
>> > > > >
>> > > > > [1]:
>> > >
>> >
>> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
>> > > > > [2]:
>> > >
>> >
>> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
>> > > > > [3]:
>> > >
>> >
>> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
>> > > > > [4]:
>> > >
>> >
>> https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
>> > > > > [5]:
>> > >
>> >
>> https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93
>> > >
>> >
>>
>

Re: [DISCUSS] Classloader change proposals

[Dave Marion <dm...@gmail.com>]

 I tend to agree with Marc on this. If we need to push out a fix for the
new classloader, then we can do it as needed and not have to rely on
another group of people to come to consensus on a release. Of course, we
could maintain a fork of it in that case, but then what's the point? It
appears that VFS does have some recent activity [1], but mostly by one
person. I'm thinking that we should create a subproject for it and notify
the commons-vfs project of its existence. If they want to copy it and
include it in their project, then they can do that.

[1] https://gitbox.apache.org/repos/asf?p=commons-vfs.git;a=shortlog

On Wed, Sep 16, 2020 at 4:48 PM Christopher <ct...@apache.org> wrote:

> Only Marc joined, and we didn't talk about anything that isn't already in
> this proposal, except he did mention how difficult it might be to try to
> maintain the class loader in commons-vfs2, rather than as our own small
> subproject, which is relatively easy.
>
> On Wed, Sep 16, 2020, 16:11 Dave Marion <dm...@gmail.com> wrote:
>
> > Did anyone join the call? Any notes?
> >
> > On Wed, Sep 16, 2020 at 12:44 PM Christopher <ct...@apache.org>
> wrote:
> >
> > > I just want to remind everybody that I'm available in Slack now to
> > > discuss this in the ongoing video call I created in the #accumulo
> > > room, if you want to join.
> > >
> > > On Mon, Sep 14, 2020 at 10:41 PM Christopher <ct...@apache.org>
> > wrote:
> > > >
> > > > Also, if anybody is interested in a live video conversation to
> discuss
> > > > this interactively, I intend to be on Slack on Wednesday afternoon
> > > > (EDT) starting around noon.
> > > >
> > > > On Mon, Sep 14, 2020 at 5:30 PM Christopher <ct...@apache.org>
> > wrote:
> > > > >
> > > > > Hi Accumulo Devs,
> > > > >
> > > > > Lately, Dave Marion (Apache ID: dlmarion) has been working on
> > > > > prototyping some new class loader concepts for Accumulo that he
> and I
> > > > > have discussed, and I wanted to pitch the idea here for
> consideration
> > > > > for the project.
> > > > >
> > > > > # Background:
> > > > >
> > > > > Accumulo currently has two classloaders that are instantiated at
> > > > > startup, and which can be used to bootstrap Accumulo dependencies
> (at
> > > > > least, those not needed for the classloader code itself). This
> allows
> > > > > us to use the `general.classpaths`[1] and
> > > > > `general.dynamic.classpaths`[2] properties, as well as the
> > per-context
> > > > > classloaders (`general.vfs.*`[3] and `table.classpath.context`[4])
> > for
> > > > > things like iterator class isolation. Since 2.0.0, we have
> deprecated
> > > > > `general.classpaths` and `general.dynamic.classpaths`, the former
> > > > > supplanted by the better use of the `CLASSPATH` environment
> variable
> > > > > (along with much improved scripts in 2.0.0), and the latter being
> > > > > replaceable by a user-provided class loader using the built-in Java
> > > > > property, `java.system.class.loader`[5], at their discretion.
> > > > >
> > > > > # The Problem:
> > > > >
> > > > > The main problem with the current code is: complexity. Accumulo is
> > > > > already complex enough without needing to be in the business of
> > > > > developing and supporting complex custom class loading features,
> > > > > especially when users have viable alternatives that can be better
> > > > > supported by independent, dedicated projects. Furthermore, these
> > > > > custom class loaders also have a dependency on commons-vfs2, which
> > has
> > > > > been the source of numerous problems and bugs that we have needed
> to
> > > > > deal with, and which affect Accumulo, even though they are not
> > > > > necessarily bugs in Accumulo itself. This also brings in a lot of
> > > > > optional dependencies that aren't needed by users who don't rely on
> > > > > these features.
> > > > >
> > > > > # The Requirements:
> > > > >
> > > > > In spite of these problems, I believe we still want to enable the
> use
> > > > > cases that our classloaders are currently enabling.
> > > > >
> > > > > Specifically,
> > > > > 1) the ability to have separate contexts for iterator class
> isolation
> > > > > (A/B testing of iterators, updating iterators in a live system,
> > etc.),
> > > > > and
> > > > > 2) the ability for users to bootstrap their class path from some
> > other
> > > > > distributed storage than local disk.
> > > > >
> > > > > # The Proposal:
> > > > >
> > > > > 1. Create a new reloading vfs class loader, with similar
> > functionality
> > > > > as our current two-classloaders that do the reloading and provide
> vfs
> > > > > features, that can be easily used as a system class loader, if the
> > > > > user chooses to, and deprecate (for removal in 3.0) the built-in
> > > > > implementations. This class loader could not only be used with
> > > > > Accumulo, but it could also be used by any other project that
> chooses
> > > > > to use it, because it will not have much, if any, dependencies
> beyond
> > > > > commons-vfs2, and will certainly not depend on Accumulo. Creating
> > this
> > > > > separate class loader provides us a path forward to simplify
> Accumulo
> > > > > by removing these features from Accumulo directly (the properties
> are
> > > > > already deprecated), and enabling it to be maintained
> independently.
> > > > > 2. Create a new class loader factory property in Accumulo, with
> > > > > corresponding SPI interface, for users to provide their own
> > > > > implementation of a class loader factory, that can map a per-table
> > > > > "context" to a ClassLoader of the implementation's choosing.
> > > > >
> > > > > The result of doing these two things will allow us to more flexibly
> > > > > support user class loading needs, without being directly
> responsible
> > > > > for class loading implementations inside Accumulo's core code. All
> > the
> > > > > same functionality that is available today will continue to exist,
> > but
> > > > > will be configured differently. The resulting code in Accumulo will
> > be
> > > > > dramatically simpler, as we would no longer have any complex class
> > > > > loading implementations in our code base, and we would no longer
> have
> > > > > any direct dependency on commons-vfs2, which has been problematic.
> > > > > Independent implementations may use commons-vfs2, or something
> else,
> > > > > but will be more easily testable and maintainable as independent
> > > > > projects that are pluggable in Accumulo.
> > > > >
> > > > > Dave has already been working on prototyping these proposed
> changes,
> > > > > and it is looking very feasible.
> > > > >
> > > > > We are now ready to:
> > > > > 1. get feedback on the overall proposal, and
> > > > > 2. decide on where to maintain the separate class loader.
> > > > >
> > > > > For where to maintain, the options seem to be: A) try to donate to
> > > > > commons-vfs2 OR B) maintain as a new repository,
> > > > > accumulo-vfs-classloader.
> > > > >
> > > > > Note that we have not yet proposed the idea of a user-facing,
> > > > > configurable, reloading vfs classloader to the commons-vfs2
> > > > > developers. We wanted to get our own community's feedback on this
> > > > > first.
> > > > >
> > > > > Please discuss.
> > > > >
> > > > > Thanks,
> > > > > Christopher (in collaboration with Dave)
> > > > >
> > > > > [1]:
> > >
> >
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
> > > > > [2]:
> > >
> >
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
> > > > > [3]:
> > >
> >
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
> > > > > [4]:
> > >
> >
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
> > > > > [5]:
> > >
> >
> https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93
> > >
> >
>

Re: [DISCUSS] Classloader change proposals

[Christopher <ct...@apache.org>]

Only Marc joined, and we didn't talk about anything that isn't already in
this proposal, except he did mention how difficult it might be to try to
maintain the class loader in commons-vfs2, rather than as our own small
subproject, which is relatively easy.

On Wed, Sep 16, 2020, 16:11 Dave Marion <dm...@gmail.com> wrote:

> Did anyone join the call? Any notes?
>
> On Wed, Sep 16, 2020 at 12:44 PM Christopher <ct...@apache.org> wrote:
>
> > I just want to remind everybody that I'm available in Slack now to
> > discuss this in the ongoing video call I created in the #accumulo
> > room, if you want to join.
> >
> > On Mon, Sep 14, 2020 at 10:41 PM Christopher <ct...@apache.org>
> wrote:
> > >
> > > Also, if anybody is interested in a live video conversation to discuss
> > > this interactively, I intend to be on Slack on Wednesday afternoon
> > > (EDT) starting around noon.
> > >
> > > On Mon, Sep 14, 2020 at 5:30 PM Christopher <ct...@apache.org>
> wrote:
> > > >
> > > > Hi Accumulo Devs,
> > > >
> > > > Lately, Dave Marion (Apache ID: dlmarion) has been working on
> > > > prototyping some new class loader concepts for Accumulo that he and I
> > > > have discussed, and I wanted to pitch the idea here for consideration
> > > > for the project.
> > > >
> > > > # Background:
> > > >
> > > > Accumulo currently has two classloaders that are instantiated at
> > > > startup, and which can be used to bootstrap Accumulo dependencies (at
> > > > least, those not needed for the classloader code itself). This allows
> > > > us to use the `general.classpaths`[1] and
> > > > `general.dynamic.classpaths`[2] properties, as well as the
> per-context
> > > > classloaders (`general.vfs.*`[3] and `table.classpath.context`[4])
> for
> > > > things like iterator class isolation. Since 2.0.0, we have deprecated
> > > > `general.classpaths` and `general.dynamic.classpaths`, the former
> > > > supplanted by the better use of the `CLASSPATH` environment variable
> > > > (along with much improved scripts in 2.0.0), and the latter being
> > > > replaceable by a user-provided class loader using the built-in Java
> > > > property, `java.system.class.loader`[5], at their discretion.
> > > >
> > > > # The Problem:
> > > >
> > > > The main problem with the current code is: complexity. Accumulo is
> > > > already complex enough without needing to be in the business of
> > > > developing and supporting complex custom class loading features,
> > > > especially when users have viable alternatives that can be better
> > > > supported by independent, dedicated projects. Furthermore, these
> > > > custom class loaders also have a dependency on commons-vfs2, which
> has
> > > > been the source of numerous problems and bugs that we have needed to
> > > > deal with, and which affect Accumulo, even though they are not
> > > > necessarily bugs in Accumulo itself. This also brings in a lot of
> > > > optional dependencies that aren't needed by users who don't rely on
> > > > these features.
> > > >
> > > > # The Requirements:
> > > >
> > > > In spite of these problems, I believe we still want to enable the use
> > > > cases that our classloaders are currently enabling.
> > > >
> > > > Specifically,
> > > > 1) the ability to have separate contexts for iterator class isolation
> > > > (A/B testing of iterators, updating iterators in a live system,
> etc.),
> > > > and
> > > > 2) the ability for users to bootstrap their class path from some
> other
> > > > distributed storage than local disk.
> > > >
> > > > # The Proposal:
> > > >
> > > > 1. Create a new reloading vfs class loader, with similar
> functionality
> > > > as our current two-classloaders that do the reloading and provide vfs
> > > > features, that can be easily used as a system class loader, if the
> > > > user chooses to, and deprecate (for removal in 3.0) the built-in
> > > > implementations. This class loader could not only be used with
> > > > Accumulo, but it could also be used by any other project that chooses
> > > > to use it, because it will not have much, if any, dependencies beyond
> > > > commons-vfs2, and will certainly not depend on Accumulo. Creating
> this
> > > > separate class loader provides us a path forward to simplify Accumulo
> > > > by removing these features from Accumulo directly (the properties are
> > > > already deprecated), and enabling it to be maintained independently.
> > > > 2. Create a new class loader factory property in Accumulo, with
> > > > corresponding SPI interface, for users to provide their own
> > > > implementation of a class loader factory, that can map a per-table
> > > > "context" to a ClassLoader of the implementation's choosing.
> > > >
> > > > The result of doing these two things will allow us to more flexibly
> > > > support user class loading needs, without being directly responsible
> > > > for class loading implementations inside Accumulo's core code. All
> the
> > > > same functionality that is available today will continue to exist,
> but
> > > > will be configured differently. The resulting code in Accumulo will
> be
> > > > dramatically simpler, as we would no longer have any complex class
> > > > loading implementations in our code base, and we would no longer have
> > > > any direct dependency on commons-vfs2, which has been problematic.
> > > > Independent implementations may use commons-vfs2, or something else,
> > > > but will be more easily testable and maintainable as independent
> > > > projects that are pluggable in Accumulo.
> > > >
> > > > Dave has already been working on prototyping these proposed changes,
> > > > and it is looking very feasible.
> > > >
> > > > We are now ready to:
> > > > 1. get feedback on the overall proposal, and
> > > > 2. decide on where to maintain the separate class loader.
> > > >
> > > > For where to maintain, the options seem to be: A) try to donate to
> > > > commons-vfs2 OR B) maintain as a new repository,
> > > > accumulo-vfs-classloader.
> > > >
> > > > Note that we have not yet proposed the idea of a user-facing,
> > > > configurable, reloading vfs classloader to the commons-vfs2
> > > > developers. We wanted to get our own community's feedback on this
> > > > first.
> > > >
> > > > Please discuss.
> > > >
> > > > Thanks,
> > > > Christopher (in collaboration with Dave)
> > > >
> > > > [1]:
> >
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
> > > > [2]:
> >
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
> > > > [3]:
> >
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
> > > > [4]:
> >
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
> > > > [5]:
> >
> https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93
> >
>

Re: [DISCUSS] Classloader change proposals

[Dave Marion <dm...@gmail.com>]

Did anyone join the call? Any notes?

On Wed, Sep 16, 2020 at 12:44 PM Christopher <ct...@apache.org> wrote:

> I just want to remind everybody that I'm available in Slack now to
> discuss this in the ongoing video call I created in the #accumulo
> room, if you want to join.
>
> On Mon, Sep 14, 2020 at 10:41 PM Christopher <ct...@apache.org> wrote:
> >
> > Also, if anybody is interested in a live video conversation to discuss
> > this interactively, I intend to be on Slack on Wednesday afternoon
> > (EDT) starting around noon.
> >
> > On Mon, Sep 14, 2020 at 5:30 PM Christopher <ct...@apache.org> wrote:
> > >
> > > Hi Accumulo Devs,
> > >
> > > Lately, Dave Marion (Apache ID: dlmarion) has been working on
> > > prototyping some new class loader concepts for Accumulo that he and I
> > > have discussed, and I wanted to pitch the idea here for consideration
> > > for the project.
> > >
> > > # Background:
> > >
> > > Accumulo currently has two classloaders that are instantiated at
> > > startup, and which can be used to bootstrap Accumulo dependencies (at
> > > least, those not needed for the classloader code itself). This allows
> > > us to use the `general.classpaths`[1] and
> > > `general.dynamic.classpaths`[2] properties, as well as the per-context
> > > classloaders (`general.vfs.*`[3] and `table.classpath.context`[4]) for
> > > things like iterator class isolation. Since 2.0.0, we have deprecated
> > > `general.classpaths` and `general.dynamic.classpaths`, the former
> > > supplanted by the better use of the `CLASSPATH` environment variable
> > > (along with much improved scripts in 2.0.0), and the latter being
> > > replaceable by a user-provided class loader using the built-in Java
> > > property, `java.system.class.loader`[5], at their discretion.
> > >
> > > # The Problem:
> > >
> > > The main problem with the current code is: complexity. Accumulo is
> > > already complex enough without needing to be in the business of
> > > developing and supporting complex custom class loading features,
> > > especially when users have viable alternatives that can be better
> > > supported by independent, dedicated projects. Furthermore, these
> > > custom class loaders also have a dependency on commons-vfs2, which has
> > > been the source of numerous problems and bugs that we have needed to
> > > deal with, and which affect Accumulo, even though they are not
> > > necessarily bugs in Accumulo itself. This also brings in a lot of
> > > optional dependencies that aren't needed by users who don't rely on
> > > these features.
> > >
> > > # The Requirements:
> > >
> > > In spite of these problems, I believe we still want to enable the use
> > > cases that our classloaders are currently enabling.
> > >
> > > Specifically,
> > > 1) the ability to have separate contexts for iterator class isolation
> > > (A/B testing of iterators, updating iterators in a live system, etc.),
> > > and
> > > 2) the ability for users to bootstrap their class path from some other
> > > distributed storage than local disk.
> > >
> > > # The Proposal:
> > >
> > > 1. Create a new reloading vfs class loader, with similar functionality
> > > as our current two-classloaders that do the reloading and provide vfs
> > > features, that can be easily used as a system class loader, if the
> > > user chooses to, and deprecate (for removal in 3.0) the built-in
> > > implementations. This class loader could not only be used with
> > > Accumulo, but it could also be used by any other project that chooses
> > > to use it, because it will not have much, if any, dependencies beyond
> > > commons-vfs2, and will certainly not depend on Accumulo. Creating this
> > > separate class loader provides us a path forward to simplify Accumulo
> > > by removing these features from Accumulo directly (the properties are
> > > already deprecated), and enabling it to be maintained independently.
> > > 2. Create a new class loader factory property in Accumulo, with
> > > corresponding SPI interface, for users to provide their own
> > > implementation of a class loader factory, that can map a per-table
> > > "context" to a ClassLoader of the implementation's choosing.
> > >
> > > The result of doing these two things will allow us to more flexibly
> > > support user class loading needs, without being directly responsible
> > > for class loading implementations inside Accumulo's core code. All the
> > > same functionality that is available today will continue to exist, but
> > > will be configured differently. The resulting code in Accumulo will be
> > > dramatically simpler, as we would no longer have any complex class
> > > loading implementations in our code base, and we would no longer have
> > > any direct dependency on commons-vfs2, which has been problematic.
> > > Independent implementations may use commons-vfs2, or something else,
> > > but will be more easily testable and maintainable as independent
> > > projects that are pluggable in Accumulo.
> > >
> > > Dave has already been working on prototyping these proposed changes,
> > > and it is looking very feasible.
> > >
> > > We are now ready to:
> > > 1. get feedback on the overall proposal, and
> > > 2. decide on where to maintain the separate class loader.
> > >
> > > For where to maintain, the options seem to be: A) try to donate to
> > > commons-vfs2 OR B) maintain as a new repository,
> > > accumulo-vfs-classloader.
> > >
> > > Note that we have not yet proposed the idea of a user-facing,
> > > configurable, reloading vfs classloader to the commons-vfs2
> > > developers. We wanted to get our own community's feedback on this
> > > first.
> > >
> > > Please discuss.
> > >
> > > Thanks,
> > > Christopher (in collaboration with Dave)
> > >
> > > [1]:
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
> > > [2]:
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
> > > [3]:
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
> > > [4]:
> https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
> > > [5]:
> https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93
>

Re: [DISCUSS] Classloader change proposals

[Christopher <ct...@apache.org>]

I just want to remind everybody that I'm available in Slack now to
discuss this in the ongoing video call I created in the #accumulo
room, if you want to join.

On Mon, Sep 14, 2020 at 10:41 PM Christopher <ct...@apache.org> wrote:
>
> Also, if anybody is interested in a live video conversation to discuss
> this interactively, I intend to be on Slack on Wednesday afternoon
> (EDT) starting around noon.
>
> On Mon, Sep 14, 2020 at 5:30 PM Christopher <ct...@apache.org> wrote:
> >
> > Hi Accumulo Devs,
> >
> > Lately, Dave Marion (Apache ID: dlmarion) has been working on
> > prototyping some new class loader concepts for Accumulo that he and I
> > have discussed, and I wanted to pitch the idea here for consideration
> > for the project.
> >
> > # Background:
> >
> > Accumulo currently has two classloaders that are instantiated at
> > startup, and which can be used to bootstrap Accumulo dependencies (at
> > least, those not needed for the classloader code itself). This allows
> > us to use the `general.classpaths`[1] and
> > `general.dynamic.classpaths`[2] properties, as well as the per-context
> > classloaders (`general.vfs.*`[3] and `table.classpath.context`[4]) for
> > things like iterator class isolation. Since 2.0.0, we have deprecated
> > `general.classpaths` and `general.dynamic.classpaths`, the former
> > supplanted by the better use of the `CLASSPATH` environment variable
> > (along with much improved scripts in 2.0.0), and the latter being
> > replaceable by a user-provided class loader using the built-in Java
> > property, `java.system.class.loader`[5], at their discretion.
> >
> > # The Problem:
> >
> > The main problem with the current code is: complexity. Accumulo is
> > already complex enough without needing to be in the business of
> > developing and supporting complex custom class loading features,
> > especially when users have viable alternatives that can be better
> > supported by independent, dedicated projects. Furthermore, these
> > custom class loaders also have a dependency on commons-vfs2, which has
> > been the source of numerous problems and bugs that we have needed to
> > deal with, and which affect Accumulo, even though they are not
> > necessarily bugs in Accumulo itself. This also brings in a lot of
> > optional dependencies that aren't needed by users who don't rely on
> > these features.
> >
> > # The Requirements:
> >
> > In spite of these problems, I believe we still want to enable the use
> > cases that our classloaders are currently enabling.
> >
> > Specifically,
> > 1) the ability to have separate contexts for iterator class isolation
> > (A/B testing of iterators, updating iterators in a live system, etc.),
> > and
> > 2) the ability for users to bootstrap their class path from some other
> > distributed storage than local disk.
> >
> > # The Proposal:
> >
> > 1. Create a new reloading vfs class loader, with similar functionality
> > as our current two-classloaders that do the reloading and provide vfs
> > features, that can be easily used as a system class loader, if the
> > user chooses to, and deprecate (for removal in 3.0) the built-in
> > implementations. This class loader could not only be used with
> > Accumulo, but it could also be used by any other project that chooses
> > to use it, because it will not have much, if any, dependencies beyond
> > commons-vfs2, and will certainly not depend on Accumulo. Creating this
> > separate class loader provides us a path forward to simplify Accumulo
> > by removing these features from Accumulo directly (the properties are
> > already deprecated), and enabling it to be maintained independently.
> > 2. Create a new class loader factory property in Accumulo, with
> > corresponding SPI interface, for users to provide their own
> > implementation of a class loader factory, that can map a per-table
> > "context" to a ClassLoader of the implementation's choosing.
> >
> > The result of doing these two things will allow us to more flexibly
> > support user class loading needs, without being directly responsible
> > for class loading implementations inside Accumulo's core code. All the
> > same functionality that is available today will continue to exist, but
> > will be configured differently. The resulting code in Accumulo will be
> > dramatically simpler, as we would no longer have any complex class
> > loading implementations in our code base, and we would no longer have
> > any direct dependency on commons-vfs2, which has been problematic.
> > Independent implementations may use commons-vfs2, or something else,
> > but will be more easily testable and maintainable as independent
> > projects that are pluggable in Accumulo.
> >
> > Dave has already been working on prototyping these proposed changes,
> > and it is looking very feasible.
> >
> > We are now ready to:
> > 1. get feedback on the overall proposal, and
> > 2. decide on where to maintain the separate class loader.
> >
> > For where to maintain, the options seem to be: A) try to donate to
> > commons-vfs2 OR B) maintain as a new repository,
> > accumulo-vfs-classloader.
> >
> > Note that we have not yet proposed the idea of a user-facing,
> > configurable, reloading vfs classloader to the commons-vfs2
> > developers. We wanted to get our own community's feedback on this
> > first.
> >
> > Please discuss.
> >
> > Thanks,
> > Christopher (in collaboration with Dave)
> >
> > [1]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
> > [2]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
> > [3]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
> > [4]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
> > [5]: https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93

Re: [DISCUSS] Classloader change proposals

[Christopher <ct...@apache.org>]

Also, if anybody is interested in a live video conversation to discuss
this interactively, I intend to be on Slack on Wednesday afternoon
(EDT) starting around noon.

On Mon, Sep 14, 2020 at 5:30 PM Christopher <ct...@apache.org> wrote:
>
> Hi Accumulo Devs,
>
> Lately, Dave Marion (Apache ID: dlmarion) has been working on
> prototyping some new class loader concepts for Accumulo that he and I
> have discussed, and I wanted to pitch the idea here for consideration
> for the project.
>
> # Background:
>
> Accumulo currently has two classloaders that are instantiated at
> startup, and which can be used to bootstrap Accumulo dependencies (at
> least, those not needed for the classloader code itself). This allows
> us to use the `general.classpaths`[1] and
> `general.dynamic.classpaths`[2] properties, as well as the per-context
> classloaders (`general.vfs.*`[3] and `table.classpath.context`[4]) for
> things like iterator class isolation. Since 2.0.0, we have deprecated
> `general.classpaths` and `general.dynamic.classpaths`, the former
> supplanted by the better use of the `CLASSPATH` environment variable
> (along with much improved scripts in 2.0.0), and the latter being
> replaceable by a user-provided class loader using the built-in Java
> property, `java.system.class.loader`[5], at their discretion.
>
> # The Problem:
>
> The main problem with the current code is: complexity. Accumulo is
> already complex enough without needing to be in the business of
> developing and supporting complex custom class loading features,
> especially when users have viable alternatives that can be better
> supported by independent, dedicated projects. Furthermore, these
> custom class loaders also have a dependency on commons-vfs2, which has
> been the source of numerous problems and bugs that we have needed to
> deal with, and which affect Accumulo, even though they are not
> necessarily bugs in Accumulo itself. This also brings in a lot of
> optional dependencies that aren't needed by users who don't rely on
> these features.
>
> # The Requirements:
>
> In spite of these problems, I believe we still want to enable the use
> cases that our classloaders are currently enabling.
>
> Specifically,
> 1) the ability to have separate contexts for iterator class isolation
> (A/B testing of iterators, updating iterators in a live system, etc.),
> and
> 2) the ability for users to bootstrap their class path from some other
> distributed storage than local disk.
>
> # The Proposal:
>
> 1. Create a new reloading vfs class loader, with similar functionality
> as our current two-classloaders that do the reloading and provide vfs
> features, that can be easily used as a system class loader, if the
> user chooses to, and deprecate (for removal in 3.0) the built-in
> implementations. This class loader could not only be used with
> Accumulo, but it could also be used by any other project that chooses
> to use it, because it will not have much, if any, dependencies beyond
> commons-vfs2, and will certainly not depend on Accumulo. Creating this
> separate class loader provides us a path forward to simplify Accumulo
> by removing these features from Accumulo directly (the properties are
> already deprecated), and enabling it to be maintained independently.
> 2. Create a new class loader factory property in Accumulo, with
> corresponding SPI interface, for users to provide their own
> implementation of a class loader factory, that can map a per-table
> "context" to a ClassLoader of the implementation's choosing.
>
> The result of doing these two things will allow us to more flexibly
> support user class loading needs, without being directly responsible
> for class loading implementations inside Accumulo's core code. All the
> same functionality that is available today will continue to exist, but
> will be configured differently. The resulting code in Accumulo will be
> dramatically simpler, as we would no longer have any complex class
> loading implementations in our code base, and we would no longer have
> any direct dependency on commons-vfs2, which has been problematic.
> Independent implementations may use commons-vfs2, or something else,
> but will be more easily testable and maintainable as independent
> projects that are pluggable in Accumulo.
>
> Dave has already been working on prototyping these proposed changes,
> and it is looking very feasible.
>
> We are now ready to:
> 1. get feedback on the overall proposal, and
> 2. decide on where to maintain the separate class loader.
>
> For where to maintain, the options seem to be: A) try to donate to
> commons-vfs2 OR B) maintain as a new repository,
> accumulo-vfs-classloader.
>
> Note that we have not yet proposed the idea of a user-facing,
> configurable, reloading vfs classloader to the commons-vfs2
> developers. We wanted to get our own community's feedback on this
> first.
>
> Please discuss.
>
> Thanks,
> Christopher (in collaboration with Dave)
>
> [1]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
> [2]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
> [3]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
> [4]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
> [5]: https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93

RE: [DISCUSS] Classloader change proposals

["Owens, Mark" <jm...@evoforge.org>]

I don't really know enough about classloaders to contribute much to the discussion. But I am in definitely in favor of changes that seek to simplify the core codebase. If you and Dave feel the update is feasible I would support the update.

-----Original Message-----
From: Christopher <ct...@apache.org> 
Sent: Monday, September 14, 2020 5:30 PM
To: accumulo-dev <de...@accumulo.apache.org>
Subject: [DISCUSS] Classloader change proposals

Hi Accumulo Devs,

Lately, Dave Marion (Apache ID: dlmarion) has been working on prototyping some new class loader concepts for Accumulo that he and I have discussed, and I wanted to pitch the idea here for consideration for the project.

# Background:

Accumulo currently has two classloaders that are instantiated at startup, and which can be used to bootstrap Accumulo dependencies (at least, those not needed for the classloader code itself). This allows us to use the `general.classpaths`[1] and `general.dynamic.classpaths`[2] properties, as well as the per-context classloaders (`general.vfs.*`[3] and `table.classpath.context`[4]) for things like iterator class isolation. Since 2.0.0, we have deprecated `general.classpaths` and `general.dynamic.classpaths`, the former supplanted by the better use of the `CLASSPATH` environment variable (along with much improved scripts in 2.0.0), and the latter being replaceable by a user-provided class loader using the built-in Java property, `java.system.class.loader`[5], at their discretion.

# The Problem:

The main problem with the current code is: complexity. Accumulo is already complex enough without needing to be in the business of developing and supporting complex custom class loading features, especially when users have viable alternatives that can be better supported by independent, dedicated projects. Furthermore, these custom class loaders also have a dependency on commons-vfs2, which has been the source of numerous problems and bugs that we have needed to deal with, and which affect Accumulo, even though they are not necessarily bugs in Accumulo itself. This also brings in a lot of optional dependencies that aren't needed by users who don't rely on these features.

# The Requirements:

In spite of these problems, I believe we still want to enable the use cases that our classloaders are currently enabling.

Specifically,
1) the ability to have separate contexts for iterator class isolation (A/B testing of iterators, updating iterators in a live system, etc.), and
2) the ability for users to bootstrap their class path from some other distributed storage than local disk.

# The Proposal:

1. Create a new reloading vfs class loader, with similar functionality as our current two-classloaders that do the reloading and provide vfs features, that can be easily used as a system class loader, if the user chooses to, and deprecate (for removal in 3.0) the built-in implementations. This class loader could not only be used with Accumulo, but it could also be used by any other project that chooses to use it, because it will not have much, if any, dependencies beyond commons-vfs2, and will certainly not depend on Accumulo. Creating this separate class loader provides us a path forward to simplify Accumulo by removing these features from Accumulo directly (the properties are already deprecated), and enabling it to be maintained independently.
2. Create a new class loader factory property in Accumulo, with corresponding SPI interface, for users to provide their own implementation of a class loader factory, that can map a per-table "context" to a ClassLoader of the implementation's choosing.

The result of doing these two things will allow us to more flexibly support user class loading needs, without being directly responsible for class loading implementations inside Accumulo's core code. All the same functionality that is available today will continue to exist, but will be configured differently. The resulting code in Accumulo will be dramatically simpler, as we would no longer have any complex class loading implementations in our code base, and we would no longer have any direct dependency on commons-vfs2, which has been problematic.
Independent implementations may use commons-vfs2, or something else, but will be more easily testable and maintainable as independent projects that are pluggable in Accumulo.

Dave has already been working on prototyping these proposed changes, and it is looking very feasible.

We are now ready to:
1. get feedback on the overall proposal, and 2. decide on where to maintain the separate class loader.

For where to maintain, the options seem to be: A) try to donate to
commons-vfs2 OR B) maintain as a new repository, accumulo-vfs-classloader.

Note that we have not yet proposed the idea of a user-facing, configurable, reloading vfs classloader to the commons-vfs2 developers. We wanted to get our own community's feedback on this first.

Please discuss.

Thanks,
Christopher (in collaboration with Dave)

[1]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_classpaths
[2]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_dynamic_classpaths
[3]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#general_vfs_context_classpath_prefix
[4]: https://accumulo.apache.org/docs/2.x/configuration/server-properties#table_classpath_context
[5]: https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#getSystemClassLoader%E2%80%93