[DISCUSS] Plugin classpaths

[Matt Casters <ma...@neo4j.com.INVALID>]

Hello Hops,

I've been struggling with a few classpath related issues:

* Plugin data types are only accessible from the plugin they were
introduced with (Avro, Graph)
* It's not possible in a safe way for another plugin to add plugins (Beam)
to the root class loader

This has been causing all sorts of class loader problems which are
typically resolved by either shoving everything in the root classloader
(Avro data type) or by having large blobs for a plugin (engines/beam).

In the ideal scenario we'd have for example all the Kafka plugins in one
plugin with all the dependencies nicely grouped together in one
plugins/transforms/kafka folder and this would include all the Beam related
code as well.  The caveats being that we can't ship the Beam libs in every
plugin and that it should be easy to get rid of functionality.

What I would love to do is come up with an alternative way of assembling
and building our software.  For this to happen I think it should be
possible for any external "plugin" project to register classes in the root
class loader.  To make this happen there are various options like for
example having an extra folder like "libroot" in the plugin folder.  It
would have to act as if the libraries in there belonged to the root
classloader and our scripts would need to be able to pick this up.

I would also love to see some extra metadata around the libraries that we
have assembled in folders.  For example we'll want to create a smarter "fat
jar" builder which knows that Spark, Flink and Dataflow are different
platforms and that we don't need all libraries from all 3 platforms to run
something on either of those.  Perhaps by splitting up libraries in a more
fine-grained manner we can also add a small JSON file like
"library-metadata.json" containing some metadata that can then be picked up
by the fat jar builder?
In the plugins/engines/beam case you'd have folders: lib, libroot,
libspark, libflink, libdataflow, ...
The Kafka transforms code could move to plugins/transforms/kafka and so on.

Let's brainstorm around the possibilities and the possible problems to come
up with the next architecture for Hop.

Cheers,
Matt