On OSGi and class loading.

[Matt Sicker <bo...@gmail.com>]

So we have all these issues with supporting an OSGi environment, mainly due
to not many contributors using it in the first place. Most of the problems
with the OSGi bundles stem from naive class loading techniques like the
thread context class loader.

The short story is this. Each bundle in an OSGi framework has its own class
loader that only knows about the classes in the bundle plus any others that
were imported via the manifest file. The maven bundle plugin takes care of
managing the imports, so that's no big deal. It's nice to minimize the
imports by manually declaring all the relevant ones, but that can be
error-prone and tedious.

So, we have some class loading that happens in LogManager in order to load
the first API provider on the classpath. Well, this trick works if
log4j-api is a fragment bundle that hosts all the others. That seems to
work OK, but it's fragile and doesn't work for every common usage of OSGi.
A better solution would be a sort of ClassLoaderContext that can be
independently set up before LogManager initializes itself.

The next major OSGi problem is during plugin loading based on the
configuration files. Currently, the code scans for annotated classes on the
classpath, but this is another hugely fragile process. The largest problem
from here is that log4j-core-reduced won't be the parent fragment of, let's
say, all the other OSGi bundles like async, jdbc, mongo, couch, etc. This
is a further problem if we want to split up all the appenders and such into
separate bundles for the best modularity.

This issue can be solved a number of ways. We could change how plugins are
registered by flipping the side of registration by using a sort of
whiteboard pattern recommended by Knopflerfish (another OSGi provider).
That's probably the most old-school method of doing so, and there are
better ways to do it.

Another way would be to make all the plugins be services, basically, and
use something like ServiceTracker in OSGi core to keep track of this. The
problem here is that it introduces a dependency on OSGi for non-OSGi
environments. I don't know how feasible it would be to have log4j-core all
managed in OSGi regardless of environment, but it seems like an unfair
dependency to add. It's like how Eclipse uses OSGi internally, so any
plugin you write for it also needs to use Eclipse's OSGi facade and such.
It does make programming plugins easier IMO, but this could be too large a
change to get done for release.

There's another option that can be done two different ways (at least):
declarative services. This is like the services idea from above, but
instead of writing code to register and query OSGi and such, we instead
provide metadata about the plugins so that an OSGi framework can infer the
services from that. This can be done via annotations (or even XML files,
but annotations give a way to generate said XML files): you can use the
standard OSGi SCR annotations for some basic support; you can use the Felix
SCR annotations for more complex configuration (probably necessary), or you
can extend the Felix SCR annotation processor to assemble the service
declarations using your own annotations. I like the custom annotations idea
best since we're already using the @Plugin annotation on all the relevant
classes, so this might make the DS configuration trivial. Then again, this
would also require a bit of an audit through the code to split up dependent
classes that aren't plugins into categories: general-purpose shared classes
and plugin-specific "private" classes.

Finally, I have an idea on how to go about solving these sorts of problems
in general that don't rely on OSGi at all. Instead of using methods that
just try to find an appropriate class loader for a given class name, we
could use a central class loader registry. It could work similarly to how
the named logger hierarchy works. Then the exported bundles for each subset
of log4j-core can scan its own classpath for @Plugins, and then it could
register all those classes in the central class loader registry using its
own classloader.

In regards to this class loader registry, there would still be a couple
ways to go about doing it. This registry could be lazy and only store class
name strings paired to class loader weak references (gotta use references
to let the bundles get unloaded), or it could be more active and store the
actual loaded classes (which sort of makes it a weird type of class
pre-loader), but then bundles would have to deregister themselves on the
way out. Either way, the registry will have to have some sort of control
over the code that requests these classes so that when a bundle wants to
get unloaded, the class loader registry would be able to stop the classes
using them.

As you can see, this is a bit of a hard problem. This is all I've
discovered while trying to write a "simple" patch to fix the OSGi bugs.
There could be more issues lurking behind these, but I'm not sure yet.
Overall, we need a more robust plugin manager due to the separate bundles.

I'd like some feedback on this. I could work on this problem and submit
changes, but I have a feeling that it would be a rather large change. If
you guys would rather wait until 2.0.x or 2.1 to get proper OSGi support
bundled in, I'd be OK with that. The thing is, though, that waiting until
after GA will limit our options somewhat. Perhaps it would be prudent to
note in the documentation that core plugins are subject to change in how
they're done in the near future, but based on the clusterfuck that
HttpComponents 4.1 is (which I have extensive experience with at my job
thanks to other programmers not spending any time in updating their own
code to use 4.3), this could also be a bit of a disaster. I wouldn't want a
huge split in the versions people use because of API changes like that. The
custom Felix SCR processor idea would probably be the most
backward-compatible way to do this in that regard.

-- 
Matt Sicker <bo...@gmail.com>