Re: [logging] make dependency on servlet api optional

["J.Pietschmann" <j3...@yahoo.de>]

Simon Kitching wrote:
> This is only a *compile-time* dependency.
> 
> Currently there is a single utility class provided in the standard
> logging jar which can be used to avoid memory leaks when using
> commons-logging in servlet containers.
> 
> The presence of the class doesn't do any harm when used in
> non-servlet-containers; commons-logging will run fine and the class is
> simply not used.

There's a recurring problem: projects developing into kitchen-sinks
(Damn hard to avoid writing "kitching sink" here :-) ).

I had to build Apache software from source multiple times for debugging
purposes, and I don't like to track down all of the obscure compile
time dependencies. Nor do I like to wait half an hour until all
dependencies have been fetched from ibiblio or whatever.

I'd prefer a more modular approach:
- Split the project (preferred, but not all that easy)
- Provide several build targets according to dependencies
- Compile "optional" (non-core, or less likely to be missed) components
  or features only if libraries they depend on are in the classpath.
  Issue a warning otherwise: "Component/Feature FooBar not compiled
  because FooLib not found..."

Take a look at the FOP build file to see the third approach in action.

J.Pietschmann

---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org

Re: [logging] make dependency on servlet api optional

[robert burrell donkin <ro...@blueyonder.co.uk>]

(getting back on track now, i hope)

On Wed, 2005-07-06 at 13:21 +1200, Simon Kitching wrote:
> On Tue, 2005-07-05 at 23:47 +0100, robert burrell donkin wrote:
> > On Sun, 2005-07-03 at 11:51 +1200, Simon Kitching wrote: 
> > >
> > > The memory-leak-in-servlet-engine issue is a common problem and does
> > > need to be addressed one way or another. Taking a wild guess, I would
> > > think that perhaps 50% of all uses of commons-logging is in a J2EE or
> > > servlet container, so including this code in core seems reasonable to
> > > me.
> > 
> > that depends on whether the concept of core comes from a dependency
> > perspective or a usage perspective. there are advantages in structuring
> > builds so that dependencies can be managed most effectively and then
> > distributing a combined headlining jar containing the most popular
> > classes. 
> 
> By "core" I meant the jar that users are generally recommended to
> download: commons-logging.jar. 
> 
> The ServletContextCleaner also needs to be in
> commons-logging-adapters.jar as that is the jar recommended for webapps
> where commons-logging or commons-logging-api is in the parent.

that's fine by me. i'm happy with a fat commons-logging.jar so long as
finer grained distributions can be built.

what bothers me is the dependency graph for the most basic distribution.
a good example of the problems that bigger graphs create is debian. JCL
is depended upon by a lot of java libraries and debian checks that all
dependencies will run on a free JVM. adding a dependency may mean that
JCL and all components that depend upon it would have to be moved into
contrib.
 
<snip>
 
> > > > I'd prefer a more modular approach:
> > > > - Split the project (preferred, but not all that easy)
> > > > - Provide several build targets according to dependencies
> > > > - Compile "optional" (non-core, or less likely to be missed) components
> > > >   or features only if libraries they depend on are in the classpath.
> > > >   Issue a warning otherwise: "Component/Feature FooBar not compiled
> > > >   because FooLib not found..."
> > > 
> > > This is pretty much what commons-logging currently does. So obviously
> > > people share your view.
> > 
> > there are several reasons why experience has taught that this approach
> > often proves the best (i'll give some reasons for this statement at the
> > bottom rather than break up the flow now). however, the modularity in
> > the current build isn't as clearly reflected in the project structure as
> > it might be. it might be worth considering a move to a structure better
> > reflecting this (which may be Jörg's suggestion).
> 
> There are two issues:
> * modularity of the build
> * optionality[1] of the build
>   
> [1] This new word is copyright by me :-)

-1 a coiner should be generous with those words he coins ;)

> I've got no real objection to making the build more modular. I do think
> making parts of the build optional is going to cause grief.

we've been over this before so here's a summary. i'd like to see a
minimal build which creates a JCL which will function in a very basic
fashion for other libraries to depend upon. in addition, there would a
number of modules capable of being build separately. i would support the
primary user distribution being a series of fat jars but think that a
minimal jar for use by libraries is important. expert users would be
able to use the source to roll custom combinations.

> > however, i'm now of the opinion that JCL requires lots more
> > documentation than it has. IMHO optional jar's are much easier to
> > explain than classloading.
> 
> More docs are definitely needed. I don't see how having a set of jars is
> going to help us avoid explaining classloader issues though.

don't think i expressed myself very well: given that explaining
classloading is difficult, we should be capable of explaining an
optional jar.

> > this can be addressed by looking at the jars we distribution. what works
> > well with a modular approach is to separate the builds but distribute a
> > fat jar (containing everything) as the standard (named) jar.
> > 
> > adopting this policy would lead to a distributing something like:
> > 
> > commons-logging.jar (containing everything)  
> > 
> > commons-logging-core.jar (containing everything but optional)
> > commons-logging-optional.jar (optional stuff including deprecated)
> > commons-logging-api.jar 
> 
> Yep, I could go for this. 
> 
> Personally I would generate this jar by compiling commons-logging.jar
> then creating other jars with sets of classes extracted from the
> "everything" jar. [but see comments re gump below]
> 
> I get the impression that is not what you mean by "modular", and that
> you would prefer to generate different jars then combine their classes
> to create the "everything" jar. 

yes, that's the way i saw it working. suspect that this style works
better with maven. agnostic about the practicalities, though.

> > monolithic builds sooner or later run into difficulties with
> > dependencies. these difficulties come in different forms. 
> > 
> > there is the issue of dependency proliferation. the fight against
> > dependency proliferation means that good code that is only useful in a
> > limited number of use cases is not accepted whereas the ill effects of
> > drifting towards kitchen sink dependency are well known. monolithic
> > builds have to err on the side of caution. 
> 
> One could argue that the LogKitLogger is in this category - it's hardly
> ever used, and so it's debatable whether it should be included in the
> standard log4j jar. I agree that with the current setup, if something
> like LogKitLogger came along we would probably reject it as not being
> worth including in the standard jars.
> 
> I don't quite see how a "modular" build would deal with this though..

if the build was modular, i'd be included to accept code to log to that
hypothetical logger as a module.

> > sooner or later, it is necessary to cope with dependencies which break
> > backwards compatibility. monolithic builds are usually force to adopt a
> > single choice of dependency. this causes major pain to downstream users
> > by forcing a particular choice of library versions upon. it is often
> > possible to structure modular builds so that different versions of the
> > same library can be supported.
> 
> Sorry, I don't see what you mean by this. We do have this situation now,
> where log4j12 and log4j13 are incompatible. But commons-logging will be
> able to support them both fine.
> 
> commons-logging is in the fortunate state of having *no* core
> dependencies. Only the adapter classes have dependencies, and they are
> all "optional" in the sense that their dependencies are irrelevant
> unless the user selects one.
> 
> So while this may be a reasonable issue for other projects, I don't see
> its relevance to commons-logging.

my concern is that if the build requires all those dependencies then
downstream builders (automatic like gump or manual like debian) will
list JCL as requiring all those dependencies. 
 
> > JCL is used by a huge number of downstream users. a failure in the JCL
> > gump build (a good metric) causes a gump storm containing hundreds of
> > consequent failures. gump is an example of a dependency management tool.
> > maven is another. the number of dependencies required to build a basic,
> > functional JCL is small (though larger than it should be). if the basic
> > build compiles more than is strictly needed by dependent, then this
> > causes problems for dependency management tools. in this end, this may
> > also prove a source of difficulties since JCL may end up depending
> > (indirectly) upon itself.
> > 
> > these forces often result in monolithic builds tending towards modular
> > ones.
> 
> So you're saying that if we provide a commons-logging-core build, then
> other components in gump can depend on that. And then if the log4j
> adapter fails to build we get a single failure for "commons-logging.jar"
> but succeed for "commons-logging-core.jar" and so all the other projects
> build ok?

+1

> That's a good point.

a better one is that debian can build JCL using that build and will not
have to build and verify all the other dependencies in order to
distribute JCL.

<snip>

> We're agreed that the current setup sucks. I do know how to simplify the
> existing build to:
> * require all (currently optional) dependencies 
> * build everything
> * build subjars out of the everything jar
> 
> But I don't know how to achieve what you're looking for.

it's really a long term goal (rather than a short term one). it'd
probably be reason to just add a task that builds a small core which can
be used by downstream builders.

- robert


---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org

Re: [logging] make dependency on servlet api optional

[Simon Kitching <sk...@apache.org>]

On Tue, 2005-07-05 at 23:47 +0100, robert burrell donkin wrote:
> On Sun, 2005-07-03 at 11:51 +1200, Simon Kitching wrote: 
> >
> > The memory-leak-in-servlet-engine issue is a common problem and does
> > need to be addressed one way or another. Taking a wild guess, I would
> > think that perhaps 50% of all uses of commons-logging is in a J2EE or
> > servlet container, so including this code in core seems reasonable to
> > me.
> 
> that depends on whether the concept of core comes from a dependency
> perspective or a usage perspective. there are advantages in structuring
> builds so that dependencies can be managed most effectively and then
> distributing a combined headlining jar containing the most popular
> classes. 

By "core" I meant the jar that users are generally recommended to
download: commons-logging.jar. 

The ServletContextCleaner also needs to be in
commons-logging-adapters.jar as that is the jar recommended for webapps
where commons-logging or commons-logging-api is in the parent.

> 
> > Tracking down obscure compile time dependencies is not an issue; they
> > are all right there on ibiblio. And we can provide an Ant task to
> > download them.
> > 

> but i agree with simon that maven now solves this problem very well.
> (indeed IIRC, this issue was one of those that maven was created to
> address.) 

Ant can do this too. In fact I'll try to commit a patch to build.xml to
auto-download all dependencies today.

>  
> > > I'd prefer a more modular approach:
> > > - Split the project (preferred, but not all that easy)
> > > - Provide several build targets according to dependencies
> > > - Compile "optional" (non-core, or less likely to be missed) components
> > >   or features only if libraries they depend on are in the classpath.
> > >   Issue a warning otherwise: "Component/Feature FooBar not compiled
> > >   because FooLib not found..."
> > 
> > This is pretty much what commons-logging currently does. So obviously
> > people share your view.
> 
> there are several reasons why experience has taught that this approach
> often proves the best (i'll give some reasons for this statement at the
> bottom rather than break up the flow now). however, the modularity in
> the current build isn't as clearly reflected in the project structure as
> it might be. it might be worth considering a move to a structure better
> reflecting this (which may be Jörg's suggestion).

There are two issues:
* modularity of the build
* optionality[1] of the build
  
[1] This new word is copyright by me :-)

I've got no real objection to making the build more modular. I do think
making parts of the build optional is going to cause grief.

> however, i'm now of the opinion that JCL requires lots more
> documentation than it has. IMHO optional jar's are much easier to
> explain than classloading.

More docs are definitely needed. I don't see how having a set of jars is
going to help us avoid explaining classloader issues though.


> this can be addressed by looking at the jars we distribution. what works
> well with a modular approach is to separate the builds but distribute a
> fat jar (containing everything) as the standard (named) jar.
> 
> adopting this policy would lead to a distributing something like:
> 
> commons-logging.jar (containing everything)  
> 
> commons-logging-core.jar (containing everything but optional)
> commons-logging-optional.jar (optional stuff including deprecated)
> commons-logging-api.jar 

Yep, I could go for this. 

Personally I would generate this jar by compiling commons-logging.jar
then creating other jars with sets of classes extracted from the
"everything" jar. [but see comments re gump below]

I get the impression that is not what you mean by "modular", and that
you would prefer to generate different jars then combine their classes
to create the "everything" jar. 



> monolithic builds sooner or later run into difficulties with
> dependencies. these difficulties come in different forms. 
> 
> there is the issue of dependency proliferation. the fight against
> dependency proliferation means that good code that is only useful in a
> limited number of use cases is not accepted whereas the ill effects of
> drifting towards kitchen sink dependency are well known. monolithic
> builds have to err on the side of caution. 

One could argue that the LogKitLogger is in this category - it's hardly
ever used, and so it's debatable whether it should be included in the
standard log4j jar. I agree that with the current setup, if something
like LogKitLogger came along we would probably reject it as not being
worth including in the standard jars.

I don't quite see how a "modular" build would deal with this though..

> sooner or later, it is necessary to cope with dependencies which break
> backwards compatibility. monolithic builds are usually force to adopt a
> single choice of dependency. this causes major pain to downstream users
> by forcing a particular choice of library versions upon. it is often
> possible to structure modular builds so that different versions of the
> same library can be supported.

Sorry, I don't see what you mean by this. We do have this situation now,
where log4j12 and log4j13 are incompatible. But commons-logging will be
able to support them both fine.

commons-logging is in the fortunate state of having *no* core
dependencies. Only the adapter classes have dependencies, and they are
all "optional" in the sense that their dependencies are irrelevant
unless the user selects one.

So while this may be a reasonable issue for other projects, I don't see
its relevance to commons-logging.

> 
> JCL is used by a huge number of downstream users. a failure in the JCL
> gump build (a good metric) causes a gump storm containing hundreds of
> consequent failures. gump is an example of a dependency management tool.
> maven is another. the number of dependencies required to build a basic,
> functional JCL is small (though larger than it should be). if the basic
> build compiles more than is strictly needed by dependent, then this
> causes problems for dependency management tools. in this end, this may
> also prove a source of difficulties since JCL may end up depending
> (indirectly) upon itself.
> 
> these forces often result in monolithic builds tending towards modular
> ones.

So you're saying that if we provide a commons-logging-core build, then
other components in gump can depend on that. And then if the log4j
adapter fails to build we get a single failure for "commons-logging.jar"
but succeed for "commons-logging-core.jar" and so all the other projects
build ok?

That's a good point.

If you think you know how to restructure commons-logging so we build a
core separately from the other bits, so that gump projects can depend on
the core only, then go for it. 

If this is restructure is going to take a significant amount of
time/research, though, then we need to look at the practical side of
this: who's got the time? I expect to lose almost all my OSS development
time within the next few weeks and will pretty much disappear from this
list. And while Brian and yourself are both skilled logging developers
neither of you seem to have a lot of time to spare. So unless someone
else is going to step up, we need to look at getting a release out
within the next few weeks or it probably won't happen until next year. 

We're agreed that the current setup sucks. I do know how to simplify the
existing build to:
* require all (currently optional) dependencies 
* build everything
* build subjars out of the everything jar

But I don't know how to achieve what you're looking for.

Regards,

Simon


---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org

Re: [logging] make dependency on servlet api optional

[robert burrell donkin <ro...@blueyonder.co.uk>]

On Sun, 2005-07-03 at 11:51 +1200, Simon Kitching wrote: 
> On Sat, 2005-07-02 at 21:29 +0200, J.Pietschmann wrote:
> > Simon Kitching wrote:
> > > This is only a *compile-time* dependency.
> > > 
> > > Currently there is a single utility class provided in the standard
> > > logging jar which can be used to avoid memory leaks when using
> > > commons-logging in servlet containers.
> > > 
> > > The presence of the class doesn't do any harm when used in
> > > non-servlet-containers; commons-logging will run fine and the class is
> > > simply not used.
> > 
> > There's a recurring problem: projects developing into kitchen-sinks
> 
> I don't believe this is the issue here. Commons-logging has a very
> definite focus so arbitrary feature growth is not really a danger.
>
> The memory-leak-in-servlet-engine issue is a common problem and does
> need to be addressed one way or another. Taking a wild guess, I would
> think that perhaps 50% of all uses of commons-logging is in a J2EE or
> servlet container, so including this code in core seems reasonable to
> me.

that depends on whether the concept of core comes from a dependency
perspective or a usage perspective. there are advantages in structuring
builds so that dependencies can be managed most effectively and then
distributing a combined headlining jar containing the most popular
classes. 

> > I had to build Apache software from source multiple times for debugging
> > purposes, and I don't like to track down all of the obscure compile
> > time dependencies. Nor do I like to wait half an hour until all
> > dependencies have been fetched from ibiblio or whatever.
> 
> Tracking down obscure compile time dependencies is not an issue; they
> are all right there on ibiblio. And we can provide an Ant task to
> download them.
> 
> I guess for people on dial-up connections the download time could be a
> nuisance.

in my time, i've also spent too long tracking down obscure references
when wanting to build a project just to hack a simple fix. (if you want
real pain, then i'd suggest setting up a full gump run on a local
machine.)

but i agree with simon that maven now solves this problem very well.
(indeed IIRC, this issue was one of those that maven was created to
address.) 
 
> > I'd prefer a more modular approach:
> > - Split the project (preferred, but not all that easy)
> > - Provide several build targets according to dependencies
> > - Compile "optional" (non-core, or less likely to be missed) components
> >   or features only if libraries they depend on are in the classpath.
> >   Issue a warning otherwise: "Component/Feature FooBar not compiled
> >   because FooLib not found..."
> 
> This is pretty much what commons-logging currently does. So obviously
> people share your view.

there are several reasons why experience has taught that this approach
often proves the best (i'll give some reasons for this statement at the
bottom rather than break up the flow now). however, the modularity in
the current build isn't as clearly reflected in the project structure as
it might be. it might be worth considering a move to a structure better
reflecting this (which may be Jörg's suggestion).

i'm not really sure that the layout adopted works very well (a separate
optional directory) and now we use subversion, it's cheap to reform the
layout. (i do want to move the demonstration out sometime soon BTW.)   

> However it complicates things:
> * build.xml harder to read and maintain. For example, if building
>   various parts is optional, then all the related unit tests also
>   need to be made optional, and run only when the corresponding
>   parts are present.

true

it's easier to go for an explicit module structure rather than use build
tricks

> * we need more documentation, and more complex instructions for users.

true

however, i'm now of the opinion that JCL requires lots more
documentation than it has. IMHO optional jar's are much easier to
explain than classloading.

> It has to be remembered that commons-logging is generally something that
> people don't deliberately use; it's a dependency that is dragged along
> with some other lib that they are really interested in. So people don't
> have much interest in reading docs or configuring it correctly.
> Commons-logging really needs to work out-of-the-box as much as possible.

+1 (but we all know how difficult that make it for us :)

> And already people moan about having to include commons-logging.jar when
> using commons libs like digester or configuration. Providing them with
> complex documentation indicating under what circumstances they need to
> include additional jars won't make commons-logging more popular.

+1

this can be addressed by looking at the jars we distribution. what works
well with a modular approach is to separate the builds but distribute a
fat jar (containing everything) as the standard (named) jar.

adopting this policy would lead to a distributing something like:

commons-logging.jar (containing everything)  

commons-logging-core.jar (containing everything but optional)
commons-logging-optional.jar (optional stuff including deprecated)
commons-logging-api.jar 
... 

> > Take a look at the FOP build file to see the third approach in action.
> 
> Yes, but FOP is quite a different beast from logging. People explicitly
> download and use FOP, while people usually get commons-logging as an
> accidental side-effect of some other lib.

AIUI FOP is also used quite extensively as a library so it has some
similarities as well as differences. 

> And commons-logging is only half-a-dozen classes while FOP is -- just a
> little bit larger!

that's certainly true!


monolithic builds sooner or later run into difficulties with
dependencies. these difficulties come in different forms. 

there is the issue of dependency proliferation. the fight against
dependency proliferation means that good code that is only useful in a
limited number of use cases is not accepted whereas the ill effects of
drifting towards kitchen sink dependency are well known. monolithic
builds have to err on the side of caution. 

sooner or later, it is necessary to cope with dependencies which break
backwards compatibility. monolithic builds are usually force to adopt a
single choice of dependency. this causes major pain to downstream users
by forcing a particular choice of library versions upon. it is often
possible to structure modular builds so that different versions of the
same library can be supported.

JCL is used by a huge number of downstream users. a failure in the JCL
gump build (a good metric) causes a gump storm containing hundreds of
consequent failures. gump is an example of a dependency management tool.
maven is another. the number of dependencies required to build a basic,
functional JCL is small (though larger than it should be). if the basic
build compiles more than is strictly needed by dependent, then this
causes problems for dependency management tools. in this end, this may
also prove a source of difficulties since JCL may end up depending
(indirectly) upon itself.

these forces often result in monolithic builds tending towards modular
ones.

- robert


---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org

Re: [logging] make dependency on servlet api optional

[Simon Kitching <sk...@apache.org>]

On Sat, 2005-07-02 at 21:29 +0200, J.Pietschmann wrote:
> Simon Kitching wrote:
> > This is only a *compile-time* dependency.
> > 
> > Currently there is a single utility class provided in the standard
> > logging jar which can be used to avoid memory leaks when using
> > commons-logging in servlet containers.
> > 
> > The presence of the class doesn't do any harm when used in
> > non-servlet-containers; commons-logging will run fine and the class is
> > simply not used.
> 
> There's a recurring problem: projects developing into kitchen-sinks

I don't believe this is the issue here. Commons-logging has a very
definite focus so arbitrary feature growth is not really a danger.

The memory-leak-in-servlet-engine issue is a common problem and does
need to be addressed one way or another. Taking a wild guess, I would
think that perhaps 50% of all uses of commons-logging is in a J2EE or
servlet container, so including this code in core seems reasonable to
me.


> (Damn hard to avoid writing "kitching sink" here :-) ).

I won't comment on this!

> 
> I had to build Apache software from source multiple times for debugging
> purposes, and I don't like to track down all of the obscure compile
> time dependencies. Nor do I like to wait half an hour until all
> dependencies have been fetched from ibiblio or whatever.

Tracking down obscure compile time dependencies is not an issue; they
are all right there on ibiblio. And we can provide an Ant task to
download them.

I guess for people on dial-up connections the download time could be a
nuisance.

> 
> I'd prefer a more modular approach:
> - Split the project (preferred, but not all that easy)
> - Provide several build targets according to dependencies
> - Compile "optional" (non-core, or less likely to be missed) components
>   or features only if libraries they depend on are in the classpath.
>   Issue a warning otherwise: "Component/Feature FooBar not compiled
>   because FooLib not found..."

This is pretty much what commons-logging currently does. So obviously
people share your view.

However it complicates things:
* build.xml harder to read and maintain. For example, if building
  various parts is optional, then all the related unit tests also
  need to be made optional, and run only when the corresponding
  parts are present.
* we need more documentation, and more complex instructions for users.

It has to be remembered that commons-logging is generally something that
people don't deliberately use; it's a dependency that is dragged along
with some other lib that they are really interested in. So people don't
have much interest in reading docs or configuring it correctly.
Commons-logging really needs to work out-of-the-box as much as possible.

And already people moan about having to include commons-logging.jar when
using commons libs like digester or configuration. Providing them with
complex documentation indicating under what circumstances they need to
include additional jars won't make commons-logging more popular.

> Take a look at the FOP build file to see the third approach in action.

Yes, but FOP is quite a different beast from logging. People explicitly
download and use FOP, while people usually get commons-logging as an
accidental side-effect of some other lib.

And commons-logging is only half-a-dozen classes while FOP is -- just a
little bit larger!


Note that I'm not trying to "shoot down" your view. I'm just putting the
opposing case, and we'll see what the consensus is.

Regards,

Simon


---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org