Improving the accessibility of the Jackrabbit core

[Jukka Zitting <ju...@gmail.com>]

Hi,

Based on private discussions I'd like to raise the issue of the
accessibility of the Jackrabbit core codebase. We have a small number
of people who are intimately familiar with the core codebase (see the
numbers below), but others find the core hard to navigate and that
this drives up the barrier of entry of contributing to Jackrabbit.

Please share any good ideas on how we could best lower the barrier.
I'm open to all sorts of ideas, like more documentation (javadocs, UML
diagrams, architectural descriptions, etc.), scheduled Q&A sessions on
IRC, an informal Jackrabbit workshop during the Hackathon in
ApacheCon, etc. I'm also interested in the priorities, i.e. what would
give us the most "bang for buck" in terms of making it easier for
people to get familiar with the Jackrabbit core and start
contributing.

$ svn log src/main/java/org/apache/jackrabbit/core | \
  perl -lne '/^r[0-9]+ \| (.*?) \|/ and print $1' | sort | uniq -c | sort -n -r
    371 stefan
    199 tripod
    185 mreutegg
    127 jukka
     27 dpfister
     13 fielding
     10 angela
      4 fmeschbe
      3 edgarpoce
      2 sylvain

BR,

Jukka Zitting

-- 
Yukatan - http://yukatan.fi/ - info@yukatan.fi
Software craftsmanship, JCR consulting, and Java development

Re: Improving the accessibility of the Jackrabbit core

[David Nuescheler <da...@gmail.com>]

Hi Roy,

Thanks for helping me crystallize my thoughts and
trying to sharpen my statements ;)

> That is why each of the core developers has veto power over the code.
> If we want to ensure that every line is adequately reviewed, then ask
> for the core code to be governed by the RTC (review-then-commit) rule.
> Note, however, that such a requirement will extend to all commits on
> that part of the code.
I would never doubt that the ASF has all the machinery in
place to ensure code quality on levels that are way beyond
what Jackrabbit might ever reach. I think an open source
landmark project like the HTTP Server is proving this in a
very impressive way.

> I don't think you understand.  This is an Apache project and anyone can
> contribute to any part of it.
I would like to apologize, it was not my intention to
imply or suggest anything different.

Let me try to rephrase my personal feeling on
this subject as follows:

Inherently in any piece of software there are certain
areas of the code that are used and run more frequently
and others that are used less frequently. In my
experience the pieces that are at the core of any piece of
software have a greater impact on the overall stability
and performance and changes may have a side effects that
are far reaching. Also the core pieces tend to
be the most mature code, in many cases some of
those source files have been updated more frequently.

Within the realm and maturity of the Jackrabbit codebase
I think the best maintained and most mature code that
we have is in the core. Which is both obvious and good.
(And just to be clear, "well maintained" and "mature"
do not imply an architectural statement)

The essence of my post should have expressed (and we may
very well disagree on that) is that biggest stumbling blocks in
the uptake of Jackrabbit adoption are not complexities or
deficiencies in the core architecture, but the issues that Dave
mentioned in his post earlier in this thread and I wanted to
find out if others have a different opinion on that.

> The degree of review we require of those
> contributions is decided by the PMC (our committers).
> We can increase the requirements on review of the core code
> and we can separate compatible and incompatible changes
> into versioned branches, but we cannot ask of
> others what we do not accept of ourselves.
Agreed.

> In my opinion, the core code continues to evolve as people try to do
> larger and more expressive things with Jackrabbit and apply JCR to
> real problem sets.  We need to welcome that and change things based
> on their technical merits, not any preconceived notions of how much
> a person knows about the current (highly opaque) core architecture.
> Most likely, this will mean simplifying the core by removing or
> refactoring some of the spaghetti dependencies.
When I read your comment, I wondered what I may have said
that made you think that I am opposed to that. I realized that
I completely failed to share my thoughts about "the SPI" and how
that might stimulate a more modular and more transparent architecture
of the core. I will (re-)initiate a separate thread on that.
Sorry for the confusion.

> One of those things that will change is the degree of extensibility,
> since that is the heart of any successful open source project and
> Jackrabbit isn't even halfway there yet.  I am sure that others with
> fresh energy will see new ways to solve the same problem that will
> not be burdened with the legacy decisions that we made for one reason
> or another.  When those ideas are presented, they will be subject to
> intense scrutiny and adopted based only on their proven benefits.
> They will not be judged based on who wrote them or how much time they
> spent writing the initial core code.
I couldn't agree more.

regards,
david

Re: Improving the accessibility of the Jackrabbit core

["Roy T. Fielding" <fi...@gbiv.com>]

On Sep 6, 2006, at 4:14 AM, David Nuescheler wrote:

> Personally, I believe that for example a "restore" facility has to be
> buried deep down in the core and therefore the code has to comply
> with the high quality requirements that we have for code in the core
> and for the seasoned "Jackrabbit experience" of a developer.

That is why each of the core developers has veto power over the code.
If we want to ensure that every line is adequately reviewed, then ask
for the core code to be governed by the RTC (review-then-commit) rule.
Note, however, that such a requirement will extend to all commits on
that part of the code.

> In my mind your experience with developing very close
> to the "heart" of Jackrabbit should not lead us to opening
> up the core so inexperienced Jackrabbit developers can
> contribute, but it should help us realize that we have
> very high requirements for Jackrabbit developers that make
> modifications to the core.

I don't think you understand.  This is an Apache project and anyone can
contribute to any part of it.  The degree of review we require of those
contributions is decided by the PMC (our committers).  We can increase
the requirements on review of the core code and we can separate  
compatible
and incompatible changes into versioned branches, but we cannot ask of
others what we do not accept of ourselves.

In my opinion, the core code continues to evolve as people try to do
larger and more expressive things with Jackrabbit and apply JCR to
real problem sets.  We need to welcome that and change things based
on their technical merits, not any preconceived notions of how much
a person knows about the current (highly opaque) core architecture.
Most likely, this will mean simplifying the core by removing or
refactoring some of the spaghetti dependencies.

One of those things that will change is the degree of extensibility,
since that is the heart of any successful open source project and
Jackrabbit isn't even halfway there yet.  I am sure that others with
fresh energy will see new ways to solve the same problem that will
not be burdened with the legacy decisions that we made for one reason
or another.  When those ideas are presented, they will be subject to
intense scrutiny and adopted based only on their proven benefits.
They will not be judged based on who wrote them or how much time they
spent writing the initial core code.

....Roy

Re: Improving the accessibility of the Jackrabbit core

[David Nuescheler <da...@gmail.com>]

Hi Nico,

Thanks for your mail.

> I will work on the documentation directly on the wiki (when I can start this
> task). I will ask a lot of questions *though*.
Looking forward to it ;)

> One precision on the backup tool: it is working (and I am polishing the code
> that needs to fit in Core). And with my "new" JR understanding, I plan to
> start implementing a version 2 in my spare time having hotbackup.
Excellent, thanks for all your efforts.

I did not mean to imply that the backup tool was not working. If I should
have said anything like that, I would like to apologize.

regards,
david

Re: Improving the accessibility of the Jackrabbit core

[Nicolas <nt...@gmail.com>]

Hi David,

Good points.

I will work on the documentation directly on the wiki (when I can start this
task). I will ask a lot of questions *though*.

One precision on the backup tool: it is working (and I am polishing the code
that needs to fit in Core). And with my "new" JR understanding, I plan to
start implementing a version 2 in my spare time having hotbackup.

BR
Nico
my blog! http://www.deviant-abstraction.net !!

Re: Improving the accessibility of the Jackrabbit core

[David Nuescheler <da...@gmail.com>]

Hi Nico,

thanks for your explanations.

> My point was unclear: I meant JR core would have more functionnality (and
> some new ones are still needed) with a little bit of documentation of this
> part. I would gladly work on this part with your help since I will need to
> understand some parts better than I currently do.

That sounds fantastic and I think we can all benefit from a
documentation of your experience with the Jackrabbit core.

I agree with you that the features like the hot backup/restore or
clustering functionality are not even close to being complete in the
current core.

Nevertheless, I think that the amount of functionality missing in the core
is both well-known and also finite. Personally, I don't think
that we are looking at much more beyond the two mentioned features.

Personally, I believe that for example a "restore" facility has to be
buried deep down in the core and therefore the code has to comply
with the high quality requirements that we have for code in the core
and for the seasoned "Jackrabbit experience" of a developer.

In my mind your experience with developing very close
to the "heart" of Jackrabbit should not lead us to opening
up the core so inexperienced Jackrabbit developers can
contribute, but it should help us realize that we have
very high requirements for Jackrabbit developers that make
modifications to the core.

While I agree that it would be great to make the core
smaller and therefore offer more extension points, I think
that both "backup/restore" and "clustering" have to go into the
core and have to be developed very carefully and therefore are
in my mind not really suited to be developed by inexperienced
developers.

Or in short: In hindsight I think it is questionable to offer the
"Backup tool" as a Summer of Code Project.

...and please don't get me wrong, I can only blame myself for
not speaking up at the time, but I didn't see it coming either.
But I guess having this discussion now, is very valuable and will help
us to make smarter decisions in the future.

regards,
david

Re: Improving the accessibility of the Jackrabbit core

[Nicolas <nt...@gmail.com>]

Hi Stefan,

Cleaner code is a consequence of the current quality control process through
patches review; not something my point would lead to.

My point was unclear: I meant JR core would have more functionnality (and
some new ones are still needed) with a little bit of documentation of this
part. I would gladly work on this part with your help since I will need to
understand some parts better than I currently do.

BR,
Nico
my blog! http://www.deviant-abstraction.net !!

Re: Improving the accessibility of the Jackrabbit core

[Stefan Guggisberg <st...@gmail.com>]

On 9/6/06, Nicolas <nt...@gmail.com> wrote:
> On 9/6/06, David Nuescheler <da...@gmail.com> wrote:
>
> > While I agree that we need to have a modular design where people can
> > plug-in their extensions at certain defined interfaces and extension
> > points,
> > I would discourage the idea that every user needs to be able to submit
> > patches to the core.
> >
> > In my mind the core should be very compact and very controlled since
> > it has to be extremely stable and scalable, meaning that there is not
> > really a need to have dozens of developers working on a more
> > "smallish" core.
> >
>
> Hi,
>
> My two cents on the subject drawing from my experience on the backup tool.
>
> At first Jukka and I wanted to avoid impact on the core for the reasons you
> mentionned. It turned out we had to eventually update some parts of the
> core: some functionnalities were simply not there. We minimized the changes
> (only a few lines)... But they were quite bad (I exposed something that
> shouldn't). After some rethinking and a few try out, I am back to my initial
> plan with a few classes added to the core.
>
> This example shows the Core is "not over" in the sense, it lacks some
> functionnality (for instance in my case a way to import the versions). I
> think we need to remember JR is still a fairly new project and some use
> cases have still not been detected.
>
> Some functionnalities have not been needed yet for the core contributors but
> might emerge from other companies/individual (for instance my company would
> need to extend JR to support our needs). I think discouraging those
> contributions can be a bad idea: we should encourage them, keep the code and
> refactor them if necessary. This way both the contributor and the communitu
> take benefit from it: a new functionnality with a cleaner code.

i don't follow your argumentation. why would this lead to cleaner code?

cheers
stefan

>
> I agree with you though that we should encourage contribution and not update
> to the core. But we should document the core. In my case, it took me a lot
> of time the part I needed (I wrote a new UpdatableStateManager since I
> couldn't figure out how the EventFactory was working).
>
> BR
> Nicolas
> my blog! http://www.deviant-abstraction.net !!
>
>

Re: Improving the accessibility of the Jackrabbit core

[Nicolas <nt...@gmail.com>]

On 9/6/06, David Nuescheler <da...@gmail.com> wrote:

> While I agree that we need to have a modular design where people can
> plug-in their extensions at certain defined interfaces and extension
> points,
> I would discourage the idea that every user needs to be able to submit
> patches to the core.
>
> In my mind the core should be very compact and very controlled since
> it has to be extremely stable and scalable, meaning that there is not
> really a need to have dozens of developers working on a more
> "smallish" core.
>

Hi,

My two cents on the subject drawing from my experience on the backup tool.

At first Jukka and I wanted to avoid impact on the core for the reasons you
mentionned. It turned out we had to eventually update some parts of the
core: some functionnalities were simply not there. We minimized the changes
(only a few lines)... But they were quite bad (I exposed something that
shouldn't). After some rethinking and a few try out, I am back to my initial
plan with a few classes added to the core.

This example shows the Core is "not over" in the sense, it lacks some
functionnality (for instance in my case a way to import the versions). I
think we need to remember JR is still a fairly new project and some use
cases have still not been detected.

Some functionnalities have not been needed yet for the core contributors but
might emerge from other companies/individual (for instance my company would
need to extend JR to support our needs). I think discouraging those
contributions can be a bad idea: we should encourage them, keep the code and
refactor them if necessary. This way both the contributor and the communitu
take benefit from it: a new functionnality with a cleaner code.

I agree with you though that we should encourage contribution and not update
to the core. But we should document the core. In my case, it took me a lot
of time the part I needed (I wrote a new UpdatableStateManager since I
couldn't figure out how the EventFactory was working).

BR
Nicolas
my blog! http://www.deviant-abstraction.net !!

Re: Improving the accessibility of the Jackrabbit core

[Jukka Zitting <ju...@gmail.com>]

Hi,

[Hmm, too quick with the send button... more below.]

On 9/6/06, Jukka Zitting <ju...@gmail.com> wrote:
> On 9/6/06, David Nuescheler <da...@gmail.com> wrote:
> > While I agree that we need to have a modular design where people can
> > plug-in their extensions at certain defined interfaces and extension points,
> > I would discourage the idea that every user needs to be able to submit
> > patches to the core.
>
> I'm most concerned about the overhead for people going in trying to
> trace why Jackrabbit is behaving the way it does in some specific
> issue. This is often the first step of becoming a contributor, and in
> my opinion it's currently quite a high step to overcome.

What I'm really after here is enhancing the "given enough eyeballs,
all bugs are shallow" effect by better enabling not only the detection
but also the analysis and perhaps even fixing process of issues. I
think the accessibility of the core codebase is a key issue in
empowering users to get more involved in the bug resolution process.

> > In my mind the core should be very compact and very controlled since
> > it has to be extremely stable and scalable, meaning that there is not
> > really a need to have dozens of developers working on a more
> > "smallish" core.

We already have meritocracy and the patch review process in place for
quality control so I don't really see an issue with making
contribution easier. You are right in focusing the core goals and that
there is probably only so much work to be done there, but I'd rather
see people not contributing to the core because of not needing to
instead of not being able to.

BR,

Jukka Zitting

-- 
Yukatan - http://yukatan.fi/ - info@yukatan.fi
Software craftsmanship, JCR consulting, and Java development

Re: Improving the accessibility of the Jackrabbit core

[Jukka Zitting <ju...@gmail.com>]

Hi,

On 9/6/06, David Nuescheler <da...@gmail.com> wrote:
> Got it. Generally, I am more of a "given the right eyeballs, all bugs
> are shallow" type of person to begin with.

Perhaps we can find common ground at "enough right eyeballs". ;-)

> If I currently take look at the "shallowness" of actual "core" bugs ;)
> in Jackrabbit I see that the Jackrabbit community has an
> outstanding bug resolution time. To me this is probably one of the
> biggest strengths of Jackrabbit and its community.
> Do you see this as a weakness that needs improvement?

Definitely not. :-) What I do see as a weakness is that we rely on a
handful of core developers to keep up this level of support when we
could better tap the great potential within the community. In fact I'd
rather see the core developers spending more time being proactive
designing new features and improvements (like improving performance,
scalability, etc.) than reactive analyzing user issues when large
parts of that work could be distributed.

> I think in the end it all boils down to matter of priorities and
> I would be very interested in having a discussion around what
> we think drives and hinders the Jackrabbit adoption and community
> today and tomorrow, and therefore what we should focus on.

+1 There's already quite a lot of feedback on the adoption part, but
that would need to be summarized and analyzed to better focus the
efforts.

BR,

Jukka Zitting

-- 
Yukatan - http://yukatan.fi/ - info@yukatan.fi
Software craftsmanship, JCR consulting, and Java development

Re: Improving the accessibility of the Jackrabbit core

[David Nuescheler <da...@gmail.com>]

Hi Jukka,

Thanks for your thoughtful comments.

> I'm most concerned about the overhead for people going in trying to
> trace why Jackrabbit is behaving the way it does in some specific
> issue. This is often the first step of becoming a contributor, and in
> my opinion it's currently quite a high step to overcome.
Agreed.

> What I'm really after here is enhancing the "given enough eyeballs,
> all bugs are shallow" effect by better enabling not only the detection
> but also the analysis and perhaps even fixing process of issues. I
> think the accessibility of the core codebase is a key issue in
> empowering users to get more involved in the bug resolution process.
Got it. Generally, I am more of a "given the right eyeballs, all bugs
are shallow" type of person to begin with. But I guess that's based on
my personal experience.

If I currently take look at the "shallowness" of actual "core" bugs ;)
in Jackrabbit I see that the Jackrabbit community has an
outstanding bug resolution time. To me this is probably one of the
biggest strengths of Jackrabbit and its community.
Do you see this as a weakness that needs improvement?

> We already have meritocracy and the patch review process in place for
> quality control so I don't really see an issue with making
> contribution easier.
Me neither. I probably did not express that properly, apologies.
I am not afraid of contributions, not at all.
I find the core is very well supported right now. I wonder how
much of our resources we should focus on getting that level
of support and quality even higher in light of other tasks
at hand.

> You are right in focusing the core goals and that there is probably
> only so much work to be done there, but I'd rather see people
> not contributing to the core because of not needing to instead of
> not being able to.
I agree with you 100%.

I think in the end it all boils down to matter of priorities and
I would be very interested in having a discussion around what
we think drives and hinders the Jackrabbit adoption and community
today and tomorrow, and therefore what we should focus on.

regards,
david

Re: Improving the accessibility of the Jackrabbit core

[Jukka Zitting <ju...@gmail.com>]

Hi,

On 9/6/06, David Nuescheler <da...@gmail.com> wrote:
> Personally, I think we have to separate the concerns though, I think
> Jukka's initial post was going into the direction of making the internals
> of the core more accessible to more developers.

Correct. In any case, Dave's points are a valuable addition to the
feedback I gathered a while ago before the 1.0 release with the issue
of streamlining the end-user experience.

> While I agree that we need to have a modular design where people can
> plug-in their extensions at certain defined interfaces and extension points,
> I would discourage the idea that every user needs to be able to submit
> patches to the core.

I'm most concerned about the overhead for people going in trying to
trace why Jackrabbit is behaving the way it does in some specific
issue. This is often the first step of becoming a contributor, and in
my opinion it's currently quite a high step to overcome.



> In my mind the core should be very compact and very controlled since
> it has to be extremely stable and scalable, meaning that there is not
> really a need to have dozens of developers working on a more
> "smallish" core.



BR,

Jukka Zitting

-- 
Yukatan - http://yukatan.fi/ - info@yukatan.fi
Software craftsmanship, JCR consulting, and Java development

Re: Improving the accessibility of the Jackrabbit core

[David Nuescheler <da...@gmail.com>]

Hi All,

Dave, thanks a lot for your input.

> . Screenshots or easily downloadable sample app which
> actually does something with custom node types. the base war
> download is good, but how far could you go with it. Most open
> source applications have a contacts application or a phone book,
> or something similar. something that has a face, like a jsp to
> view whats in the repository would be great
> . the wiki has not been updated regularly, either the information is old or not many people go to it
> . the deployment models - creating a complete tomcat dist, which has the various deployment options running right out of the box would be nice.
> . a java example to add node types, for example for a phone book, which CRUDs the  node types would be nice
> . maybe a page, which lists the possibilities of applications that could be built with JR will be useful for newbies.

I completely agree with you that all of the above are excellent measures
that we should be looking at to ease the adoption of new
"content application" developers. I think it is very important that people
get things up and running very quickly and are equipped with very good
user documentation.

Personally, I think we have to separate the concerns though, I think
Jukka's initial post was going into the direction of making the internals
of the core more accessible to more developers.

I think that there are a number of steps that we can take into that
direction and I also think that for example the separation eventually
provided by the SPI will bring some more architectural clarity.

While I agree that we need to have a modular design where people can
plug-in their extensions at certain defined interfaces and extension points,
I would discourage the idea that every user needs to be able to submit
patches to the core.

In my mind the core should be very compact and very controlled since
it has to be extremely stable and scalable, meaning that there is not
really a need to have dozens of developers working on a more
"smallish" core.

regards,
david

Re: Improving the accessibility of the Jackrabbit core

[Dave Bobby <dv...@yahoo.com>]

I will put in my 2c since I did not see many replies to this post and I think addressing this question is very important for any open source project. i have not had much time to play with JR due to other work, so some of this might already be there.

. Screenshots or easily downloadable sample app which actually does something with custom node types. the base war download is good, but how far could you go with it. Most open source applications have a contacts application or a phone book, or something similar. something that has a face, like a jsp to view whats in the repository would be great
. the wiki has not been updated regularly, either the information is old or not many people go to it
. the deployment models - creating a complete tomcat dist, which has the various deployment options running right out of the box would be nice. 
. a java example to add node types, for example for a phone book, which CRUDs the  node types would be nice 
. maybe a page, which lists the possibilities of applications that could be built with JR will be useful for newbies.

just my 2c.

Thanks

Dave

Nicolas  <nt...@gmail.com> wrote: Hi,

I have got familiar with JR codebase in the last few months and follow is
based on my experience in the backup tool.
The community is really helpful when you need some help but in order to
understand the basic concept you need to dig into the code and into the JCR
spec.

A general documentation might be a good idea: a user one where key concepts
are explained (versioning, nodetypes, and so on). We can I think mostly
copy/paste from the JCR to the Wiki. We also need  I believe some
documentations about JR 's internals: how a node is updated what is an
ItemState.

BR
Nico
my blog! http://www.deviant-abstraction.net !!


 		
---------------------------------
 All-new Yahoo! Mail - Fire up a more powerful email and get things done faster.

Re: Improving the accessibility of the Jackrabbit core

[Nicolas <nt...@gmail.com>]

Hi,

I have got familiar with JR codebase in the last few months and follow is
based on my experience in the backup tool.
The community is really helpful when you need some help but in order to
understand the basic concept you need to dig into the code and into the JCR
spec.

A general documentation might be a good idea: a user one where key concepts
are explained (versioning, nodetypes, and so on). We can I think mostly
copy/paste from the JCR to the Wiki. We also need  I believe some
documentations about JR 's internals: how a node is updated what is an
ItemState.

BR
Nico
my blog! http://www.deviant-abstraction.net !!