Self-governance

[Marvin Humphrey <ma...@rectangular.com>]

Greets,

A thread about Lucy recently wrapped up on the Lucene general@ list.  

   http://markmail.org/message/iar5sbfqnzcsvro5

The board at Apache is breaking up umbrella projects.  The Lucene TLP (top
level project, as opposed to the Lucene Java subproject) has already spun off
Tika, Mahout, and Nutch; Lucy is next.  However, Lucy is not ready to become a
TLP yet, so that spawned a discussion of what should happen.

To summarize the outcome of the discussion:

  * Lucy must aim to "graduate" and become a TLP.
  * Lucy will stay where it is for now, rather than e.g. moving to the
    Incubator.
  * Lucy must regularly assess its progress towards graduation using the
    incubation checklist at http://incubator.apache.org/guides/graduation.html
    and report to the Lucene PMC.
  * No official deadline was set by the PMC, but Doug Cutting threw out "six
    months" and so we should probably start with that as our target.

Here's a template for us laid out by Grant Ingersoll, the Lucene PMC chair:

  1. Doing a release. 
  2. Showing some user list traction (i.e. real users) 
  3. Identifying and cultivating other contributors (via patches, discussions,
     JIRA issues, helping others, etc.) who can then become committers. 

The primary measure of success for a project at Apache is not technical: the
goal is to create a robust meritocratic development community.  In order to
graduate, we will need not only committers and users, but our own PMC which
takes responsibility for governing the project.

  http://www.apache.org/foundation/how-it-works.html#structure
  http://www.apache.org/dev/pmc.html

Over the next week, we should perform a self-critique and create our first
formal report, judging ourselves by the incubator criteria.  Our biggest
weakness from the ASF perspective, clearly, is that the project is too reliant
on me and would be unlikely to survive my departure at this time.  We should
set a goal of making me dispensible.

Here's where I think we stand with regards to Grant's plan, in a nutshell:

Lucy's high-level design is done, and a lot of the coding is done as well.
Developing and refining the Clownfish object model was the hardest part, and
that finished up last fall.  I think an initial release within 3 months is
doable, and we may be able to accelerate that.

I expect acquiring users after the release to be relatively straightforward.
Near-realtime search powered by mmap is a killer feature.

As for cultivating committers and PMC members, that will come naturally if we
acquire a significant user base.  I don't think we need to do anything special
for now beyond releasing good software, publicizing it effectively, and making
it easy and rewarding to contribute -- as we have tried to do over the last
year with our existing contributor base, with some success.

Marvin Humphrey

Re: Do you need help on Lucy?

[Marvin Humphrey <ma...@rectangular.com>]

On Sat, Jun 19, 2010 at 09:48:19PM +0200, VOITPTRPTR wrote:
> To be honest, I've no knowledge on how Lucene core is implemented.

Naivete can be helpful for some tasks, such as reviewing documentation and
APIs for clarity -- see the wiki page I put up earlier today at
<http://wiki.apache.org/lucy/BrainLog>.

> But I'm pretty confident when coding in C (portabilty, clarity, refactoring ...).
> More over I've strong background on algorithms and algorithms optimization.

Certainly those are valuable skills.  

Some parts of Lucy have been highly optimized, but there's a lot of room for
improvement and a lot of known inefficiencies.  Lucene has focused on
optimization much more than we have, so we get to study what they've done over
the last couple years and apply the lessons they've learned.

Up till now it's been more important to emphasize modularity and information
hiding so that the system we build can *withstand* optimization without
becoming an impenetrable rat's nest.

> Is it possible to ask questions about design choices of Lucy (how indexes
> are built, algorithms behind the scene...) in this mailing list as I'm
> missing this Information Retrieval skills?

There will be many opportunities for such conversations, particularly since a
lot of indexing classes are going to get submitted to JIRA over the next few
weeks.  When code gets submitted for inclusion is an ideal time to comment.

Personally, I enjoy giving long-winded answers as much as the next person, but
I've learned that I have to control my impulse to bloviate and invest my time
in ways that yield maximum return.  The more you contribute, the more you will
get back.

Marvin Humphrey

Re: Do you need help on Lucy?

[VOITPTRPTR <vo...@gmail.com>]

Hi Marvin,

> We have a wiki page up that covers the mechanics of contributing:
>     http://wiki.apache.org/lucy/HowToContribute

Excellent. I'll checkout the code and start reading on monday!

> Most people choose what they want to work on based on what they need or what
> interests them.  Since you don't mention wanting to work on a particular
> problem, there are a some general C tasks we could use help on and that don't
> require a lot of prior knowledge about the code base; I'll describe one of
> those.

To be honest, I've no knowledge on how Lucene core is implemented.
But I'm pretty confident when coding in C (portabilty, clarity, refactoring ...).
More over I've strong background on algorithms and algorithms optimization.

> A lot of Lucy code was originally written for C89.  We have since changed our
> C dialect to "the overlap of C99 and C++".

> allowing us to use a number of idioms which result in cleaner, more readable code.  One of these is the
> declaration of loop variables within a "for" construct:
> 
>    Index: core/Lucy/Object/VArray.c
>    ===================================================================
>    --- core/Lucy/Object/VArray.c    (revision 956160)
>    +++ core/Lucy/Object/VArray.c    (working copy)
>    @@ -55,8 +55,7 @@
>     VA_dump(VArray *self)
>     {
>         VArray *dump = VA_new(self->size);
>    -    uint32_t i, max;
>    -    for (i = 0, max = self->size; i < max; i++) {
>    +    for (uint32_t i = 0, max = self->size; i < max; i++) {
>             Obj *elem = VA_Fetch(self, i);
>             if (elem) { VA_Store(dump, i, Obj_Dump(elem)); }
>         }
> 
> A good place to start would be that file, VArray.c.
> Thanks for inquiring,

OK, I see.

Is it possible to ask questions about design choices of Lucy (how indexes are built, algorithms
behind the scene...) in this mailing list as I'm missing this Information Retrieval skills?

Regards
----------------------
voidptrptr@gmail.com

Re: Do you need help on Lucy?

[Marvin Humphrey <ma...@rectangular.com>]

On Sat, Jun 19, 2010 at 07:00:56AM +0200, VOITPTRPTR wrote:
> How one can contribute  (I'm a C developer) to Lucy?

We have a wiki page up that covers the mechanics of contributing:

    http://wiki.apache.org/lucy/HowToContribute

Most people choose what they want to work on based on what they need or what
interests them.  Since you don't mention wanting to work on a particular
problem, there are a some general C tasks we could use help on and that don't
require a lot of prior knowledge about the code base; I'll describe one of
those.

A lot of Lucy code was originally written for C89.  We have since changed our
C dialect to "the overlap of C99 and C++", allowing us to use a number of
idioms which result in cleaner, more readable code.  One of these is the
declaration of loop variables within a "for" construct:

    Index: core/Lucy/Object/VArray.c
    ===================================================================
    --- core/Lucy/Object/VArray.c    (revision 956160)
    +++ core/Lucy/Object/VArray.c    (working copy)
    @@ -55,8 +55,7 @@
     VA_dump(VArray *self)
     {
         VArray *dump = VA_new(self->size);
    -    uint32_t i, max;
    -    for (i = 0, max = self->size; i < max; i++) {
    +    for (uint32_t i = 0, max = self->size; i < max; i++) {
             Obj *elem = VA_Fetch(self, i);
             if (elem) { VA_Store(dump, i, Obj_Dump(elem)); }
         }

A good place to start would be that file, VArray.c.

Thanks for inquiring,

Marvin Humphrey

Do you need help on Lucy?

[VOITPTRPTR <vo...@gmail.com>]

Hi List,

How one can contribute  (I'm a C developer) to Lucy?

Regards
voidptrptr