You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucy.apache.org by Peter Karman <pe...@peknet.com> on 2010/07/23 18:00:58 UTC
Re: [Lucy] Roadmap for first release
Marvin Humphrey wrote on 7/18/10 12:59 PM:
>
> I propose a minimalist strategy that will allow us to get to a release as
> quickly as possible:
>
> 1. Branch Lucy off the last KS bugfix release rather than svn trunk.
> 2. Perform IP clearance and relicensing.
> 3. Perform a few massive find and replace operations to change the imported
> codebase to "Lucy".
> 4. Add code to enable Lucy to read existing KinoSearch 0.3x indexes.
> 5. Consider moving a few classes around.
> 6. Write a Lucy::Docs::KinoSearch2Lucy transition guide and a
> "kinosearch2lucy.pl" tool to adapt user codebases using 0.3x
> automatically.
>
+! to all those.
>
> It probably makes sense to make one more KinoSearch release addressing some of
> the issues for the transition. In particular, in svn trunk, FullTextType and
> StringType have been consolidated into TextType. It would be nice if new Lucy
> users never had to think about distinguishing between FullTextType and
> StringType, since they're going away anyway.
>
> I think we draw the line at moving classes around, though.
Yes.
I'd like to see a stable KinoSearch3 release with the least amount of changes
possible from the current stable branch.
> There are a number
> of other issues that need to be addressed before we fork off a stable "Lucy1"
> branch.
[snip]
> I think if we market the initial Lucy release as "help us get to a stable
> release", then A) people will be more forgiving of our "work in progress", and
> B) we may attract more contributors. We can put off the more disruptive work
> until later.
>
> Sound like a plan?
those all sound good for Lucy. Should not impede the KS3 release though. I
imagine Lucy1 as an improvement on KS3, inspiring users to migrate.
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: [Lucy] Roadmap for first release
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Fri, Jul 30, 2010 at 09:38:06PM -0700, Mattmann, Chris A (388J) wrote:
> Yeah, my big concern is that you¹re talking about a project that doesn¹t
> live at Apache (yet) on Apache lists. I like your timeline below, but there
> are no dates behind them? How long does it take to cut a 0.31 Kinosearch
> release? My hope is a few days, not a few weeks.
Days. In fact, in the interest of expediting the process, I withdraw my
proposal to consider moving stuff around now. Let's just plan to deal with
all that later. That means that I only have a couple bugfixes left and
0.30_11 can go, to be followed shortly by 0.31.
> We have Incubator reports to file over here in Apache-land (in a few weeks),
I believe our first monthly report is due no later than Wednesay August, 11 --
less than two weeks. :) We'll be ready.
> This dualism has to stop and the development/mailing list discussion/release
> cycle needs to occur over here at Apache.
Point taken. I agree.
Marvin Humphrey
Re: [Lucy] Roadmap for first release
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Marvin,
Yeah, my big concern is that you¹re talking about a project that doesn¹t
live at Apache (yet) on Apache lists. I like your timeline below, but there
are no dates behind them? How long does it take to cut a 0.31 Kinosearch
release? My hope is a few days, not a few weeks. We have Incubator reports
to file over here in Apache-land (in a few weeks), and those reports need
*Apache* milestones -- not milestones from a project that isn't at Apache.
This dualism has to stop and the development/mailing list discussion/release
cycle needs to occur over here at Apache.
Cheers,
Chris
On 7/30/10 9:55 AM, "Marvin Humphrey" <ma...@rectangular.com> wrote:
> On Thu, Jul 29, 2010 at 08:31:27PM -0700, Mattmann, Chris A (388J) wrote:
>> Thanks, Peter for the email. I kind of guessed it was more related to
>> KinoSearch.
>
> Since Lucy is about to assimilate the KinoSearch code base, they are now
> effectively the same project. I was perusing the early days of the
> SpamAssassin dev mailing list today to get a feel for how our code import
> would go, and I found a thread called "Where to fix the stable branch":
>
> http://markmail.org/message/e5avbm6z2pldv55u
>
> It addressed the problem of making a last bugfix release on GPL'd code in the
> 2.6 branch, when there was CLA'd code at Apache for 2.7+. I think the
> situation is analogous, and that like SpamAssassin, we will move on and leave
> the issue behind with time.
>
>> I think it would be good to keep the discussions focused on Apache over
>> here, even though I'm aware of the transition as part of the podling.
>
> I appreciate you're keeping our eyes on the prize. I agree that completing
> the transition as soon as possible is very important. Frankly, nobody wants
> this dualism to end more than I do.
>
> For me, this is mostly about compromise and community. Peter is an important
> contributor. Father Chrysostomos is an important contributor. I want them to
> feel that the KinoSearch community appreciated the extensions that they wrote
> and supported them. My preferred mechanism for that would have been to fork
> off Lucy1 and provide them with patches for their extensions to work within
> that namespace. But Peter, at least, feels very strongly that we ought to do
> a "KinoSearch" release. I'm -0.1 on the idea, but I don't think the
> consequences will be severe, and I want Peter to feel like a stakeholder.
>
> Here's the roadmap as I see things now:
>
> KinoSearch 0.30_11
> * Misc Bugfixes.
> * Move some classes around.
> KinoSearch 0.31
> * KS 0.30_11 with a version number increment.
> Lucy 0.1
> * KS 0.31 with a new namespace, new license, and a new home.
> Lucy 0.2
> * Introduce numeric field types.
> Lucy1 1.0
> * Forked from Lucy 0.2.x once things setle down.
>
> The only significant change of plans here is the insertion of KinoSearch 0.31
> into the roadmap. The changes that will go into KS 0.30_11 are the same as
> what we would have done anyway, and I believe the discussions about those
> belong here, as they will directly impact the API of Lucy 0.1. For example,
> I'm about to propose moving all of our Analyzers out of core, like Lucene has.
> I don't think that discussion should take place on the KS list.
>
> I also believe that potential Mentor impatience will help keep us from getting
> sidetracked and spending too much time on pre-transition changes. :)
>
> Marvin Humphrey
>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: [Lucy] Roadmap for first release
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Thu, Jul 29, 2010 at 08:31:27PM -0700, Mattmann, Chris A (388J) wrote:
> Thanks, Peter for the email. I kind of guessed it was more related to
> KinoSearch.
Since Lucy is about to assimilate the KinoSearch code base, they are now
effectively the same project. I was perusing the early days of the
SpamAssassin dev mailing list today to get a feel for how our code import
would go, and I found a thread called "Where to fix the stable branch":
http://markmail.org/message/e5avbm6z2pldv55u
It addressed the problem of making a last bugfix release on GPL'd code in the
2.6 branch, when there was CLA'd code at Apache for 2.7+. I think the
situation is analogous, and that like SpamAssassin, we will move on and leave
the issue behind with time.
> I think it would be good to keep the discussions focused on Apache over
> here, even though I'm aware of the transition as part of the podling.
I appreciate you're keeping our eyes on the prize. I agree that completing
the transition as soon as possible is very important. Frankly, nobody wants
this dualism to end more than I do.
For me, this is mostly about compromise and community. Peter is an important
contributor. Father Chrysostomos is an important contributor. I want them to
feel that the KinoSearch community appreciated the extensions that they wrote
and supported them. My preferred mechanism for that would have been to fork
off Lucy1 and provide them with patches for their extensions to work within
that namespace. But Peter, at least, feels very strongly that we ought to do
a "KinoSearch" release. I'm -0.1 on the idea, but I don't think the
consequences will be severe, and I want Peter to feel like a stakeholder.
Here's the roadmap as I see things now:
KinoSearch 0.30_11
* Misc Bugfixes.
* Move some classes around.
KinoSearch 0.31
* KS 0.30_11 with a version number increment.
Lucy 0.1
* KS 0.31 with a new namespace, new license, and a new home.
Lucy 0.2
* Introduce numeric field types.
Lucy1 1.0
* Forked from Lucy 0.2.x once things setle down.
The only significant change of plans here is the insertion of KinoSearch 0.31
into the roadmap. The changes that will go into KS 0.30_11 are the same as
what we would have done anyway, and I believe the discussions about those
belong here, as they will directly impact the API of Lucy 0.1. For example,
I'm about to propose moving all of our Analyzers out of core, like Lucene has.
I don't think that discussion should take place on the KS list.
I also believe that potential Mentor impatience will help keep us from getting
sidetracked and spending too much time on pre-transition changes. :)
Marvin Humphrey
Re: [Lucy] Roadmap for first release
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Thanks, Peter for the email. I kind of guessed it was more related to KinoSearch. I think it would be good to keep the discussions focused on Apache over here, even though I'm aware of the transition as part of the podling.
Cheers,
Chris
On 7/29/10 8:21 PM, "Peter Karman" <pe...@peknet.com> wrote:
Mattmann, Chris A (388J) wrote on 7/29/10 9:07 PM:
> Guys:
>
> Can you enlighten me as to what this has to do with *Apache* Lucy? And
> furthermore, what it has to do with the *Incubator podling*?
>
Chris,
This thread veered off into the future of KS vis-a-vis Lucy, and probably could
have been moved to the KS list. It's related to Apache Lucy and the Incubator
podling in terms of (a) the feature set that Apache Lucy 1.0 will have, and (b)
the strategy for how to encourage the existing KS community to migrate to Lucy.
But I think the thread is done, unless Marvin has more to add.
pek
--
Peter Karman . http://peknet.com/ . peter@peknet.com
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: [Lucy] Roadmap for first release
Posted by Peter Karman <pe...@peknet.com>.
Mattmann, Chris A (388J) wrote on 7/29/10 9:07 PM:
> Guys:
>
> Can you enlighten me as to what this has to do with *Apache* Lucy? And
> furthermore, what it has to do with the *Incubator podling*?
>
Chris,
This thread veered off into the future of KS vis-a-vis Lucy, and probably could
have been moved to the KS list. It's related to Apache Lucy and the Incubator
podling in terms of (a) the feature set that Apache Lucy 1.0 will have, and (b)
the strategy for how to encourage the existing KS community to migrate to Lucy.
But I think the thread is done, unless Marvin has more to add.
pek
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: [Lucy] Roadmap for first release
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Guys:
Can you enlighten me as to what this has to do with *Apache* Lucy? And
furthermore, what it has to do with the *Incubator podling*?
Thanks,
Chris
On 7/29/10 6:40 PM, "Peter Karman" <pe...@peknet.com> wrote:
> Marvin Humphrey wrote on 7/29/10 8:37 PM:
>> Peter,
>>
>> I'm willing to go with making a KinoSearch 0.31 release. It would also be
>> nice if we could go through the various KSx extensions on CPAN (particularly
>> yours and Father C's) and update them work with it.
>
> great.
>
> I think most of Father C's wildcard features are re-implemented in
> Search::Query::Dialect::KSx which works with the latest KS. So I'm at least
> familiar with his code to that extent.
>
>> I'd also been planning on shutting down the KinoSearch lists and reworking
>> rectangular.com to point everybody at Lucy.... hmmm... In fact, I think we
>> should still do just that: shunt all attention and communication to Lucy, by
>> listing the Lucy website and mailing lists in the KinoSearch documentation
>> under "SUPPORT".
>
> +1
>
> --
> Peter Karman . http://peknet.com/ . peter@peknet.com
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: [Lucy] Roadmap for first release
Posted by Peter Karman <pe...@peknet.com>.
Marvin Humphrey wrote on 7/29/10 8:37 PM:
> Peter,
>
> I'm willing to go with making a KinoSearch 0.31 release. It would also be
> nice if we could go through the various KSx extensions on CPAN (particularly
> yours and Father C's) and update them work with it.
great.
I think most of Father C's wildcard features are re-implemented in
Search::Query::Dialect::KSx which works with the latest KS. So I'm at least
familiar with his code to that extent.
> I'd also been planning on shutting down the KinoSearch lists and reworking
> rectangular.com to point everybody at Lucy.... hmmm... In fact, I think we
> should still do just that: shunt all attention and communication to Lucy, by
> listing the Lucy website and mailing lists in the KinoSearch documentation
> under "SUPPORT".
+1
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: [Lucy] Roadmap for first release
Posted by Marvin Humphrey <ma...@rectangular.com>.
Peter,
I'm willing to go with making a KinoSearch 0.31 release. It would also be
nice if we could go through the various KSx extensions on CPAN (particularly
yours and Father C's) and update them work with it.
Releasing KS 0.31 would follow through on at least part of the roadmap we'd
laid out for KinoSearch prior to the Lucy assimilation plan's genesis. People
have known that we were going to update "KinoSearch" and break back compat,
and KinoSearch1 has been made available in order to make the transition more
convenient.
The only change of plan is that we would expect to declare KinoSearch 0.3x
EOL'd so that it becomes the effective stable branch rather than forking off
KinoSearch3 for that purpose.
I still have some reservations, though.
> I think a KS release does two things:
>
> (1) it says to the existing community that they have not spent years waiting in
> vain for a stable, production-ready, blessed release of KS
I guess I've been seeing Lucy as a continuation of KinoSearch now that Lucy's
going to assimilate the KinoSearch code base. So in my mind, Lucy1 *was* to
be that stable, production-ready, blessed release -- it had taken the place of
KinoSearch3.
Anyway, that was my thinking. It's moot now.
> You wrote earlier in this thread:
>
> It probably makes sense to make one more KinoSearch release
> addressing some of the issues for the transition.
>
> I am agreeing with that. I think it should either be KinoSearch3, or KinoSearch
> 0.3, I don't care which. KinoSearch 0.3 seems like it would be less confusing.
Well, what I'd meant by "one more KinoSearch release" was a final dev
release. :\
> I don't think a stable KS release undermines Lucy's marketing efforts.
I do think a stable release of KS will compete with Lucy at first -- users
will naturally go with the stable KinoSearch rather than the unstable Lucy.
When somebody asks on PerlMonks "which search library should I use", the
recommendation will be "KinoSearch" over "Lucy" -- at least until we fork a
stable Lucy1 release that improves on KinoSearch.
However, that will only impact the immediate future, I suppose. In time, the
accumulation of improvements to Lucy will persuade people to come over.
> We could make a final KS release tomorrow, handshakes and champagne all around,
> and move on.
If it's successful, I'd expect bug reports. :) And possibly patches. We'd
have to figure out how to accept those without causing licensing problems --
probably by shunting them through Lucy's JIRA.
I'd also been planning on shutting down the KinoSearch lists and reworking
rectangular.com to point everybody at Lucy.... hmmm... In fact, I think we
should still do just that: shunt all attention and communication to Lucy, by
listing the Lucy website and mailing lists in the KinoSearch documentation
under "SUPPORT".
Marvin Humphrey
Re: [Lucy] Roadmap for first release
Posted by Peter Karman <pe...@peknet.com>.
Marvin Humphrey wrote on 7/23/10 3:27 PM:
> On Fri, Jul 23, 2010 at 11:00:58AM -0500, Peter Karman wrote:
>> those all sound good for Lucy. Should not impede the KS3 release though. I
>> imagine Lucy1 as an improvement on KS3, inspiring users to migrate.
>
> Forking and releasing KS3 is not a huge development burden in the grand scheme
> of things, but I'm not sure it's wise from a marketing perspective.
I think a KS release does two things:
(1) it says to the existing community that they have not spent years waiting in
vain for a stable, production-ready, blessed release of KS, and
(2) it is consistent with release early and often.
You wrote earlier in this thread:
It probably makes sense to make one more KinoSearch release
addressing some of the issues for the transition.
I am agreeing with that. I think it should either be KinoSearch3, or KinoSearch
0.3, I don't care which. KinoSearch 0.3 seems like it would be less confusing.
I don't think a stable KS release undermines Lucy's marketing efforts. Working,
stable code is a good thing. The KS release could clearly state in its
documentation that it represents several years of development effort,
culminating in a final, stable release of the mmap-based index format, and that
future development will be on Lucy. Anyone who has paid a minute's attention to
the KS project over the last few years knows about this "Lucy thing" but the
timing and schedule has all been a bit hand-wavey. So there's no surprise here;
instead there are definites: a final stable KS release, and adoption by the
Apache Incubator of the Lucy fork of KS.
We could make a final KS release tomorrow, handshakes and champagne all around,
and move on.
>
> From my perspective, the sooner that KinoSearch gets EOL'd and we can focus on
> Lucy development in earnest, the better. I wouldn't mind having some extra
> pressure to release Lucy1 ASAP because we didn't release KinoSearch3. :)
I agree with that first sentence. I think we're negotiating what "EOL'd" means.
I think it should mean a 0.3 release, with no more releases except for security
fixes.
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: [Lucy] Roadmap for first release
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Fri, Jul 23, 2010 at 11:00:58AM -0500, Peter Karman wrote:
> those all sound good for Lucy. Should not impede the KS3 release though. I
> imagine Lucy1 as an improvement on KS3, inspiring users to migrate.
Forking and releasing KS3 is not a huge development burden in the grand scheme
of things, but I'm not sure it's wise from a marketing perspective. Do we
really want to introduce this codebase under the name KinoSearch3 and then
deprecate it right away? Wouldn't that be confusing to our users? What if
we're lucky enough to get some press, articles get written about KinoSearch3,
users subscribe to the KinoSearch mailing list, and links get pointed at
rectangular.com? Won't we be diffusing some of that goodwill by trying to
redirect it towards Lucy later? Would all of those prospective KinoSearch3
users be turned off by its imminent deprecation and be less enthusiastic about
adopting it in the first place? Would we be squandering our big feature
rollout of mmap-powered near-real-time search by bundling it with a
soon-to-be-deprecated library, making Lucy's release less momentous?
As far as I can see, the only advantage KinoSearch3 has over Lucy1 is that it
could happen slightly sooner. A KinoSearch3 release won't automatically help
people like Father Chrysostomos and yourself who have released extensions --
you'll have to adapt your existing distros to a new namespace regardless, and
"KinoSearch3" doesn't seem to offer any compelling advantages over "Lucy1".
In fact, from Lucy's perspective, it's a negative to have the developer
community fractured and supporting incompatible extensions.
Is there something I haven't thought of? Is there some reason other than
timing that you want to see a KinoSearch3 release?
>From my perspective, the sooner that KinoSearch gets EOL'd and we can focus on
Lucy development in earnest, the better. I wouldn't mind having some extra
pressure to release Lucy1 ASAP because we didn't release KinoSearch3. :)
Marvin Humphrey