[lucy-dev] RFC: LucyX::Suggester

[Peter Karman <pe...@peknet.com>]

Cooked this up tonight and would appreciate comments on concept, design, ways to
improve, etc.

https://github.com/karpet/lucyx-suggester


-- 
Peter Karman  .  http://peknet.com/  .  peter@peknet.com

Re: [lucy-dev] RFC: LucyX::Suggester

[Peter Karman <pe...@peknet.com>]

Marvin Humphrey wrote on 10/16/12 5:42 PM:
> On Mon, Oct 15, 2012 at 9:34 PM, Peter Karman <pe...@peknet.com> wrote:
>> Cooked this up tonight and would appreciate comments on concept, design, ways to
>> improve, etc.
>>
>> https://github.com/karpet/lucyx-suggester
> 
> It is possible to write much more elaborate suggesters (and Lucene bundles
> lots: <http://lucene.apache.org/core/4_0_0/suggest/index.html>).

Yes, I took the name from Lucene/Solr. To be honest, I quickly found myself lost
trying to understand the Lucene implementations.

> 
> However, in my experience, suggestions are just as sensitive as search
> results, requiring elaborate weighting and control.  People often have strong
> opinions about what the suggestions ought to be and can become quite uneasy
> when what shows up doesn't match their expectations.
> 
> Therefore, to scale upwards, there is a significant benefit in creating a
> dedicated index to handle suggestions, derived from the content of the primary
> index.  You don't get to use the same disk space, but that's immaterial these
> days -- the important thing is to make the best use of programmer time and
> skillset when tuning the suggestions, and a satellite index is a good data
> structure for that task.
> 
> IMO a simple suggester like this one hits the sweet spot for coding
> return-on-investment for a general tool, since highly engineered tools often
> prove unsatisfactory in the general case.  Scaling up would best be achieved
> through a cookbook entry illustrating how to implement suggestions using a
> search index, rather than by trying to build a one-size-fits-all super-awesome
> fast-and-relevant suggester.
> 

+1 to all that. I would love to see a Cookbook entry on the subject.

Thanks for the feedback, Marvin.


-- 
Peter Karman  .  http://peknet.com/  .  peter@peknet.com

Re: [lucy-dev] RFC: LucyX::Suggester

[Marvin Humphrey <ma...@rectangular.com>]

On Mon, Oct 15, 2012 at 9:34 PM, Peter Karman <pe...@peknet.com> wrote:
> Cooked this up tonight and would appreciate comments on concept, design, ways to
> improve, etc.
>
> https://github.com/karpet/lucyx-suggester

It is possible to write much more elaborate suggesters (and Lucene bundles
lots: <http://lucene.apache.org/core/4_0_0/suggest/index.html>).

However, in my experience, suggestions are just as sensitive as search
results, requiring elaborate weighting and control.  People often have strong
opinions about what the suggestions ought to be and can become quite uneasy
when what shows up doesn't match their expectations.

Therefore, to scale upwards, there is a significant benefit in creating a
dedicated index to handle suggestions, derived from the content of the primary
index.  You don't get to use the same disk space, but that's immaterial these
days -- the important thing is to make the best use of programmer time and
skillset when tuning the suggestions, and a satellite index is a good data
structure for that task.

IMO a simple suggester like this one hits the sweet spot for coding
return-on-investment for a general tool, since highly engineered tools often
prove unsatisfactory in the general case.  Scaling up would best be achieved
through a cookbook entry illustrating how to implement suggestions using a
search index, rather than by trying to build a one-size-fits-all super-awesome
fast-and-relevant suggester.

Marvin Humphrey

Re: [lucy-dev] RFC: LucyX::Suggester

["David E. Wheeler" <da...@justatheory.com>]

On Oct 16, 2012, at 8:38 PM, Peter Karman <pe...@peknet.com> wrote:

>> Ooh, nice. I will use this in PGXN shortly after you release it.
> 
> 0.001 just uploaded to CPAN. Let me know how it works for you.

Well, not *that* shortly. :-)

I need to find some time in the next couple of months to give PGXN a little love…

David

Re: [lucy-dev] RFC: LucyX::Suggester

[Peter Karman <pe...@peknet.com>]

David E. Wheeler wrote on 10/16/12 11:02 AM:
> On Oct 15, 2012, at 9:34 PM, Peter Karman <pe...@peknet.com> wrote:
> 
>> Cooked this up tonight and would appreciate comments on concept, design, ways to
>> improve, etc.
>>
>> https://github.com/karpet/lucyx-suggester
> 
> Ooh, nice. I will use this in PGXN shortly after you release it.
> 


0.001 just uploaded to CPAN. Let me know how it works for you.


-- 
Peter Karman  .  http://peknet.com/  .  peter@peknet.com

Re: [lucy-dev] RFC: LucyX::Suggester

["David E. Wheeler" <da...@justatheory.com>]

On Oct 15, 2012, at 9:34 PM, Peter Karman <pe...@peknet.com> wrote:

> Cooked this up tonight and would appreciate comments on concept, design, ways to
> improve, etc.
> 
> https://github.com/karpet/lucyx-suggester

Ooh, nice. I will use this in PGXN shortly after you release it.

Best,

David