SearchComponents and Distributed Search

[Grant Ingersoll <gs...@apache.org>]

Seems like distributed search is starting to hit some critical mass  
(there has been a whole bunch of patches to make components  
distributed lately.)

I think it is reasonable to say, at this point, that we should now  
require any new SearchComponents to be distributed capable before  
committing.  To help with this, I think we need to shore up http://wiki.apache.org/solr/WritingDistributedSearchComponents 
, especially with information about testing.

WDYT?

-Grant

Re: SearchComponents and Distributed Search

[Grant Ingersoll <gs...@apache.org>]

On Jun 19, 2009, at 6:35 PM, Chris Hostetter wrote:

>
> On Fri, 22 May 2009, Yonik Seeley wrote:
>
> : > I think it is reasonable to say, at this point, that we should  
> now require
> : > any new SearchComponents to be distributed capable before  
> committing.
> :
> : That's too high of a bar.
> : If something is good and useful, it should be committed.
> : And we certainly never want to discourage anyone from submitting a
> : patch just because it lacks distributed search support.
>
> +1 ... as long as components are well documented about what they do/ 
> don't
> support i think we're fine.

If it is going to be a default component, I don't agree.  It should be  
distributed before committed.

If it's an add-on, then agreed, although I still think we are just  
asking for headaches.

-Grant

Re: SearchComponents and Distributed Search

[Chris Hostetter <ho...@fucit.org>]

On Fri, 22 May 2009, Yonik Seeley wrote:

: > I think it is reasonable to say, at this point, that we should now require
: > any new SearchComponents to be distributed capable before committing.
: 
: That's too high of a bar.
: If something is good and useful, it should be committed.
: And we certainly never want to discourage anyone from submitting a
: patch just because it lacks distributed search support.

+1 ... as long as components are well documented about what they do/don't 
support i think we're fine.  

The real key is to make sure we don't have any new components that *look* 
like they work fine in distributed search, but are silently ignoring 
certain options, or skipping steps in some calculation.  (if i remember 
correctly there are some features of hte FacetComponent that violate this 
principle)

if it looks like it worked when you do a distributed query, it should 
really work, if it doesn't work it should error. 



-Hoss

Re: SearchComponents and Distributed Search

[Yonik Seeley <ys...@gmail.com>]

On Fri, May 22, 2009 at 6:50 AM, Grant Ingersoll <gs...@apache.org> wrote:
> Seems like distributed search is starting to hit some critical mass (there
> has been a whole bunch of patches to make components distributed lately.)
>
> I think it is reasonable to say, at this point, that we should now require
> any new SearchComponents to be distributed capable before committing.

That's too high of a bar.
If something is good and useful, it should be committed.
And we certainly never want to discourage anyone from submitting a
patch just because it lacks distributed search support.
Of course the external API of any new components should certainly be
mindful not do do anything to preclude distributed search.

> To help with this, I think we need to shore up
> http://wiki.apache.org/solr/WritingDistributedSearchComponents, especially
> with information about testing.

Anyone looking at this should also look to simplify / tweak / clean-up
the actual code/APIs involved too (ResponseBuilder and friends).  It's
really a first cut, and could be refined a lot before being set in
stone too much.  That doesn't mean that the work people are doing to
enable other components for distributed search should stop... just
that they should keep a lookout for how to improve the distributed
search process rather than just work around any shortcomings that are
found.

-Yonik
http://www.lucidimagination.com