faster example schema

[Yonik Seeley <yo...@lucidimagination.com>]

I've occasionally run across people going with another search engine
because it was faster at indexing.
The example schema that people may be using as a base to do their
benchmarking (with perhaps minimal modifications) is slow.
There are many people out there that check what's fastest first, and
*then* check if it is satisfactory to meet their needs in other areas.

With very simple synthetic test documents (just a few fields each) and
the CSV loader, I've personally seen the indexing rate go from
~330/sec to ~3000/sec, when I removed the default field values, term
vectors, copyFields, etc.  The default example schema should still be
able to show how something can be done, but that doesn't mean it needs
to be enabled by default.

So what do people think about speeding up the default/example schema before 1.4?

-Yonik
http://www.lucidimagination.com

Re: faster example schema

[Michael McCandless <lu...@mikemccandless.com>]

+1

I think appropriate selection of defaults, biasing for performance in  
this
case, is extremely important yet often overlooked.

The out-of-the-box first-impression experience that people see, with  
Solr
or Lucene, often decides whether Solr or Lucene should be considered
for an application.

Mike

Yonik Seeley wrote:

> I've occasionally run across people going with another search engine
> because it was faster at indexing.
> The example schema that people may be using as a base to do their
> benchmarking (with perhaps minimal modifications) is slow.
> There are many people out there that check what's fastest first, and
> *then* check if it is satisfactory to meet their needs in other areas.
>
> With very simple synthetic test documents (just a few fields each) and
> the CSV loader, I've personally seen the indexing rate go from
> ~330/sec to ~3000/sec, when I removed the default field values, term
> vectors, copyFields, etc.  The default example schema should still be
> able to show how something can be done, but that doesn't mean it needs
> to be enabled by default.
>
> So what do people think about speeding up the default/example schema  
> before 1.4?
>
> -Yonik
> http://www.lucidimagination.com

Re: faster example schema

[Grant Ingersoll <gs...@apache.org>]

+1.


On Mar 7, 2009, at 10:12 AM, Yonik Seeley wrote:

> I've occasionally run across people going with another search engine
> because it was faster at indexing.
> The example schema that people may be using as a base to do their
> benchmarking (with perhaps minimal modifications) is slow.
> There are many people out there that check what's fastest first, and
> *then* check if it is satisfactory to meet their needs in other areas.
>
> With very simple synthetic test documents (just a few fields each) and
> the CSV loader, I've personally seen the indexing rate go from
> ~330/sec to ~3000/sec, when I removed the default field values, term
> vectors, copyFields, etc.  The default example schema should still be
> able to show how something can be done, but that doesn't mean it needs
> to be enabled by default.
>
> So what do people think about speeding up the default/example schema  
> before 1.4?
>
> -Yonik
> http://www.lucidimagination.com

Re: faster example schema

[Chris Hostetter <ho...@fucit.org>]

: perhaps we want a "schema.xml"  and a "schema-kitchen-sink.xml"
: 
: While it is good that the default is fast, we also want to make sure
: everything has a functioning example somewhere.

if we follow through with some of hte ideas discussed in the past for 
better organizing the various example config setups, this would be fairly 
easy to do...

http://www.nabble.com/Re%3A-Cleaning-up-a-Few-things-p20670765.html

... examples/simple could be the default, with just some barebones stuff, 
and examples/kitchen-sink could show off every concievably useful 
combination of fieldtypes, while examples/* could represent other good 
niches (multicore, tika, dih, etc...) in their simple forms.



-Hoss

Re: faster example schema

[Ryan McKinley <ry...@gmail.com>]

+1

perhaps we want a "schema.xml"  and a "schema-kitchen-sink.xml"

While it is good that the default is fast, we also want to make sure  
everything has a functioning example somewhere.


On Mar 7, 2009, at 10:12 AM, Yonik Seeley wrote:

> I've occasionally run across people going with another search engine
> because it was faster at indexing.
> The example schema that people may be using as a base to do their
> benchmarking (with perhaps minimal modifications) is slow.
> There are many people out there that check what's fastest first, and
> *then* check if it is satisfactory to meet their needs in other areas.
>
> With very simple synthetic test documents (just a few fields each) and
> the CSV loader, I've personally seen the indexing rate go from
> ~330/sec to ~3000/sec, when I removed the default field values, term
> vectors, copyFields, etc.  The default example schema should still be
> able to show how something can be done, but that doesn't mean it needs
> to be enabled by default.
>
> So what do people think about speeding up the default/example schema  
> before 1.4?
>
> -Yonik
> http://www.lucidimagination.com

Re: faster example schema

[Otis Gospodnetic <ot...@yahoo.com>]

+1


I think we could try just comment out the kitchen sick portions and avoid maintaining 2 config files.
 
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----
> From: Shalin Shekhar Mangar <sh...@gmail.com>
> To: solr-dev@lucene.apache.org; yonik@lucidimagination.com
> Sent: Sunday, March 8, 2009 1:20:50 AM
> Subject: Re: faster example schema
> 
> +1
> 
> perhaps schema.xml and schema-example.xml?
> 
> On Sat, Mar 7, 2009 at 8:42 PM, Yonik Seeley wrote:
> 
> > I've occasionally run across people going with another search engine
> > because it was faster at indexing.
> > The example schema that people may be using as a base to do their
> > benchmarking (with perhaps minimal modifications) is slow.
> > There are many people out there that check what's fastest first, and
> > *then* check if it is satisfactory to meet their needs in other areas.
> >
> > With very simple synthetic test documents (just a few fields each) and
> > the CSV loader, I've personally seen the indexing rate go from
> > ~330/sec to ~3000/sec, when I removed the default field values, term
> > vectors, copyFields, etc.  The default example schema should still be
> > able to show how something can be done, but that doesn't mean it needs
> > to be enabled by default.
> >
> > So what do people think about speeding up the default/example schema before
> > 1.4?
> >
> > -Yonik
> > http://www.lucidimagination.com
> >
> 
> 
> 
> -- 
> Regards,
> Shalin Shekhar Mangar.

Re: faster example schema

[Shalin Shekhar Mangar <sh...@gmail.com>]

+1

perhaps schema.xml and schema-example.xml?

On Sat, Mar 7, 2009 at 8:42 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:

> I've occasionally run across people going with another search engine
> because it was faster at indexing.
> The example schema that people may be using as a base to do their
> benchmarking (with perhaps minimal modifications) is slow.
> There are many people out there that check what's fastest first, and
> *then* check if it is satisfactory to meet their needs in other areas.
>
> With very simple synthetic test documents (just a few fields each) and
> the CSV loader, I've personally seen the indexing rate go from
> ~330/sec to ~3000/sec, when I removed the default field values, term
> vectors, copyFields, etc.  The default example schema should still be
> able to show how something can be done, but that doesn't mean it needs
> to be enabled by default.
>
> So what do people think about speeding up the default/example schema before
> 1.4?
>
> -Yonik
> http://www.lucidimagination.com
>



-- 
Regards,
Shalin Shekhar Mangar.