You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Venkatesh Seetharam <vs...@gmail.com> on 2007/03/07 22:35:00 UTC
Solr and Multiple Index Partitions
Hello there,
Howdy. I was wondering if there is a way to configure one Solr instance to
search multiple Index partitions? I read the wiki and found the entry in
SolrConfig.xml:
<dataDir>/var/data/solr</dataDir>
Can I have mutiple directories? Comma separated?
Any help is greatly appreciated.
--
Thanks,
Venkatesh
"Perfection (in design) is achieved not when there is nothing more to add,
but rather when there is nothing more to take away."
- Antoine de Saint-Exupéry
Re: Solr and Multiple Index Partitions
Posted by Venkatesh Seetharam <vs...@gmail.com>.
Thanks Chris for a wonderful explanation. I completely get it now. Thanks
for the handy URL too.
Venkatesh
On 3/8/07, Chris Hostetter <ho...@fucit.org> wrote:
>
>
> : I use a custom Analyzer which extends Lucene's StandardAnalyzer. When I
> : configured Solr to use this one, It throws an exception
> : RuntimeException("Can't set positionIncrementGap on custom analyzer " +
> : analyzer.getClass()).
> :
> : Do I need to extend a specific Analyzer for it to work with Solr?
>
> you can use any Analyzer you want, but you can't configure a
> positionIncrementGap in the schema.xml unless your Analyzer extends
> SolrAnalyzer (the concept of a position increment gap is an inherient
> property that Lucene Analyzers can specify, but configuring it explicitly
> is a Solr concept)
>
>
>
>
> -Hoss
>
>
Re: Solr and Multiple Index Partitions
Posted by Chris Hostetter <ho...@fucit.org>.
whoops .. forgot the documentaiton link...
http://wiki.apache.org/solr/SolrPlugins#head-9939da9abe85a79eb30a026e85cc4aec0beac10c
: you can use any Analyzer you want, but you can't configure a
: positionIncrementGap in the schema.xml unless your Analyzer extends
: SolrAnalyzer (the concept of a position increment gap is an inherient
: property that Lucene Analyzers can specify, but configuring it explicitly
: is a Solr concept)
-Hoss
Re: Solr and Multiple Index Partitions
Posted by Chris Hostetter <ho...@fucit.org>.
: I use a custom Analyzer which extends Lucene's StandardAnalyzer. When I
: configured Solr to use this one, It throws an exception
: RuntimeException("Can't set positionIncrementGap on custom analyzer " +
: analyzer.getClass()).
:
: Do I need to extend a specific Analyzer for it to work with Solr?
you can use any Analyzer you want, but you can't configure a
positionIncrementGap in the schema.xml unless your Analyzer extends
SolrAnalyzer (the concept of a position increment gap is an inherient
property that Lucene Analyzers can specify, but configuring it explicitly
is a Solr concept)
-Hoss
Re: Solr and Multiple Index Partitions
Posted by Venkatesh Seetharam <vs...@gmail.com>.
Thanks Ryan for your insight. I do not wish to change Solr.
> Although it seems weird, just adding a "type" field makes it possible
I'm not working with a DB but since we are indexing a huge vault of XML
documents, I use Hadoop + Lucene for indexing which finally generates N
number of partitions, I was thinking to use Solr as the searcher and will
have a broker which merges results from individual Solr searchers.
I do not know if I can ask another question here or start a new thread.
I use a custom Analyzer which extends Lucene's StandardAnalyzer. When I
configured Solr to use this one, It throws an exception
RuntimeException("Can't set positionIncrementGap on custom analyzer " +
analyzer.getClass()).
Do I need to extend a specific Analyzer for it to work with Solr?
Venkatesh
On 3/7/07, Ryan McKinley <ry...@gmail.com> wrote:
>
> As it is now... I don't think so. SolrCore is a static singleton
> class -- without some serious reworking, i think there is only one
> instance per jvm.
>
> I think getting rid of the static singleton should go on the long term
> TODO list, but that doesn't help you now.
>
> Although it seems weird, just adding a "type" field makes it possible
> to put things that seem like they need their own database into the
> same index - you select the 'type' you want by adding "+type:mytype"
> to your query
>
>
> On 3/7/07, Venkatesh Seetharam <vs...@gmail.com> wrote:
> > Thanks Ryan for your inputs. If I'm not using Solr webapp but wrapping
> Solr
> > in plain Java, is there any way that I could get Solr to work with
> multiple
> > index partitions?
> >
> > Venkatesh
> >
> > On 3/7/07, Ryan McKinley <ry...@gmail.com> wrote:
> > >
> > > Solr looks at one index - If you want to look at multiple indexes, you
> > > need multiple solr instances running. Check the wiki for how to set
> > > that up:
> > >
> > > http://wiki.apache.org/solr/SolrJetty
> > >
> > > (the resin and tomcat pages have something similar)
> > >
> > >
> > >
> > > On 3/7/07, Venkatesh Seetharam <vs...@gmail.com> wrote:
> > > > Hello there,
> > > >
> > > > Howdy. I was wondering if there is a way to configure one Solr
> instance
> > > to
> > > > search multiple Index partitions? I read the wiki and found the
> entry in
> > > > SolrConfig.xml:
> > > >
> > > > <dataDir>/var/data/solr</dataDir>
> > > >
> > > > Can I have mutiple directories? Comma separated?
> > > >
> > > > Any help is greatly appreciated.
> > > >
> > > > --
> > > > Thanks,
> > > > Venkatesh
> > > >
> > > > "Perfection (in design) is achieved not when there is nothing more
> to
> > > add,
> > > > but rather when there is nothing more to take away."
> > > > - Antoine de Saint-Exupéry
> > > >
> > >
> >
>
Re: Re[2]: Solr and Multiple Index Partitions
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Mar 7, 2007, at 9:20 PM, Jack L wrote:
> Selecting by type will do the job. But I suppose it sacrifice
> performance because having multiple document types in the same
> index will render a larger index. Is it bad?
A many documents we talking here?
My hunch is you'll be fine :)
Erik
Re[2]: Solr and Multiple Index Partitions
Posted by Jack L <jl...@yahoo.ca>.
Selecting by type will do the job. But I suppose it sacrifice
performance because having multiple document types in the same
index will render a larger index. Is it bad?
--
Best regards,
Jack
Wednesday, March 7, 2007, 2:15:14 PM, you wrote:
> As it is now... I don't think so. SolrCore is a static singleton
> class -- without some serious reworking, i think there is only one
> instance per jvm.
> I think getting rid of the static singleton should go on the long term
> TODO list, but that doesn't help you now.
> Although it seems weird, just adding a "type" field makes it possible
> to put things that seem like they need their own database into the
> same index - you select the 'type' you want by adding "+type:mytype"
> to your query
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Re: Solr and Multiple Index Partitions
Posted by Ryan McKinley <ry...@gmail.com>.
As it is now... I don't think so. SolrCore is a static singleton
class -- without some serious reworking, i think there is only one
instance per jvm.
I think getting rid of the static singleton should go on the long term
TODO list, but that doesn't help you now.
Although it seems weird, just adding a "type" field makes it possible
to put things that seem like they need their own database into the
same index - you select the 'type' you want by adding "+type:mytype"
to your query
On 3/7/07, Venkatesh Seetharam <vs...@gmail.com> wrote:
> Thanks Ryan for your inputs. If I'm not using Solr webapp but wrapping Solr
> in plain Java, is there any way that I could get Solr to work with multiple
> index partitions?
>
> Venkatesh
>
> On 3/7/07, Ryan McKinley <ry...@gmail.com> wrote:
> >
> > Solr looks at one index - If you want to look at multiple indexes, you
> > need multiple solr instances running. Check the wiki for how to set
> > that up:
> >
> > http://wiki.apache.org/solr/SolrJetty
> >
> > (the resin and tomcat pages have something similar)
> >
> >
> >
> > On 3/7/07, Venkatesh Seetharam <vs...@gmail.com> wrote:
> > > Hello there,
> > >
> > > Howdy. I was wondering if there is a way to configure one Solr instance
> > to
> > > search multiple Index partitions? I read the wiki and found the entry in
> > > SolrConfig.xml:
> > >
> > > <dataDir>/var/data/solr</dataDir>
> > >
> > > Can I have mutiple directories? Comma separated?
> > >
> > > Any help is greatly appreciated.
> > >
> > > --
> > > Thanks,
> > > Venkatesh
> > >
> > > "Perfection (in design) is achieved not when there is nothing more to
> > add,
> > > but rather when there is nothing more to take away."
> > > - Antoine de Saint-Exupéry
> > >
> >
>
Re: Solr and Multiple Index Partitions
Posted by Venkatesh Seetharam <vs...@gmail.com>.
Yes, I'm implementing federated search. I do have N partitions of indexes
built and I'd like to have mutiple Solr instances in a cluster each serving
atleast 2 partitions. I was wondering if I could somehow find a way to make
Solr work with atleast 2 partitions. It looks like I need to have smaller
number of partitions.
Thanks,
Venkatesh
On 3/7/07, Bradley Burke <bb...@gmail.com> wrote:
>
> Venkatesh,
>
> Are you perhapes talking about Federated Searching (
> http://wiki.apache.org/solr/FederatedSearch). I am new to Solr but this
> was
> a feature I was looking far as well. I do not think its built into Solr
> at
> the momment.
>
> Regards,
> Brad
>
>
> On 3/7/07, Venkatesh Seetharam <vs...@gmail.com> wrote:
> >
> > Thanks Ryan for your inputs. If I'm not using Solr webapp but wrapping
> > Solr
> > in plain Java, is there any way that I could get Solr to work with
> > multiple
> > index partitions?
> >
> > Venkatesh
> >
> > On 3/7/07, Ryan McKinley <ry...@gmail.com> wrote:
> > >
> > > Solr looks at one index - If you want to look at multiple indexes, you
> > > need multiple solr instances running. Check the wiki for how to set
> > > that up:
> > >
> > > http://wiki.apache.org/solr/SolrJetty
> > >
> > > (the resin and tomcat pages have something similar)
> > >
> > >
> > >
> > > On 3/7/07, Venkatesh Seetharam <vs...@gmail.com> wrote:
> > > > Hello there,
> > > >
> > > > Howdy. I was wondering if there is a way to configure one Solr
> > instance
> > > to
> > > > search multiple Index partitions? I read the wiki and found the
> entry
> > in
> > > > SolrConfig.xml:
> > > >
> > > > <dataDir>/var/data/solr</dataDir>
> > > >
> > > > Can I have mutiple directories? Comma separated?
> > > >
> > > > Any help is greatly appreciated.
> > > >
> > > > --
> > > > Thanks,
> > > > Venkatesh
> > > >
> > > > "Perfection (in design) is achieved not when there is nothing more
> to
> > > add,
> > > > but rather when there is nothing more to take away."
> > > > - Antoine de Saint-Exupéry
> > > >
> > >
> >
>
Re: Solr and Multiple Index Partitions
Posted by Bradley Burke <bb...@gmail.com>.
Venkatesh,
Are you perhapes talking about Federated Searching (
http://wiki.apache.org/solr/FederatedSearch). I am new to Solr but this was
a feature I was looking far as well. I do not think its built into Solr at
the momment.
Regards,
Brad
On 3/7/07, Venkatesh Seetharam <vs...@gmail.com> wrote:
>
> Thanks Ryan for your inputs. If I'm not using Solr webapp but wrapping
> Solr
> in plain Java, is there any way that I could get Solr to work with
> multiple
> index partitions?
>
> Venkatesh
>
> On 3/7/07, Ryan McKinley <ry...@gmail.com> wrote:
> >
> > Solr looks at one index - If you want to look at multiple indexes, you
> > need multiple solr instances running. Check the wiki for how to set
> > that up:
> >
> > http://wiki.apache.org/solr/SolrJetty
> >
> > (the resin and tomcat pages have something similar)
> >
> >
> >
> > On 3/7/07, Venkatesh Seetharam <vs...@gmail.com> wrote:
> > > Hello there,
> > >
> > > Howdy. I was wondering if there is a way to configure one Solr
> instance
> > to
> > > search multiple Index partitions? I read the wiki and found the entry
> in
> > > SolrConfig.xml:
> > >
> > > <dataDir>/var/data/solr</dataDir>
> > >
> > > Can I have mutiple directories? Comma separated?
> > >
> > > Any help is greatly appreciated.
> > >
> > > --
> > > Thanks,
> > > Venkatesh
> > >
> > > "Perfection (in design) is achieved not when there is nothing more to
> > add,
> > > but rather when there is nothing more to take away."
> > > - Antoine de Saint-Exupéry
> > >
> >
>
Re: Solr and Multiple Index Partitions
Posted by Venkatesh Seetharam <vs...@gmail.com>.
Thanks Ryan for your inputs. If I'm not using Solr webapp but wrapping Solr
in plain Java, is there any way that I could get Solr to work with multiple
index partitions?
Venkatesh
On 3/7/07, Ryan McKinley <ry...@gmail.com> wrote:
>
> Solr looks at one index - If you want to look at multiple indexes, you
> need multiple solr instances running. Check the wiki for how to set
> that up:
>
> http://wiki.apache.org/solr/SolrJetty
>
> (the resin and tomcat pages have something similar)
>
>
>
> On 3/7/07, Venkatesh Seetharam <vs...@gmail.com> wrote:
> > Hello there,
> >
> > Howdy. I was wondering if there is a way to configure one Solr instance
> to
> > search multiple Index partitions? I read the wiki and found the entry in
> > SolrConfig.xml:
> >
> > <dataDir>/var/data/solr</dataDir>
> >
> > Can I have mutiple directories? Comma separated?
> >
> > Any help is greatly appreciated.
> >
> > --
> > Thanks,
> > Venkatesh
> >
> > "Perfection (in design) is achieved not when there is nothing more to
> add,
> > but rather when there is nothing more to take away."
> > - Antoine de Saint-Exupéry
> >
>
Re: Solr and Multiple Index Partitions
Posted by Ryan McKinley <ry...@gmail.com>.
Solr looks at one index - If you want to look at multiple indexes, you
need multiple solr instances running. Check the wiki for how to set
that up:
http://wiki.apache.org/solr/SolrJetty
(the resin and tomcat pages have something similar)
On 3/7/07, Venkatesh Seetharam <vs...@gmail.com> wrote:
> Hello there,
>
> Howdy. I was wondering if there is a way to configure one Solr instance to
> search multiple Index partitions? I read the wiki and found the entry in
> SolrConfig.xml:
>
> <dataDir>/var/data/solr</dataDir>
>
> Can I have mutiple directories? Comma separated?
>
> Any help is greatly appreciated.
>
> --
> Thanks,
> Venkatesh
>
> "Perfection (in design) is achieved not when there is nothing more to add,
> but rather when there is nothing more to take away."
> - Antoine de Saint-Exupéry
>