Best way to create multiple indexes

[Rishabh Joshi <ri...@coreobjects.com>]

Hi,

I have a requirement and was wondering if someone could help me in how to go about it. We have to index about 8-9 million documents and their size can be anywhere from a few KBs to a couple of MBs. These documents are categorized into many 'groups' and 'sub-groups'. I wanted to know if we can create multiple indexes based on 'groups' and then on 'sub-groups' in Solr? If yes, then how do we go about it? I tried going through the section on 'Collections' in the Solr Wiki, but could not make much use of it.

Regards,
Rishabh Joshi

Re: Best way to create multiple indexes

[Dwarak R <dw...@solutionnet.net>]

Hi Guys

How do we add word documents / pdf / text / etc documents in solr ?. How the 
content of the files are stored or indexed ?. Does the documents are stored 
as XML in the filesystem ?

Regards

Dwarak R
----- Original Message ----- 
From: "Ryan McKinley" <ry...@gmail.com>
To: <so...@lucene.apache.org>
Sent: Monday, November 12, 2007 7:43 PM
Subject: Re: Best way to create multiple indexes


> For starters, do you need to be able to search across groups or sub-groups 
> (in one query?)
>
> If so, then you have to stick everything in one index.
>
> You can add a field to each document saying what 'group' or 'sub-group' it 
> is in and then limit it at query time
>
>  q="kittens +group:A"
>
> The advantage to splitting it into multiple indexes is that you could put 
> each index on independent hardware.  Depending on your queries and index 
> size that may make a big difference.
>
> ryan
>
>
> Rishabh Joshi wrote:
>> Hi,
>>
>> I have a requirement and was wondering if someone could help me in how to 
>> go about it. We have to index about 8-9 million documents and their size 
>> can be anywhere from a few KBs to a couple of MBs. These documents are 
>> categorized into many 'groups' and 'sub-groups'. I wanted to know if we 
>> can create multiple indexes based on 'groups' and then on 'sub-groups' in 
>> Solr? If yes, then how do we go about it? I tried going through the 
>> section on 'Collections' in the Solr Wiki, but could not make much use of 
>> it.
>>
>> Regards,
>> Rishabh Joshi
>>
>>
>>
>>
>>
> 



This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender&postmaster@solutonnet.net  immediately and delete the original. Any other use of the email by you is prohibited.

RE: Best way to create multiple indexes

[Rishabh Joshi <ri...@coreobjects.com>]

Ryan,

We currently have 8-9 million documents to index and this number will grow in the future. Also, we will never have a query that will search across groups, but, we will have queries that will search across sub-groups for sure.
Now, keeping this in mind we were thinking if we could have multiple indexes at the 'group' level at least.
Also, can multiple indexes be created dynamically? For example: In my application if I create a 'logical group', then an index should be created for that group.

Rishabh

-----Original Message-----
From: Ryan McKinley [mailto:ryantxu@gmail.com]
Sent: Monday, November 12, 2007 7:44 PM
To: solr-user@lucene.apache.org
Subject: Re: Best way to create multiple indexes

For starters, do you need to be able to search across groups or
sub-groups (in one query?)

If so, then you have to stick everything in one index.

You can add a field to each document saying what 'group' or 'sub-group'
it is in and then limit it at query time

  q="kittens +group:A"

The advantage to splitting it into multiple indexes is that you could
put each index on independent hardware.  Depending on your queries and
index size that may make a big difference.

ryan


Rishabh Joshi wrote:
> Hi,
>
> I have a requirement and was wondering if someone could help me in how to go about it. We have to index about 8-9 million documents and their size can be anywhere from a few KBs to a couple of MBs. These documents are categorized into many 'groups' and 'sub-groups'. I wanted to know if we can create multiple indexes based on 'groups' and then on 'sub-groups' in Solr? If yes, then how do we go about it? I tried going through the section on 'Collections' in the Solr Wiki, but could not make much use of it.

>
> Regards,
> Rishabh Joshi
>
>
>
>
>

Re: Best way to create multiple indexes

[Ryan McKinley <ry...@gmail.com>]

For starters, do you need to be able to search across groups or 
sub-groups (in one query?)

If so, then you have to stick everything in one index.

You can add a field to each document saying what 'group' or 'sub-group' 
it is in and then limit it at query time

  q="kittens +group:A"

The advantage to splitting it into multiple indexes is that you could 
put each index on independent hardware.  Depending on your queries and 
index size that may make a big difference.

ryan


Rishabh Joshi wrote:
> Hi,
> 
> I have a requirement and was wondering if someone could help me in how to go about it. We have to index about 8-9 million documents and their size can be anywhere from a few KBs to a couple of MBs. These documents are categorized into many 'groups' and 'sub-groups'. I wanted to know if we can create multiple indexes based on 'groups' and then on 'sub-groups' in Solr? If yes, then how do we go about it? I tried going through the section on 'Collections' in the Solr Wiki, but could not make much use of it.
> 
> Regards,
> Rishabh Joshi
> 
> 
> 
> 
>