Doubt Regarding Shards Index

[Suneel <pa...@gmail.com>]

I am using solr My index becomes too large I want to implement shards concept
but i have some doubt. i searched a lot but not found satisfied result.

1. We need to create handler for shards in solrconfig.xml ?

2. Index will be different for each shards instance means we need to break
data in part to create index for each instance or index will be same?

3. How i will recognize which instance return the result ?


Please provide me above details this will be very helpful for me.

Thanks & Regards
Suneel Pandey 

--
View this message in context: http://lucene.472066.n3.nabble.com/Doubt-Regarding-Shards-Index-tp3629964p3629964.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Doubt Regarding Shards Index

[Dmitry Kan <so...@gmail.com>]

Hi,

Sharding by time by itself does not need any custom code on solr side:
start indexing your data to a shard, depending on the timestamp of your
document.

The querying part is trickier if you want to have one front end solr: it
should know which shards to query. If querying all shards for each query is
fine for you, then you are good and no changes needed. Alternatively, you
can shoot a query to a particular year shard knowing the year of your user
query.

Dmitry


On Fri, Jun 7, 2013 at 3:54 PM, sathish_ix <sk...@inautix.co.in>wrote:

> Hi ,
>
> How did you distribute the index by year to different shards,
> do we need to write any code ?
>
> Thanks,
> Sathish
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Doubt-Regarding-Shards-Index-tp3629964p4068869.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Doubt Regarding Shards Index

[sathish_ix <sk...@inautix.co.in>]

Hi ,

How did you distribute the index by year to different shards,
do we need to write any code ?

Thanks,
Sathish



--
View this message in context: http://lucene.472066.n3.nabble.com/Doubt-Regarding-Shards-Index-tp3629964p4068869.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Doubt Regarding Shards Index

["Sethi, Parampreet" <pa...@teamaol.com>]

Hi Suneel,

I have implemented Solr sharding in one of my projects where data was of
the order of 1 billion documents and my queries were throwing Out of
memory exception because of huge index. Here are my views:

- Have identical Solr server setups for each shard with same schema.

1. We need to create handler for shards in solrconfig.xml ?
- In my case, I did not add any handlers in solrconfig.xml for sharding.

2. Index will be different for each shards instance means we need to break
data in part to create index for each instance or index will be same?
- yes, Index needs to be broken into each shard instance. I used
creation_date field in my case to divide the data by years into each shard
(for example all documents with year 2007 will go to shard 1 and 2008 will
go to shard 2 and so on) and similarly while writing the data look for the
same field and index the corresponding shard.

3. How i will recognize which instance return the result ?
- Once you know how data is divided, you can easily figure out which shard
is serving. 

I have put some of my analysis on this blog post
http://www.params.me/2010/04/working-with-solr.html. Hope it helps!

Best,
Param
http://params.me



On 1/3/12 2:02 PM, "Suneel" <pa...@gmail.com> wrote:

>I am using solr My index becomes too large I want to implement shards
>concept
>but i have some doubt. i searched a lot but not found satisfied result.
>
>1. We need to create handler for shards in solrconfig.xml ?
>
>2. Index will be different for each shards instance means we need to break
>data in part to create index for each instance or index will be same?
>
>3. How i will recognize which instance return the result ?
>
>
>Please provide me above details this will be very helpful for me.
>
>Thanks & Regards
>Suneel Pandey 
>
>--
>View this message in context:
>http://lucene.472066.n3.nabble.com/Doubt-Regarding-Shards-Index-tp3629964p
>3629964.html
>Sent from the Solr - User mailing list archive at Nabble.com.