Solr design. Choose Cores or Shards?

[manju16832003 <ma...@gmail.com>]

Hi,
I have a confusion over choosing Cores or Shards for the project scenario.
My scenario is as follows
I have three entities 
 1. Customers
 2. Product Info
 3. Listings [Contains all the listings posted by customer based on product]

I'm planning to design Solr structure for the above scenario like this
 1. Customers Core
 2. Product Info Core
 3. Listings Core
 4. Searchable Listing Core [Indexing searchable parameters selected from
Listings, Product Info and Customer entities]. 

Having in mind that there wouldn't be much updates to Customers, Product
Info. There will be regular updates to Listings that in turn I need to
update Searchable listings that I could manage it.

My Confusion is it feasible to choose many cores or use shards. I do not
have much experience on how shards works and why they are used for. I would
like to know the suggestions :-) for the design like this.
What are the implications if I were to choose to use many cores and handle
stuff at application level calling different cores.

Thanks
 



--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-design-Choose-Cores-or-Shards-tp4082930.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr design. Choose Cores or Shards?

[Dhananjay Makwana <ma...@semandex.net>]

Thanks for the explanation !

On 8/8/13 4:52 AM, Shawn Heisey wrote:
> On 8/6/2013 8:49 PM, manju16832003 wrote:
>> My Confusion is it feasible to choose many cores or use shards. I do not
>> have much experience on how shards works and why they are used for. I would
>> like to know the suggestions :-) for the design like this.
>> What are the implications if I were to choose to use many cores and handle
>> stuff at application level calling different cores.
>
> Although shards and cores refer to slightly different things, when it
> comes right down to it, it's difficult to separate the two concepts. Short version: Shards are 
> implemented using cores.  The long version follows below.
>
> A core is a functionally complete Solr index.  You can have more than one core per Solr instance.  
> Multiple cores are discussed in the CoreAdmin wiki page: http://wiki.apache.org/solr/CoreAdmin
>
> Shards refer to a concept in distributed search.  The index is divided
> into pieces.  The request comes in to Solr.  Solr forwards the request to each shard.  It then 
> analyzes each shard result into a combined result, pulls the requested fields out of each shard, 
> and sends the response to the requester.
>
> If you are planning a new deployment of a sharded index, you probably
> will want to use SolrCloud.  It's possible to use shards without
> SolrCloud, but SolrCloud automates everything and makes it MUCH easier.
>
> In SolrCloud, a collection is a logical index.  A collection is composed
> of one or more shards.  It is perfectly acceptable to have only one
> shard in an index, in which case it won't be using distributed search,
> but the following still applies:
>
> Each shard is composed of replicas.  If your replicationFactor is 2,
> then when your cloud is operating normally, you'll have two replicas of
> each shard.  If the replicationFactor is 5, then you'll have five
> replicas.  One of those replicas will be elected as leader for that
> shard.  You can have a replicationFactor of 1, in which case there will only be one copy, but it 
> will not be a fault-tolerant setup.
>
> Now for the relationship between shards and cores:  Each replica of a shard *IS* a core.  All of 
> the cores in a single collection will typically have the same configuration and schema.
>
> More info about SolrCloud:
>
> http://wiki.apache.org/solr/SolrCloud
>
> Thanks,
> Shawn
>

Re: Solr design. Choose Cores or Shards?

[Shawn Heisey <so...@elyograg.org>]

On 8/6/2013 8:49 PM, manju16832003 wrote:
> My Confusion is it feasible to choose many cores or use shards. I do not
> have much experience on how shards works and why they are used for. I would
> like to know the suggestions :-) for the design like this.
> What are the implications if I were to choose to use many cores and handle
> stuff at application level calling different cores.

Although shards and cores refer to slightly different things, when it
comes right down to it, it's difficult to separate the two concepts. 
Short version: Shards are implemented using cores.  The long version 
follows below.

A core is a functionally complete Solr index.  You can have more than 
one core per Solr instance.  Multiple cores are discussed in the 
CoreAdmin wiki page: http://wiki.apache.org/solr/CoreAdmin

Shards refer to a concept in distributed search.  The index is divided
into pieces.  The request comes in to Solr.  Solr forwards the request 
to each shard.  It then analyzes each shard result into a combined 
result, pulls the requested fields out of each shard, and sends the 
response to the requester.

If you are planning a new deployment of a sharded index, you probably
will want to use SolrCloud.  It's possible to use shards without
SolrCloud, but SolrCloud automates everything and makes it MUCH easier.

In SolrCloud, a collection is a logical index.  A collection is composed
of one or more shards.  It is perfectly acceptable to have only one
shard in an index, in which case it won't be using distributed search,
but the following still applies:

Each shard is composed of replicas.  If your replicationFactor is 2,
then when your cloud is operating normally, you'll have two replicas of
each shard.  If the replicationFactor is 5, then you'll have five
replicas.  One of those replicas will be elected as leader for that
shard.  You can have a replicationFactor of 1, in which case there will 
only be one copy, but it will not be a fault-tolerant setup.

Now for the relationship between shards and cores:  Each replica of a 
shard *IS* a core.  All of the cores in a single collection will 
typically have the same configuration and schema.

More info about SolrCloud:

http://wiki.apache.org/solr/SolrCloud

Thanks,
Shawn

Re: Solr design. Choose Cores or Shards?

[manju16832003 <ma...@gmail.com>]

Hi Eric,
Thanks for your reply.



--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-design-Choose-Cores-or-Shards-tp4082930p4083178.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr design. Choose Cores or Shards?

[Erick Erickson <er...@gmail.com>]

Shards are "special" cores, usually hosted on separate machines
that comprise one single large (logical) index. Shards need to
have the same schema, config, etc usually.

So unless you have a corpus that's too large to fit on a single piece
of your hardware, you'll always be using "cores". And since your
cores have different types of data (and presumably use different
schemas), you're talking cores.

Best
Erick




On Tue, Aug 6, 2013 at 10:49 PM, manju16832003 <ma...@gmail.com>wrote:

> Hi,
> I have a confusion over choosing Cores or Shards for the project scenario.
> My scenario is as follows
> I have three entities
>  1. Customers
>  2. Product Info
>  3. Listings [Contains all the listings posted by customer based on
> product]
>
> I'm planning to design Solr structure for the above scenario like this
>  1. Customers Core
>  2. Product Info Core
>  3. Listings Core
>  4. Searchable Listing Core [Indexing searchable parameters selected from
> Listings, Product Info and Customer entities].
>
> Having in mind that there wouldn't be much updates to Customers, Product
> Info. There will be regular updates to Listings that in turn I need to
> update Searchable listings that I could manage it.
>
> My Confusion is it feasible to choose many cores or use shards. I do not
> have much experience on how shards works and why they are used for. I would
> like to know the suggestions :-) for the design like this.
> What are the implications if I were to choose to use many cores and handle
> stuff at application level calling different cores.
>
> Thanks
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-design-Choose-Cores-or-Shards-tp4082930.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>