You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by vsriram30 <vs...@gmail.com> on 2015/04/15 22:40:09 UTC
solr index design for this use case?
Hi All,
Consider this scenario : I am having around 100K content and I want to
launch 5 sites with that content. For example, around 50K content for site1,
40K content for site2, 30K for site3, 20K for site4, and 10K for site5.
As seen from this example, these sites have few overlapping content and non
overlapping content as well. In this case say if a content page is present
in all site1, site2 and site3 out of 50 fields per content page, say 30
fields remain common between site1 and site2, 25 fields common between site1
and site3 and 20 fields between site2 and site3, in this case, my aim is to
prevent duplication as much as possible without getting too much reduction
in QPS. Hence I consider the following options,
Option 1: Just maintain individual copy of duplicated content for each site
and overwrite site specific information while indexing for those sites.
Pros:
Better QPS as no query time joins are involved.
Cons:
Duplication of common fields for common content across sites.
Option 2: Maintain just a single copy of common fields per content across
all overlapping sites and separate site specific information for that
content and do a merge while serving using joins.
In this approach, for joins I looked at Block join provided by solr and
looks like it may not be a good fit for my case as if one site specific info
changes, I don't want to index the entire block containing other sites as
well.
Is there any better way to tackle this making sure we are not occupying so
much space and at the same time not reducing the QPS too much?
Thanks,
Sriram
--
View this message in context: http://lucene.472066.n3.nabble.com/solr-index-design-for-this-use-case-tp4200000.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr index design for this use case?
Posted by vsriram30 <vs...@gmail.com>.
Hi Eric,
Thanks for your response. I was planning to do the same, to store the data
in a single collection with site parameter differentiating duplicated
content for different sites. But my use case is that in future the content
would run into millions and potentially there could be large number of sites
as well. Hence I am trying to arrive at a solution that can be best of both
worlds. Is there any efficient way to store preventing duplication, merging
the site specific and common content, and still be able to edit record for
individual site without having to index the entire block.
Thanks,
Sriram
--
View this message in context: http://lucene.472066.n3.nabble.com/solr-index-design-for-this-use-case-tp4200000p4200065.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr index design for this use case?
Posted by Erick Erickson <er...@gmail.com>.
At this data size, don't worry at _all_ about duplicating content. A
single Solr node easily holds 20M docs. 50M is common and 250M is not
unheard of.
My bold claim is: you can freely duplicate the data to your heart's
content and you'll never notice it.
In fact, you can put it all in a single collection with some kind of
"site" field to distinguish which is which
and when you want to restrict results to a specific site, just use an fq clause.
HTH,
Erick
On Wed, Apr 15, 2015 at 1:40 PM, vsriram30 <vs...@gmail.com> wrote:
> Hi All,
>
> Consider this scenario : I am having around 100K content and I want to
> launch 5 sites with that content. For example, around 50K content for site1,
> 40K content for site2, 30K for site3, 20K for site4, and 10K for site5.
>
> As seen from this example, these sites have few overlapping content and non
> overlapping content as well. In this case say if a content page is present
> in all site1, site2 and site3 out of 50 fields per content page, say 30
> fields remain common between site1 and site2, 25 fields common between site1
> and site3 and 20 fields between site2 and site3, in this case, my aim is to
> prevent duplication as much as possible without getting too much reduction
> in QPS. Hence I consider the following options,
>
> Option 1: Just maintain individual copy of duplicated content for each site
> and overwrite site specific information while indexing for those sites.
> Pros:
> Better QPS as no query time joins are involved.
> Cons:
> Duplication of common fields for common content across sites.
>
> Option 2: Maintain just a single copy of common fields per content across
> all overlapping sites and separate site specific information for that
> content and do a merge while serving using joins.
> In this approach, for joins I looked at Block join provided by solr and
> looks like it may not be a good fit for my case as if one site specific info
> changes, I don't want to index the entire block containing other sites as
> well.
> Is there any better way to tackle this making sure we are not occupying so
> much space and at the same time not reducing the QPS too much?
>
> Thanks,
> Sriram
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/solr-index-design-for-this-use-case-tp4200000.html
> Sent from the Solr - User mailing list archive at Nabble.com.