You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Furkan KAMACI <fu...@gmail.com> on 2013/04/01 23:02:22 UTC
Filtering Search Cloud
I want to separate my cloud into two logical parts. One of them is indexer
cloud of SolrCloud. Second one is Searcher cloud of SolrCloud.
My first question is that. Does separating my cloud system make sense about
performance improvement. Because I think that when indexing, searching make
time to response and if I separate them I get a performance improvement. On
the other hand maybe using all Solr machines as whole (I mean not
partitioning as I mentioned) SolrCloud can make a better load balancing, I
would want to learn it.
My second question is that. Let's assume that I have separated my machines
as I mentioned. Can I filter some indexes to be searchable or not from
Searcher SolrCloud?
Re: Filtering Search Cloud
Posted by Furkan KAMACI <fu...@gmail.com>.
Ok, I will test and give you a detailed report for it, thanks for your help.
2013/4/5 Erick Erickson <er...@gmail.com>
> I cannot emphasize strongly enough that you need to _prove_ you have
> a problem before you decide on a solution! Do you have any evidence
> that solrcloud can't handle the load you intend? Might a better approach
> be just to create more shards thus spreading the load and get all the
> HA/DR goodness of SolrCloud?
>
> So far you've said you'll have a "heavy" load without giving us any
> numbers.
> 10,000 update/second? 10 updates/second? 1 query/second? 100,000
> queries/second? 100,000 documents? 1,000,000,000,000 documents?
>
> Best
> Erick
>
> On Wed, Apr 3, 2013 at 5:15 PM, Shawn Heisey <so...@elyograg.org> wrote:
> > On 4/3/2013 1:52 PM, Furkan KAMACI wrote:
> >> Thanks for your explanation, you explained every thing what I need. Just
> >> one more question. I see that I can not make it with Solr Cloud, but I
> can
> >> do something like that with master-slave replication of Solr. If I use
> >> master-slave replication of Solr, can I eliminate (filter) something
> >> (something that is indexed from master) from being a response after
> >> querying (querying from slaves) ?
> >
> > I don't understand the question. I will attempt to give you more
> > information, but it might not answer your question. If not, you'll have
> > to try to improve your question.
> >
> > Your master and each of that master's slaves will have the same index as
> > soon as replication is done. A query on the slave has no idea that the
> > master exists.
> >
> > Thanks,
> > Shawn
> >
>
Re: Filtering Search Cloud
Posted by Erick Erickson <er...@gmail.com>.
I cannot emphasize strongly enough that you need to _prove_ you have
a problem before you decide on a solution! Do you have any evidence
that solrcloud can't handle the load you intend? Might a better approach
be just to create more shards thus spreading the load and get all the
HA/DR goodness of SolrCloud?
So far you've said you'll have a "heavy" load without giving us any numbers.
10,000 update/second? 10 updates/second? 1 query/second? 100,000
queries/second? 100,000 documents? 1,000,000,000,000 documents?
Best
Erick
On Wed, Apr 3, 2013 at 5:15 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 4/3/2013 1:52 PM, Furkan KAMACI wrote:
>> Thanks for your explanation, you explained every thing what I need. Just
>> one more question. I see that I can not make it with Solr Cloud, but I can
>> do something like that with master-slave replication of Solr. If I use
>> master-slave replication of Solr, can I eliminate (filter) something
>> (something that is indexed from master) from being a response after
>> querying (querying from slaves) ?
>
> I don't understand the question. I will attempt to give you more
> information, but it might not answer your question. If not, you'll have
> to try to improve your question.
>
> Your master and each of that master's slaves will have the same index as
> soon as replication is done. A query on the slave has no idea that the
> master exists.
>
> Thanks,
> Shawn
>
Re: Filtering Search Cloud
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/3/2013 1:52 PM, Furkan KAMACI wrote:
> Thanks for your explanation, you explained every thing what I need. Just
> one more question. I see that I can not make it with Solr Cloud, but I can
> do something like that with master-slave replication of Solr. If I use
> master-slave replication of Solr, can I eliminate (filter) something
> (something that is indexed from master) from being a response after
> querying (querying from slaves) ?
I don't understand the question. I will attempt to give you more
information, but it might not answer your question. If not, you'll have
to try to improve your question.
Your master and each of that master's slaves will have the same index as
soon as replication is done. A query on the slave has no idea that the
master exists.
Thanks,
Shawn
Re: Filtering Search Cloud
Posted by Furkan KAMACI <fu...@gmail.com>.
Thanks for your explanation, you explained every thing what I need. Just
one more question. I see that I can not make it with Solr Cloud, but I can
do something like that with master-slave replication of Solr. If I use
master-slave replication of Solr, can I eliminate (filter) something
(something that is indexed from master) from being a response after
querying (querying from slaves) ?
2013/4/3 Shawn Heisey <so...@elyograg.org>
> On 4/3/2013 1:13 PM, Furkan KAMACI wrote:
> > Shawn, thanks for your detailed explanation. My system will work on high
> > load. I mean I will always index something and something always will be
> > queried at my system. That is why I consider about physically separating
> > indexer and query reply machines. I think about that: imagine a machine
> > that both does indexing (a disk IO for it, I don't know the underlying
> > system maybe Solr makes a sequential IO) and both trying to reply queries
> > (another kind of IO) That is my main challenge to decide separating them.
> > And the next step is that, if I separate them before response can I
> filter
> > the data of indexer machines (I don't have any filtering issues right
> now,
> > I just think that maybe I can need it at future)
>
> We do seem to have a language barrier, so let me try to be very clear:
> If you use SolrCloud, you can't separate querying and indexing. You
> will have to use the master-slave replication that been part of Solr
> since at least 1.4, possibly earlier.
>
> Thanks,
> Shawn
>
>
Re: Filtering Search Cloud
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/3/2013 1:13 PM, Furkan KAMACI wrote:
> Shawn, thanks for your detailed explanation. My system will work on high
> load. I mean I will always index something and something always will be
> queried at my system. That is why I consider about physically separating
> indexer and query reply machines. I think about that: imagine a machine
> that both does indexing (a disk IO for it, I don't know the underlying
> system maybe Solr makes a sequential IO) and both trying to reply queries
> (another kind of IO) That is my main challenge to decide separating them.
> And the next step is that, if I separate them before response can I filter
> the data of indexer machines (I don't have any filtering issues right now,
> I just think that maybe I can need it at future)
We do seem to have a language barrier, so let me try to be very clear:
If you use SolrCloud, you can't separate querying and indexing. You
will have to use the master-slave replication that been part of Solr
since at least 1.4, possibly earlier.
Thanks,
Shawn
Re: Filtering Search Cloud
Posted by Furkan KAMACI <fu...@gmail.com>.
Shawn, thanks for your detailed explanation. My system will work on high
load. I mean I will always index something and something always will be
queried at my system. That is why I consider about physically separating
indexer and query reply machines. I think about that: imagine a machine
that both does indexing (a disk IO for it, I don't know the underlying
system maybe Solr makes a sequential IO) and both trying to reply queries
(another kind of IO) That is my main challenge to decide separating them.
And the next step is that, if I separate them before response can I filter
the data of indexer machines (I don't have any filtering issues right now,
I just think that maybe I can need it at future)
2013/4/3 Shawn Heisey <so...@elyograg.org>
> On 4/1/2013 3:02 PM, Furkan KAMACI wrote:
> > I want to separate my cloud into two logical parts. One of them is
> indexer
> > cloud of SolrCloud. Second one is Searcher cloud of SolrCloud.
> >
> > My first question is that. Does separating my cloud system make sense
> about
> > performance improvement. Because I think that when indexing, searching
> make
> > time to response and if I separate them I get a performance improvement.
> On
> > the other hand maybe using all Solr machines as whole (I mean not
> > partitioning as I mentioned) SolrCloud can make a better load balancing,
> I
> > would want to learn it.
> >
> > My second question is that. Let's assume that I have separated my
> machines
> > as I mentioned. Can I filter some indexes to be searchable or not from
> > Searcher SolrCloud?
>
> SolrCloud gets rid of the master and slave designations. It also gets
> rid of the line between indexing and querying. Each shard has a replica
> that is designated the leader, but that has no real impact on searching
> and indexing, only on deciding which data to use when replicas get out
> of sync.
>
> In the old master-slave architecture, you indexed to the master and the
> updated index files were replicated to the slave. The slave did not
> handle the analysis for indexing, so it was usually better to send
> queries to slaves and let the master only do indexing.
>
> SolrCloud is very different. When you index, the documents are indexed
> on all replicas at about the same time. When you query, the requests
> are load balanced across all replicas. During normal operation,
> SolrCloud does not use replication at all. The replication feature is
> only used when a replica gets out of sync with the leader, and in that
> case, the entire index is replicated.
>
> Thanks,
> Shawn
>
>
Re: Filtering Search Cloud
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/1/2013 3:02 PM, Furkan KAMACI wrote:
> I want to separate my cloud into two logical parts. One of them is indexer
> cloud of SolrCloud. Second one is Searcher cloud of SolrCloud.
>
> My first question is that. Does separating my cloud system make sense about
> performance improvement. Because I think that when indexing, searching make
> time to response and if I separate them I get a performance improvement. On
> the other hand maybe using all Solr machines as whole (I mean not
> partitioning as I mentioned) SolrCloud can make a better load balancing, I
> would want to learn it.
>
> My second question is that. Let's assume that I have separated my machines
> as I mentioned. Can I filter some indexes to be searchable or not from
> Searcher SolrCloud?
SolrCloud gets rid of the master and slave designations. It also gets
rid of the line between indexing and querying. Each shard has a replica
that is designated the leader, but that has no real impact on searching
and indexing, only on deciding which data to use when replicas get out
of sync.
In the old master-slave architecture, you indexed to the master and the
updated index files were replicated to the slave. The slave did not
handle the analysis for indexing, so it was usually better to send
queries to slaves and let the master only do indexing.
SolrCloud is very different. When you index, the documents are indexed
on all replicas at about the same time. When you query, the requests
are load balanced across all replicas. During normal operation,
SolrCloud does not use replication at all. The replication feature is
only used when a replica gets out of sync with the leader, and in that
case, the entire index is replicated.
Thanks,
Shawn