Sharding strategy for optimal NRT performance

[Chris Troullis <cp...@gmail.com>]

Hi!

I am looking for some advice on an sharding strategy that will produce
optimal performance in the NRT search case for my setup. I have come up
with a strategy that I think will work based on my experience, testing, and
reading of similar questions on the mailing list, but I was hoping to run
my idea by some experts to see if I am on the right track or am completely
off base.

*Let's start off with some background info on my use case*:

We are currently using Solr (5.5.2) with the classic Master/Slave setup.
Because of our NRT requirements, the slave is pretty much only used for
failover, all writes/reads go to the master (which I know is not ideal, but
that's what we're working with!). We have 6 different indexes with
completely different schemas for various searches in our application. We
have just over 300 tenants, which currently all reside within the same
index for each of our indexes. We separate our tenants at query time via a
filter query with a tenant identifier (which works fine). Each index is not
tremendously large, they range from 1M documents to the largest being
around 12M documents. Our load is not huge as search is not the core
functionality of our application, but merely a tool to get to what they are
looking for in the app. I believe our peak load barely goes over 1 QPS.
Even though our number of documents isn't super high, we do some pretty
complex faceting, and block joins in some cases, which along with crappy
hardware in our data center (no SSDs) initially led to some pretty poor
query times for our customers. This was due to the fact that we are
constantly indexing throughout the day (job that runs once per minute), and
we auto soft commit (openSearcher=true) every 1 minute. Because of the
nature of our application, NRT updates are necessary. As we all know,
opening searches this frequently has the drawback of invalidating all of
our searcher-based caches, causing query times to be erratic, and slower on
average. With our current setup, we have solved our query performance times
by setting up autowarming, both on the filter cache, and via static warming
queries.

*The problem:*

So now for the problem. While we are now running great from a performance
perspective, we are receiving complaints from customers saying that the
changes they are making are slow to be reflected in search. Because of the
nature of our application, this has significant impact on their user
experience, and is an issue we need to solve. Overall, we would like to be
able to reduce our NRT visibility from the minutes we have now down to
seconds. The problem is doing this in a way that won't significantly affect
our query performance. We are already seeing maxWarmingSearchers warnings
in our logs occasionally with our current setup, so just indexing more
frequently is not a viable solution. In addition to this, autowarming in
itself is problematic for the NRT use case, as the new searcher won't start
serving requests until it is fully warmed anyway, which is sort of counter
to the goal of decreasing the time it takes for new documents to be visible
in search. And so this is the predicament we find ourselves in. We can
index more frequently (and soft commit more frequently), but we will have
to remove (or greatly decrease) our autowarming, which will destroy our
search performance. Obviously there is some give and take here, we can't
have true NRT search with optimal query performance, but I am hoping to
find a solution that will provide acceptable results for both.

*Proposed solution:*

I have done a lot of research and experimentation on this issue and have
started coming up with what I believe will be a decent solution to the
aforementioned problem. First off, I would like to make the move over to
Solr Cloud. We had been contemplating this for a while anyway, as we
currently have no load balancing at all (since our slave is just used for
failover), but I am also thinking that by using the right sharding strategy
we can improve our NRT experience as well. I first started looking at the
standard composite id routing, and while we can ensure that all of a single
tenant's data is located on the same shard, because there is a large
discrepancy between the amounts of data our tenants have, our shards would
be very unevenly distributed in terms of number of documents. Ideally, we
would like all of our tenants to be isolated from a performance perspective
(from a security perspective we are not really concerned, as all of our
queries have a tenant identifier filter query already). Basically, we don't
want tiny tenant A to be screwed over because they were unlucky enough to
land on Huge tenant B's shard. We do know the footprint of each tenant in
terms of number of documents, so technically we could work out a sharding
strategy manually which would evenly distribute the tenants based on number
of documents, but since we have 6 different indexes, and with each index
the tenant's document distribution will be different, this would become a
headache to manage. And so the conclusion that I have come to is that for
optimal performance for our NRT use case, we would yield the best results
by having a separate shard (and replica) per tenant for each index
(collection). The benefit to this approach is that each tenant is isolated
from a performance perspective (in terms of # of documents and commits,
obviously the resources on the node are still shared). With this approach,
each individual shard should be small enough that caching/autowarming
becomes less important, and in addition to this, even though we will be
indexing more frequently, and auto soft commiting more frequently (say
every second), new searchers shouldn't be opened as frequently on the
individual shards, as they will only have to be reopened if document is
re-indexed on that particular shard. In other words, tenant A could be
making a bunch of changes and having to open new searchers, but it
shouldn't affect tenant B at all (since searchers are tied to shards, at
least that is my understanding).

*Questions:*

Hopefully I have done a good enough job explaining our current setup, the
problem I am trying to solve, and the solution I have come up with. I would
love to hear any feedback anyone has on my proposed solution or any
corrections anyone might see with some of the assumptions I am making. As
far as specific questions goes, other than "Is this viable?", I did have a
couple of specific questions around separating concerns at a collection
level vs a shard level:

1. I have read other discussions on the mailing list around strategies for
implementing multi-tenancy, but most of them seem to revolve around having
a *collection *per tenant, as opposed to what I am thinking about doing in
having a single collection but having a *shard *per tenant. As I understand
it, each shard is essentially a separate lucene index, and there is a base
overhead cost associated with that. This cost obviously applies whether or
not you are using collection per tenant or shard per tenant, as each
collection has to have at least 1 shard. Basically I am wondering what the
pros and cons would be between the 2 approaches? Is there additional
overhead in creating additional collections? IE, is there more overhead
using the collection per tenant approach than there would be having a
single collection with 1 shard/tenant?

2. Regarding the overhead of collections/shards per tenant, I have read in
multiple places that you should be wary about creating thousands of
collections with Solr Cloud. Does this also apply for shards? For example,
for my proposed solution, I would have say 300 shards per collection x 6
collections x 2 (1 replica per shard), which would put me at 3600
shards/replicas (across 6 collections). Obviously this would be split over
multiple nodes, but is this likely to cause problems? Is the issue caused
by having too many shards on one node (aka, would adding nodes help the
issue), or is it a broader issue with zookeeper coordination?


I apologize for the giant wall of text, I just wanted to make sure I gave a
thorough explanation of my problem. Again, any advice anyone could provide
would be greatly appreciated!

Thanks!