Performance potential for updating (reindexing) documents

[tedsolr <ts...@sciquest.com>]

With a properly tuned solr cloud infrastructure and less than 1B total docs
spread out over 50 collections where the largest collection is 100M docs,
what is a reasonable target goal for entirely reindexing a single
collection?

I understand there are a lot of variables, so I'm hypothetically wiping them
away by assuming "a properly tuned infrastructure". So the hardware, RAM,
etc. is configured correctly (not so in my case).

The scenario is to add 3 fields to all the existing docs in one collection.
The fields are the same but the values vary based on the docs. So a search
is performed and finds 100 matches - all 100 docs will get the same updates.
Then another search is performed that matches 15000 docs, and these are
updated. This continues 10-20,000 times until essentially all the docs have
been updated.

The docs all have 100 - 200 fields, mostly text and mostly small in size.
What's the best possible throughput I can expect? 1000 docs/sec? 5000
docs/sec?

Using SolrJ for querying and indexing against a v5.2.1 cloud.



--
View this message in context: http://lucene.472066.n3.nabble.com/Performance-potential-for-updating-reindexing-documents-tp4265861.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Performance potential for updating (reindexing) documents

[Shawn Heisey <ap...@elyograg.org>]

On 4/1/2016 8:56 PM, Erick Erickson wrote:
> bq: The bottleneck is definitely Solr.
>
> Since you commented out the server.add(doclist), you're right to focus
> there. I've seen
> a few things that help.
>
> 1> batch the documents, i.e. in the doclist above the list should be
> on the order of 1,000 docs. Here
> are some numbers I worked up one time:
> https://lucidworks.com/blog/2015/10/05/really-batch-updates-solr-2/

For that test, I was just seeing how fast MySQL could push data.  Based
on the results I saw from a small-scale test where I *did* add them,
letting the code run the add on the entire database with a single thread
would have taken forever.  I'm aware of the need to batch -- the code
did create batches, it just didn't send them.

I have a couple of ideas for the design on a multi-threaded indexing
program, but haven't worked out how to implement it.

> 3> Make sure you're using CloudSolrClient.

It's not SolrCloud, so that wouldn't really be helpful. :)

Thanks,
Shawn

Re: Performance potential for updating (reindexing) documents

[Erick Erickson <er...@gmail.com>]

Shawn:

bq: The bottleneck is definitely Solr.

Since you commented out the server.add(doclist), you're right to focus
there. I've seen
a few things that help.

1> batch the documents, i.e. in the doclist above the list should be
on the order of 1,000 docs. Here
are some numbers I worked up one time:
https://lucidworks.com/blog/2015/10/05/really-batch-updates-solr-2/

2> If your Solr CPUs aren't running flat out, then adding threads
until they are being pretty well hammered
is A Good Thing. Of course you have to balance that off against
anything else your servers are doing like
serving queries....

3> Make sure you're using CloudSolrClient.

4> If you still need more throughput, use more shards.....

Best,
Erick

On Thu, Mar 31, 2016 at 6:39 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 3/24/2016 11:57 AM, tedsolr wrote:
>> My post was scant on details. The numbers I gave for collection sizes are
>> projections for the future. I am in the midst of an upgrade that will be
>> completed within a few weeks. My concern is that I may not be able to
>> produce the throughput necessary to index an entire collection quickly
>> enough (3 to 4 hours) for a large customer (100M docs).
>
> I can fully rebuild one of my indexes, with 146 million docs, in 8-10
> hours.  This is fairly inefficient indexing -- six large shards (not
> cloud), each one running the dataimport handler, importing from MySQL.
> I suspect I could probably get two or three times this rate (and maybe
> more) on the same hardware if I wrote a SolrJ application that uses
> multiple threads for each Solr shard.
>
> I know from experiments that the MySQL server can push over 100 million
> rows to a SolrJ program in less than an hour, including constructing
> SolrInputDocument objects.  That experiment just left out the
> "client.add(docs);" line.  The bottleneck is definitely Solr.
>
> Each machine holds three large shards(half the index),is running Solr
> 4.x (5.x upgrade is in the works), and has 64GB RAM with an 8GB heap.
> Each shard is approximately 24.4 million docs and 28GB.  These machines
> also hold another sharded index in the same Solr install, but it's quite
> a lot smaller.
>
> Thanks,
> Shawn
>

Re: Performance potential for updating (reindexing) documents

[Shawn Heisey <ap...@elyograg.org>]

On 3/24/2016 11:57 AM, tedsolr wrote:
> My post was scant on details. The numbers I gave for collection sizes are
> projections for the future. I am in the midst of an upgrade that will be
> completed within a few weeks. My concern is that I may not be able to
> produce the throughput necessary to index an entire collection quickly
> enough (3 to 4 hours) for a large customer (100M docs).

I can fully rebuild one of my indexes, with 146 million docs, in 8-10
hours.  This is fairly inefficient indexing -- six large shards (not
cloud), each one running the dataimport handler, importing from MySQL. 
I suspect I could probably get two or three times this rate (and maybe
more) on the same hardware if I wrote a SolrJ application that uses
multiple threads for each Solr shard.

I know from experiments that the MySQL server can push over 100 million
rows to a SolrJ program in less than an hour, including constructing
SolrInputDocument objects.  That experiment just left out the
"client.add(docs);" line.  The bottleneck is definitely Solr.

Each machine holds three large shards(half the index),is running Solr
4.x (5.x upgrade is in the works), and has 64GB RAM with an 8GB heap. 
Each shard is approximately 24.4 million docs and 28GB.  These machines
also hold another sharded index in the same Solr install, but it's quite
a lot smaller.

Thanks,
Shawn

Re: Performance potential for updating (reindexing) documents

[Erick Erickson <er...@gmail.com>]

Well, for comparison I routinely get 20K docs/second on my Mac Pro
indexing Wikipedia docs. I _think_ I have 4 shards when I do this, all
in the same JVM. I'd be surprised if you can't get your 5K docs/sec,
but you may indeed need more shards.

All that said, 4G  for the JVM is kind of constrained, you already
mentioned GC. There are two pitfalls here:
1> allocating too little memory and spending lots of cycles doing very
small GCs. At 4G this is likelier than:
2> having very large heaps and seeing "stop the world" GC pauses.....

So I think you're on the right track looking at memory, at least
that's what I'd be looking at first.

Note: your indexing scaling (assuming you're sending complete docs not
atomic updates) will scale better if you
1> use CloudSolrClient from Java since it routes docs to the right
leader first and avoids an extra hop.
2> batch updates. Sending one doc at a time makes things very slow, see:

https://lucidworks.com/blog/2015/10/05/really-batch-updates-solr-2/

Best,
Erick

On Thu, Mar 24, 2016 at 10:57 AM, tedsolr <ts...@sciquest.com> wrote:
> Hi Erick,
>
> My post was scant on details. The numbers I gave for collection sizes are
> projections for the future. I am in the midst of an upgrade that will be
> completed within a few weeks. My concern is that I may not be able to
> produce the throughput necessary to index an entire collection quickly
> enough (3 to 4 hours) for a large customer (100M docs).
>
> Currently:
> - single Solr instance on one host that is sharing memory and cpu with other
> applications
> - 4GB dedicated to Solr
> ~ 20M docs
> ~ 10GB index size
> - using HttpSolrClient for all queries and updates
>
> Very soon:
> - two VMs dedicated to Solr (2 nodes)
> - up to 16GB available memory
> - running in cloud mode, and can now scale horizontally
> - all collections are single sharded with 2 replicas
>
> All fields are stored. The scenario I gave is using atomic updates. The
> updates are done in large batches of 5000-10000 docs. The use case I have is
> different than most Solr setups perhaps. Indexing throughput is more
> important than qps. We have very few concurrent users that do massive
> amounts of doc updates. I am seeing lousy (production) performance currently
> (not a surprise - long GC pauses), and have just begun the process of tuning
> in a test environment.
>
> After some more weeks of testing and tweaking I hope to get to 5000
> updates/sec, but even that may not be enough. So my main concern is that
> this business model (of updating entire collections about once a day) cannot
> be supported by Solr.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Performance-potential-for-updating-reindexing-documents-tp4265861p4265922.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Performance potential for updating (reindexing) documents

[tedsolr <ts...@sciquest.com>]

Hi Erick,

My post was scant on details. The numbers I gave for collection sizes are
projections for the future. I am in the midst of an upgrade that will be
completed within a few weeks. My concern is that I may not be able to
produce the throughput necessary to index an entire collection quickly
enough (3 to 4 hours) for a large customer (100M docs).

Currently:
- single Solr instance on one host that is sharing memory and cpu with other
applications
- 4GB dedicated to Solr
~ 20M docs
~ 10GB index size
- using HttpSolrClient for all queries and updates

Very soon:
- two VMs dedicated to Solr (2 nodes)
- up to 16GB available memory
- running in cloud mode, and can now scale horizontally
- all collections are single sharded with 2 replicas

All fields are stored. The scenario I gave is using atomic updates. The
updates are done in large batches of 5000-10000 docs. The use case I have is
different than most Solr setups perhaps. Indexing throughput is more
important than qps. We have very few concurrent users that do massive
amounts of doc updates. I am seeing lousy (production) performance currently
(not a surprise - long GC pauses), and have just begun the process of tuning
in a test environment.

After some more weeks of testing and tweaking I hope to get to 5000
updates/sec, but even that may not be enough. So my main concern is that
this business model (of updating entire collections about once a day) cannot
be supported by Solr.



--
View this message in context: http://lucene.472066.n3.nabble.com/Performance-potential-for-updating-reindexing-documents-tp4265861p4265922.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Performance potential for updating (reindexing) documents

[Erick Erickson <er...@gmail.com>]

Impossible to say if for no other reason than you haven't told us
how many physical machines this is spread over ;).

For the process you've outlined to work, all the fields are stored,
right? So why not use Atomic Updates? You still have to query
the docs.

About querying. If I'm reading this right, you'll form some query
like q=whatever_identifies_docs_that_should_get_values_X_Y_Z
then process each one of those. So, really, all you need here is
the id of all the queries that satisfy that clause. You should
consider the /export handler (Streaming Aggregation). It's
designed to return large result sets with minimal memory.

So the process I'm thinking of is this (and it assumes all your
fields are stored so Atomic updates work).

Use the CloudSolrStream for each query. As the stream
comes back, you get the IDs you need and use them
to do an atomic update that adds the relevant fields.

Note that when _adding_ fields, you can change the schema
to include the new fields on an existing collection. All that
means is that any new docs added can have these fields.

Now, if all the fields are _not_ stored at least once, you can't
use atomic updates and you'll have to re-index from the system
of record.

Best,
Erick

On Thu, Mar 24, 2016 at 7:18 AM, tedsolr <ts...@sciquest.com> wrote:
> With a properly tuned solr cloud infrastructure and less than 1B total docs
> spread out over 50 collections where the largest collection is 100M docs,
> what is a reasonable target goal for entirely reindexing a single
> collection?
>
> I understand there are a lot of variables, so I'm hypothetically wiping them
> away by assuming "a properly tuned infrastructure". So the hardware, RAM,
> etc. is configured correctly (not so in my case).
>
> The scenario is to add 3 fields to all the existing docs in one collection.
> The fields are the same but the values vary based on the docs. So a search
> is performed and finds 100 matches - all 100 docs will get the same updates.
> Then another search is performed that matches 15000 docs, and these are
> updated. This continues 10-20,000 times until essentially all the docs have
> been updated.
>
> The docs all have 100 - 200 fields, mostly text and mostly small in size.
> What's the best possible throughput I can expect? 1000 docs/sec? 5000
> docs/sec?
>
> Using SolrJ for querying and indexing against a v5.2.1 cloud.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Performance-potential-for-updating-reindexing-documents-tp4265861.html
> Sent from the Solr - User mailing list archive at Nabble.com.