Indexing and Disk Writes

[Andrew Dinsmore <ac...@gmail.com>]

We are using Solr 5.4 to index TBs of documents in a bulk fashion to get
the cluster up and running. Indexing is over HTTP round robin as directed
by zookeeper.

Each of the 13 nodes is receiving about 6-8 MB/s on the NIC but solr is
writing around 20 to 25 thousand times per second (4k block size). My
question is what is Solr doing writing all this data to disk (80-100MB/s)?

Over a three hour run with 4.5 million docs we only committed 20 some times
but disk activity was pretty constant at the above levels.

Is there more going on than tlogs, commits and merges? When we moved from 1
minute autoCommit to 10 we committed less per the log messages but I
expected the bigger initial segments to result in less merging thus lower
disk activity. But testing showed no significant change in disk writing.

Thanks for any help.

Andrew

Re: Indexing and Disk Writes

[Andrew Dinsmore <ac...@gmail.com>]

Erick,

We currently have ramBufferSizeMB at 1024M. For this indexing activity, the
cluster is "offline" thus no queries coming in so not worried about any
user impact or delays should Solr terminate and need to replay. The
thinking was that increasing these values (ramBuffer, commit times, etc)
would cut down on the amount of merging by writing larger segments from the
start. segmentsPerTier is currently 15 so in theory if we only committed 15
times we would never have to merge (right?). But no real effect on the disk
metrics thus far.

For the most recent test, I was very surprised by the commit activity. Some
commits are logged by the qtp threads and some by the commit scheduler (I'm
ignoring the openSearch=true commits here. I recognize we should disable
softCommits if users are not searching during indexing). I assume the
"commit schedulers" are the autoCommits and the "qtp" are from the
ramBufferSizeMB or maxBufferedDocs thresholds. I observed that our commits
came in pairs (scheduler then qtp) usually within a minute or three of each
other and then nothing for 10 to 15 minutes and then another pair within a
minute or two. Even more surprising is that I observed commits across the
13 node cluster all within the same second. This activity isn't
synchronized, is it? I can't imagine we are indexing that uniformly in
terms of bytes to account for this behavior. And with autoCommit.maxTime
set to 10 minutes I see commits occurring closer to 15 minutes (7 logged by
/update in the admin interface over the 2 hour run) thus my question about
the cluster synchronizing commits.

We measured tlog writes to disk and they come very close to the bytes
coming in over the NIC (makes sense) so they are accounting for only 5 to
10% of the disk writes. Good savings but not enough to significantly change
the load we are putting on the SAN.

Andrew

On Fri, Nov 4, 2016 at 12:00 PM, Erick Erickson <er...@gmail.com>
wrote:

> Every time your ramBufferSizeMB limit is exceeded, a segment is
> created that's eventually merged. In terms of _throughput_, making
> this large usually doesn't help much after about 100M (the default).
> It'd be interesting to see if it changes your I/O activity though.
>
> BTW, I'd hard commit (openSearcher=false) much more frequently. As you
> see that doesn't particularly change IO, but if Solr should terminate
> abnormally the tlog will be replayed on startup and may sit there for
> 10 minutes.
>
> You could also consider disabling tlogs for the duration of your bulk
> indexing, then turn them back on for incremental.
>
> The background merging can be pretty dramatic though, that may well be
> where much of this is coming from.
>
> Best,
> Erick
>
> On Fri, Nov 4, 2016 at 8:51 AM, Andrew Dinsmore <ac...@gmail.com>
> wrote:
> > We are using Solr 5.4 to index TBs of documents in a bulk fashion to get
> > the cluster up and running. Indexing is over HTTP round robin as directed
> > by zookeeper.
> >
> > Each of the 13 nodes is receiving about 6-8 MB/s on the NIC but solr is
> > writing around 20 to 25 thousand times per second (4k block size). My
> > question is what is Solr doing writing all this data to disk
> (80-100MB/s)?
> >
> > Over a three hour run with 4.5 million docs we only committed 20 some
> times
> > but disk activity was pretty constant at the above levels.
> >
> > Is there more going on than tlogs, commits and merges? When we moved
> from 1
> > minute autoCommit to 10 we committed less per the log messages but I
> > expected the bigger initial segments to result in less merging thus lower
> > disk activity. But testing showed no significant change in disk writing.
> >
> > Thanks for any help.
> >
> > Andrew
>

Re: Indexing and Disk Writes

[Erick Erickson <er...@gmail.com>]

Every time your ramBufferSizeMB limit is exceeded, a segment is
created that's eventually merged. In terms of _throughput_, making
this large usually doesn't help much after about 100M (the default).
It'd be interesting to see if it changes your I/O activity though.

BTW, I'd hard commit (openSearcher=false) much more frequently. As you
see that doesn't particularly change IO, but if Solr should terminate
abnormally the tlog will be replayed on startup and may sit there for
10 minutes.

You could also consider disabling tlogs for the duration of your bulk
indexing, then turn them back on for incremental.

The background merging can be pretty dramatic though, that may well be
where much of this is coming from.

Best,
Erick

On Fri, Nov 4, 2016 at 8:51 AM, Andrew Dinsmore <ac...@gmail.com> wrote:
> We are using Solr 5.4 to index TBs of documents in a bulk fashion to get
> the cluster up and running. Indexing is over HTTP round robin as directed
> by zookeeper.
>
> Each of the 13 nodes is receiving about 6-8 MB/s on the NIC but solr is
> writing around 20 to 25 thousand times per second (4k block size). My
> question is what is Solr doing writing all this data to disk (80-100MB/s)?
>
> Over a three hour run with 4.5 million docs we only committed 20 some times
> but disk activity was pretty constant at the above levels.
>
> Is there more going on than tlogs, commits and merges? When we moved from 1
> minute autoCommit to 10 we committed less per the log messages but I
> expected the bigger initial segments to result in less merging thus lower
> disk activity. But testing showed no significant change in disk writing.
>
> Thanks for any help.
>
> Andrew