replication

[John Blythe <jo...@gmail.com>]

hi, all.

we're starting to dive into master/slave replication architecture. we'll
have 1 master w 4 slaves behind it. our app is NRT. if user performs an
action in section A's data they may choose to jump to section B which will
be dependent on having the updates from their action in section A. as such,
we're thinking that the replication time should be set to 1-2s (the chances
of them arriving at section B quickly enough to catch the 2s gap is highly
unlikely at best).

since the replicas will simply be looking for new files it seems like this
would be a lightweight operation even every couple seconds for 4 replicas.
that said, i'm going *entirely* off of assumption at this point and wanted
to check in w you all to see any nuances, gotchas, hidden landmines, etc.
that we should be considering before rolling things out.

thanks for any info!

--
John Blythe

Re: replication

[John Blythe <jo...@gmail.com>]

great. thanks, erick!

--
John Blythe

On Wed, Apr 11, 2018 at 12:16 PM, Erick Erickson <er...@gmail.com>
wrote:

> bq: are you simply flagging the fact that we wouldn't direct the queries
> to A
> v. B v. C since SolrCloud will make the decisions itself as to which part
> of the distro gets hit for the operation
>
> Yep. SolrCloud takes care of it all itself. I should also add that there
> are
> about a zillion metrics now available in Solr that you can use to make the
> best use of hardware, including things like CPU usage, I/O, GC etc.
> SolrCloud
> doesn't _yet_ make use of these but will in future. The current software LB
> does a pretty simple round-robin distribution.
>
> Best,
> Erick
>
> On Wed, Apr 11, 2018 at 5:57 AM, John Blythe <jo...@gmail.com> wrote:
> > thanks, erick. great info.
> >
> > although you can't (yet) direct queries to one or the other. So just
> making
> >> them all NRT and forgetting about it is reasonable.
> >
> >
> > are you simply flagging the fact that we wouldn't direct the queries to A
> > v. B v. C since SolrCloud will make the decisions itself as to which part
> > of the distro gets hit for the operation? if not, can you expound on
> this a
> > bit more?
> >
> > The very nature of merging is such that you will _always_ get large
> merges
> >> until you have 5G segments (by default)
> >
> >
> > bummer
> >
> > Quite possible, but you have to route things yourself. But in that case
> >> you're limited to one machine to handle all your NRT traffic. I skimmed
> >> your post so don't know whether your NRT traffic load is high enough to
> >> worry about.
> >
> >
> > ok. i think we'll take a two-pronged approach. for the immediate purposes
> > of trying to solve an issue we've begun encountering we will begin
> > thoroughtesting the load between various operations in the master-slave
> > setup we've set up. pending the results, we can roll forward w a
> temporary
> > patch in which all end-user touch points route through the primary box
> for
> > read/write while large scale operations/processing we do in the
> background
> > will point to the ELB the slaves are sitting behind. we'll also begin
> > setting up a simple solrcloud instance to toy with per your suggestion
> > above. inb4 tons more questions on my part :)
> >
> > thanks!
> >
> > --
> > John Blythe
> >
> > On Tue, Apr 10, 2018 at 11:14 AM, Erick Erickson <
> erickerickson@gmail.com>
> > wrote:
> >
> >> bq: should we try to bite the solrcloud bullet and be done w it
> >>
> >> that's what I'd do. As of 7.0 there are different "flavors", TLOG,
> >> PULL and NRT so that's also a possibility, although you can't (yet)
> >> direct queries to one or the other. So just making them all NRT and
> >> forgetting about it is reasonable.
> >>
> >> bq:  is there some more config work we could put in place to avoid ...
> >> commit issue and the ultra large merge dangers
> >>
> >> No. The very nature of merging is such that you will _always_ get
> >> large merges until you have 5G segments (by default). The max segment
> >> size (outside "optimize/forceMerge/expungeDeletes" which you shouldn't
> >> do) is 5G so the steady-state worst-case segment pull is limited to
> >> that.
> >>
> >> bq: maybe for our initial need we use Master for writing and user
> >> access in NRT events, but slaves for the heavier backend
> >>
> >> Quite possible, but you have to route things yourself. But in that
> >> case you're limited to one machine to handle all your NRT traffic. I
> >> skimmed your post so don't know whether your NRT traffic load is high
> >> enough to worry about.
> >>
> >> The very first thing I'd do is set up a simple SolrCloud setup and
> >> give it a spin. Unless your indexing load is quite heavy, the added
> >> work the NRT replicas have in SolrCloud isn't a problem so worrying
> >> about that is premature optimization unless you have a heavy load.....
> >>
> >> Best,
> >> Erick
> >>
> >> On Mon, Apr 9, 2018 at 4:36 PM, John Blythe <jo...@gmail.com>
> wrote:
> >> > Thanks a bunch for the thorough reply, Shawn.
> >> >
> >> > Phew. We’d chosen to go w Master-slave replication instead of
> SolrCloud
> >> per
> >> > the sudden need we had encountered and the desire to avoid the nuances
> >> and
> >> > changes related to moving to SolrCloud. But so much for this being a
> more
> >> > straightforward solution, huh?
> >> >
> >> > Few questions:
> >> > - should we try to bite the solrcloud bullet and be done w it?
> >> > - is there some more config work we could put in place to avoid the
> soft
> >> > commit issue and the ultra large merge dangers, keeping the
> replications
> >> > happening quickly?
> >> > - maybe for our initial need we use Master for writing and user
> access in
> >> > NRT events, but slaves for the heavier backend processing. Thoughts?
> >> > - anyone do consulting on this that would be interested in chatting?
> >> >
> >> > Thanks again!
> >> >
> >> > On Mon, Apr 9, 2018 at 18:18 Shawn Heisey <ap...@elyograg.org>
> wrote:
> >> >
> >> >> On 4/9/2018 12:15 PM, John Blythe wrote:
> >> >> > we're starting to dive into master/slave replication architecture.
> >> we'll
> >> >> > have 1 master w 4 slaves behind it. our app is NRT. if user
> performs
> >> an
> >> >> > action in section A's data they may choose to jump to section B
> which
> >> >> will
> >> >> > be dependent on having the updates from their action in section A.
> as
> >> >> such,
> >> >> > we're thinking that the replication time should be set to 1-2s (the
> >> >> chances
> >> >> > of them arriving at section B quickly enough to catch the 2s gap is
> >> >> highly
> >> >> > unlikely at best).
> >> >>
> >> >> Once you start talking about master-slave replication, my assumption
> is
> >> >> that you're not running SolrCloud.  You would NOT want to try and mix
> >> >> SolrCloud with replication.  The features do not play well together.
> >> >> SolrCloud with NRT replicas (this is the only replica type that
> exists
> >> >> in 6.x and earlier) may be a better option than master-slave
> >> replication.
> >> >>
> >> >> > since the replicas will simply be looking for new files it seems
> like
> >> >> this
> >> >> > would be a lightweight operation even every couple seconds for 4
> >> >> replicas.
> >> >> > that said, i'm going *entirely* off of assumption at this point and
> >> >> wanted
> >> >> > to check in w you all to see any nuances, gotchas, hidden
> landmines,
> >> etc.
> >> >> > that we should be considering before rolling things out.
> >> >>
> >> >> Most of the time, you'd be correct to think that indexing is going to
> >> >> create a new small segment and replication will have little work to
> do.
> >> >> But as you create more and more segments, eventually Lucene is going
> to
> >> >> start merging those segments.  For discussion purposes, I'm going to
> >> >> describe a situation where each new segment during indexing is about
> >> >> 100KB in size, and the merge policy is left at the default settings.
> >> >> I'm also going to assume that no documents are getting deleted or
> >> >> reindexed (which will delete the old version).  Deleted documents can
> >> >> have an impact on merging, but it will usually only be a dramatic
> impact
> >> >> if there are a LOT of deleted documents.
> >> >>
> >> >> The first ten segments created will be this 100KB size.  Then Lucene
> is
> >> >> going to see that there are enough segments to trigger the merge
> policy
> >> >> - it's going to combine ten of those segments into one that's
> >> >> approximately one megabyte.  Repeat this ten times, and ten of those
> 1
> >> >> megabyte segments will be combined into one ten megabyte segment.
> >> >> Repeat all of THAT ten times, and there will be a 100 megabyte
> segment.
> >> >> And there will eventually be another level creating 1 gigabyte
> >> >> segments.  If the index is below 5GB in size, the entire thing
> *could*
> >> >> be merged into one segment by this process.
> >> >>
> >> >> The end result of all this:  Replication is not always going to be
> >> >> super-quick.  If merging creates a 1 gigabyte segment, then the
> amount
> >> >> of time to transfer that new segment is going to depend on how fast
> your
> >> >> disks are, and how fast your network is.  If you're using commodity
> SATA
> >> >> drives in the 4 to 10 terabyte range and a gigabit network, the
> network
> >> >> is probably going to be the bottleneck -- assuming that the system
> has
> >> >> plenty of memory and isn't under a high load.  If the network is the
> >> >> bottleneck in that situation, it's probably going to take close to
> ten
> >> >> seconds to transfer a 1GB segment, and the greater part of a minute
> to
> >> >> transfer a 5GB segment, which is the biggest one that the default
> merge
> >> >> policy configuration will create without an optimize operation.
> >> >>
> >> >> Also, you should understand something that has come to my attention
> >> >> recently (and is backed up by documentation):  If the master does a
> soft
> >> >> commit and the segment that was committed remains in memory (not
> flushed
> >> >> to disk), that segment will NOT be replicated to the slaves.  It has
> to
> >> >> get flushed to disk before it can be replicated.
> >> >>
> >> >> Thanks,
> >> >> Shawn
> >> >>
> >> >> --
> >> > John Blythe
> >>
>

Re: replication

[Erick Erickson <er...@gmail.com>]

bq: are you simply flagging the fact that we wouldn't direct the queries to A
v. B v. C since SolrCloud will make the decisions itself as to which part
of the distro gets hit for the operation

Yep. SolrCloud takes care of it all itself. I should also add that there are
about a zillion metrics now available in Solr that you can use to make the
best use of hardware, including things like CPU usage, I/O, GC etc. SolrCloud
doesn't _yet_ make use of these but will in future. The current software LB
does a pretty simple round-robin distribution.

Best,
Erick

On Wed, Apr 11, 2018 at 5:57 AM, John Blythe <jo...@gmail.com> wrote:
> thanks, erick. great info.
>
> although you can't (yet) direct queries to one or the other. So just making
>> them all NRT and forgetting about it is reasonable.
>
>
> are you simply flagging the fact that we wouldn't direct the queries to A
> v. B v. C since SolrCloud will make the decisions itself as to which part
> of the distro gets hit for the operation? if not, can you expound on this a
> bit more?
>
> The very nature of merging is such that you will _always_ get large merges
>> until you have 5G segments (by default)
>
>
> bummer
>
> Quite possible, but you have to route things yourself. But in that case
>> you're limited to one machine to handle all your NRT traffic. I skimmed
>> your post so don't know whether your NRT traffic load is high enough to
>> worry about.
>
>
> ok. i think we'll take a two-pronged approach. for the immediate purposes
> of trying to solve an issue we've begun encountering we will begin
> thoroughtesting the load between various operations in the master-slave
> setup we've set up. pending the results, we can roll forward w a temporary
> patch in which all end-user touch points route through the primary box for
> read/write while large scale operations/processing we do in the background
> will point to the ELB the slaves are sitting behind. we'll also begin
> setting up a simple solrcloud instance to toy with per your suggestion
> above. inb4 tons more questions on my part :)
>
> thanks!
>
> --
> John Blythe
>
> On Tue, Apr 10, 2018 at 11:14 AM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> bq: should we try to bite the solrcloud bullet and be done w it
>>
>> that's what I'd do. As of 7.0 there are different "flavors", TLOG,
>> PULL and NRT so that's also a possibility, although you can't (yet)
>> direct queries to one or the other. So just making them all NRT and
>> forgetting about it is reasonable.
>>
>> bq:  is there some more config work we could put in place to avoid ...
>> commit issue and the ultra large merge dangers
>>
>> No. The very nature of merging is such that you will _always_ get
>> large merges until you have 5G segments (by default). The max segment
>> size (outside "optimize/forceMerge/expungeDeletes" which you shouldn't
>> do) is 5G so the steady-state worst-case segment pull is limited to
>> that.
>>
>> bq: maybe for our initial need we use Master for writing and user
>> access in NRT events, but slaves for the heavier backend
>>
>> Quite possible, but you have to route things yourself. But in that
>> case you're limited to one machine to handle all your NRT traffic. I
>> skimmed your post so don't know whether your NRT traffic load is high
>> enough to worry about.
>>
>> The very first thing I'd do is set up a simple SolrCloud setup and
>> give it a spin. Unless your indexing load is quite heavy, the added
>> work the NRT replicas have in SolrCloud isn't a problem so worrying
>> about that is premature optimization unless you have a heavy load.....
>>
>> Best,
>> Erick
>>
>> On Mon, Apr 9, 2018 at 4:36 PM, John Blythe <jo...@gmail.com> wrote:
>> > Thanks a bunch for the thorough reply, Shawn.
>> >
>> > Phew. We’d chosen to go w Master-slave replication instead of SolrCloud
>> per
>> > the sudden need we had encountered and the desire to avoid the nuances
>> and
>> > changes related to moving to SolrCloud. But so much for this being a more
>> > straightforward solution, huh?
>> >
>> > Few questions:
>> > - should we try to bite the solrcloud bullet and be done w it?
>> > - is there some more config work we could put in place to avoid the soft
>> > commit issue and the ultra large merge dangers, keeping the replications
>> > happening quickly?
>> > - maybe for our initial need we use Master for writing and user access in
>> > NRT events, but slaves for the heavier backend processing. Thoughts?
>> > - anyone do consulting on this that would be interested in chatting?
>> >
>> > Thanks again!
>> >
>> > On Mon, Apr 9, 2018 at 18:18 Shawn Heisey <ap...@elyograg.org> wrote:
>> >
>> >> On 4/9/2018 12:15 PM, John Blythe wrote:
>> >> > we're starting to dive into master/slave replication architecture.
>> we'll
>> >> > have 1 master w 4 slaves behind it. our app is NRT. if user performs
>> an
>> >> > action in section A's data they may choose to jump to section B which
>> >> will
>> >> > be dependent on having the updates from their action in section A. as
>> >> such,
>> >> > we're thinking that the replication time should be set to 1-2s (the
>> >> chances
>> >> > of them arriving at section B quickly enough to catch the 2s gap is
>> >> highly
>> >> > unlikely at best).
>> >>
>> >> Once you start talking about master-slave replication, my assumption is
>> >> that you're not running SolrCloud.  You would NOT want to try and mix
>> >> SolrCloud with replication.  The features do not play well together.
>> >> SolrCloud with NRT replicas (this is the only replica type that exists
>> >> in 6.x and earlier) may be a better option than master-slave
>> replication.
>> >>
>> >> > since the replicas will simply be looking for new files it seems like
>> >> this
>> >> > would be a lightweight operation even every couple seconds for 4
>> >> replicas.
>> >> > that said, i'm going *entirely* off of assumption at this point and
>> >> wanted
>> >> > to check in w you all to see any nuances, gotchas, hidden landmines,
>> etc.
>> >> > that we should be considering before rolling things out.
>> >>
>> >> Most of the time, you'd be correct to think that indexing is going to
>> >> create a new small segment and replication will have little work to do.
>> >> But as you create more and more segments, eventually Lucene is going to
>> >> start merging those segments.  For discussion purposes, I'm going to
>> >> describe a situation where each new segment during indexing is about
>> >> 100KB in size, and the merge policy is left at the default settings.
>> >> I'm also going to assume that no documents are getting deleted or
>> >> reindexed (which will delete the old version).  Deleted documents can
>> >> have an impact on merging, but it will usually only be a dramatic impact
>> >> if there are a LOT of deleted documents.
>> >>
>> >> The first ten segments created will be this 100KB size.  Then Lucene is
>> >> going to see that there are enough segments to trigger the merge policy
>> >> - it's going to combine ten of those segments into one that's
>> >> approximately one megabyte.  Repeat this ten times, and ten of those 1
>> >> megabyte segments will be combined into one ten megabyte segment.
>> >> Repeat all of THAT ten times, and there will be a 100 megabyte segment.
>> >> And there will eventually be another level creating 1 gigabyte
>> >> segments.  If the index is below 5GB in size, the entire thing *could*
>> >> be merged into one segment by this process.
>> >>
>> >> The end result of all this:  Replication is not always going to be
>> >> super-quick.  If merging creates a 1 gigabyte segment, then the amount
>> >> of time to transfer that new segment is going to depend on how fast your
>> >> disks are, and how fast your network is.  If you're using commodity SATA
>> >> drives in the 4 to 10 terabyte range and a gigabit network, the network
>> >> is probably going to be the bottleneck -- assuming that the system has
>> >> plenty of memory and isn't under a high load.  If the network is the
>> >> bottleneck in that situation, it's probably going to take close to ten
>> >> seconds to transfer a 1GB segment, and the greater part of a minute to
>> >> transfer a 5GB segment, which is the biggest one that the default merge
>> >> policy configuration will create without an optimize operation.
>> >>
>> >> Also, you should understand something that has come to my attention
>> >> recently (and is backed up by documentation):  If the master does a soft
>> >> commit and the segment that was committed remains in memory (not flushed
>> >> to disk), that segment will NOT be replicated to the slaves.  It has to
>> >> get flushed to disk before it can be replicated.
>> >>
>> >> Thanks,
>> >> Shawn
>> >>
>> >> --
>> > John Blythe
>>

Re: replication

[John Blythe <jo...@gmail.com>]

thanks, erick. great info.

although you can't (yet) direct queries to one or the other. So just making
> them all NRT and forgetting about it is reasonable.


are you simply flagging the fact that we wouldn't direct the queries to A
v. B v. C since SolrCloud will make the decisions itself as to which part
of the distro gets hit for the operation? if not, can you expound on this a
bit more?

The very nature of merging is such that you will _always_ get large merges
> until you have 5G segments (by default)


bummer

Quite possible, but you have to route things yourself. But in that case
> you're limited to one machine to handle all your NRT traffic. I skimmed
> your post so don't know whether your NRT traffic load is high enough to
> worry about.


ok. i think we'll take a two-pronged approach. for the immediate purposes
of trying to solve an issue we've begun encountering we will begin
thoroughtesting the load between various operations in the master-slave
setup we've set up. pending the results, we can roll forward w a temporary
patch in which all end-user touch points route through the primary box for
read/write while large scale operations/processing we do in the background
will point to the ELB the slaves are sitting behind. we'll also begin
setting up a simple solrcloud instance to toy with per your suggestion
above. inb4 tons more questions on my part :)

thanks!

--
John Blythe

On Tue, Apr 10, 2018 at 11:14 AM, Erick Erickson <er...@gmail.com>
wrote:

> bq: should we try to bite the solrcloud bullet and be done w it
>
> that's what I'd do. As of 7.0 there are different "flavors", TLOG,
> PULL and NRT so that's also a possibility, although you can't (yet)
> direct queries to one or the other. So just making them all NRT and
> forgetting about it is reasonable.
>
> bq:  is there some more config work we could put in place to avoid ...
> commit issue and the ultra large merge dangers
>
> No. The very nature of merging is such that you will _always_ get
> large merges until you have 5G segments (by default). The max segment
> size (outside "optimize/forceMerge/expungeDeletes" which you shouldn't
> do) is 5G so the steady-state worst-case segment pull is limited to
> that.
>
> bq: maybe for our initial need we use Master for writing and user
> access in NRT events, but slaves for the heavier backend
>
> Quite possible, but you have to route things yourself. But in that
> case you're limited to one machine to handle all your NRT traffic. I
> skimmed your post so don't know whether your NRT traffic load is high
> enough to worry about.
>
> The very first thing I'd do is set up a simple SolrCloud setup and
> give it a spin. Unless your indexing load is quite heavy, the added
> work the NRT replicas have in SolrCloud isn't a problem so worrying
> about that is premature optimization unless you have a heavy load.....
>
> Best,
> Erick
>
> On Mon, Apr 9, 2018 at 4:36 PM, John Blythe <jo...@gmail.com> wrote:
> > Thanks a bunch for the thorough reply, Shawn.
> >
> > Phew. We’d chosen to go w Master-slave replication instead of SolrCloud
> per
> > the sudden need we had encountered and the desire to avoid the nuances
> and
> > changes related to moving to SolrCloud. But so much for this being a more
> > straightforward solution, huh?
> >
> > Few questions:
> > - should we try to bite the solrcloud bullet and be done w it?
> > - is there some more config work we could put in place to avoid the soft
> > commit issue and the ultra large merge dangers, keeping the replications
> > happening quickly?
> > - maybe for our initial need we use Master for writing and user access in
> > NRT events, but slaves for the heavier backend processing. Thoughts?
> > - anyone do consulting on this that would be interested in chatting?
> >
> > Thanks again!
> >
> > On Mon, Apr 9, 2018 at 18:18 Shawn Heisey <ap...@elyograg.org> wrote:
> >
> >> On 4/9/2018 12:15 PM, John Blythe wrote:
> >> > we're starting to dive into master/slave replication architecture.
> we'll
> >> > have 1 master w 4 slaves behind it. our app is NRT. if user performs
> an
> >> > action in section A's data they may choose to jump to section B which
> >> will
> >> > be dependent on having the updates from their action in section A. as
> >> such,
> >> > we're thinking that the replication time should be set to 1-2s (the
> >> chances
> >> > of them arriving at section B quickly enough to catch the 2s gap is
> >> highly
> >> > unlikely at best).
> >>
> >> Once you start talking about master-slave replication, my assumption is
> >> that you're not running SolrCloud.  You would NOT want to try and mix
> >> SolrCloud with replication.  The features do not play well together.
> >> SolrCloud with NRT replicas (this is the only replica type that exists
> >> in 6.x and earlier) may be a better option than master-slave
> replication.
> >>
> >> > since the replicas will simply be looking for new files it seems like
> >> this
> >> > would be a lightweight operation even every couple seconds for 4
> >> replicas.
> >> > that said, i'm going *entirely* off of assumption at this point and
> >> wanted
> >> > to check in w you all to see any nuances, gotchas, hidden landmines,
> etc.
> >> > that we should be considering before rolling things out.
> >>
> >> Most of the time, you'd be correct to think that indexing is going to
> >> create a new small segment and replication will have little work to do.
> >> But as you create more and more segments, eventually Lucene is going to
> >> start merging those segments.  For discussion purposes, I'm going to
> >> describe a situation where each new segment during indexing is about
> >> 100KB in size, and the merge policy is left at the default settings.
> >> I'm also going to assume that no documents are getting deleted or
> >> reindexed (which will delete the old version).  Deleted documents can
> >> have an impact on merging, but it will usually only be a dramatic impact
> >> if there are a LOT of deleted documents.
> >>
> >> The first ten segments created will be this 100KB size.  Then Lucene is
> >> going to see that there are enough segments to trigger the merge policy
> >> - it's going to combine ten of those segments into one that's
> >> approximately one megabyte.  Repeat this ten times, and ten of those 1
> >> megabyte segments will be combined into one ten megabyte segment.
> >> Repeat all of THAT ten times, and there will be a 100 megabyte segment.
> >> And there will eventually be another level creating 1 gigabyte
> >> segments.  If the index is below 5GB in size, the entire thing *could*
> >> be merged into one segment by this process.
> >>
> >> The end result of all this:  Replication is not always going to be
> >> super-quick.  If merging creates a 1 gigabyte segment, then the amount
> >> of time to transfer that new segment is going to depend on how fast your
> >> disks are, and how fast your network is.  If you're using commodity SATA
> >> drives in the 4 to 10 terabyte range and a gigabit network, the network
> >> is probably going to be the bottleneck -- assuming that the system has
> >> plenty of memory and isn't under a high load.  If the network is the
> >> bottleneck in that situation, it's probably going to take close to ten
> >> seconds to transfer a 1GB segment, and the greater part of a minute to
> >> transfer a 5GB segment, which is the biggest one that the default merge
> >> policy configuration will create without an optimize operation.
> >>
> >> Also, you should understand something that has come to my attention
> >> recently (and is backed up by documentation):  If the master does a soft
> >> commit and the segment that was committed remains in memory (not flushed
> >> to disk), that segment will NOT be replicated to the slaves.  It has to
> >> get flushed to disk before it can be replicated.
> >>
> >> Thanks,
> >> Shawn
> >>
> >> --
> > John Blythe
>

Re: replication

[Shawn Heisey <ap...@elyograg.org>]

On 4/10/2018 9:14 AM, Erick Erickson wrote:
> The very first thing I'd do is set up a simple SolrCloud setup and
> give it a spin. Unless your indexing load is quite heavy, the added
> work the NRT replicas have in SolrCloud isn't a problem so worrying
> about that is premature optimization unless you have a heavy load.....

<snip>

>>
>>> Also, you should understand something that has come to my attention
>>> recently (and is backed up by documentation):  If the master does a soft
>>> commit and the segment that was committed remains in memory (not flushed
>>> to disk), that segment will NOT be replicated to the slaves.  It has to
>>> get flushed to disk before it can be replicated.

Erick, I wasn't sure whether you caught onto that part of what I said.  
The "backed up by documentation" part is this:  The reference guide 
specifically says that TLOG and PULL replica types do not support NRT 
indexing (soft commit).  I was actually unaware of that limitation, but 
in hindsight it makes sense.

Thanks,
Shawn

Re: replication

[Erick Erickson <er...@gmail.com>]

bq: should we try to bite the solrcloud bullet and be done w it

that's what I'd do. As of 7.0 there are different "flavors", TLOG,
PULL and NRT so that's also a possibility, although you can't (yet)
direct queries to one or the other. So just making them all NRT and
forgetting about it is reasonable.

bq:  is there some more config work we could put in place to avoid ...
commit issue and the ultra large merge dangers

No. The very nature of merging is such that you will _always_ get
large merges until you have 5G segments (by default). The max segment
size (outside "optimize/forceMerge/expungeDeletes" which you shouldn't
do) is 5G so the steady-state worst-case segment pull is limited to
that.

bq: maybe for our initial need we use Master for writing and user
access in NRT events, but slaves for the heavier backend

Quite possible, but you have to route things yourself. But in that
case you're limited to one machine to handle all your NRT traffic. I
skimmed your post so don't know whether your NRT traffic load is high
enough to worry about.

The very first thing I'd do is set up a simple SolrCloud setup and
give it a spin. Unless your indexing load is quite heavy, the added
work the NRT replicas have in SolrCloud isn't a problem so worrying
about that is premature optimization unless you have a heavy load.....

Best,
Erick

On Mon, Apr 9, 2018 at 4:36 PM, John Blythe <jo...@gmail.com> wrote:
> Thanks a bunch for the thorough reply, Shawn.
>
> Phew. We’d chosen to go w Master-slave replication instead of SolrCloud per
> the sudden need we had encountered and the desire to avoid the nuances and
> changes related to moving to SolrCloud. But so much for this being a more
> straightforward solution, huh?
>
> Few questions:
> - should we try to bite the solrcloud bullet and be done w it?
> - is there some more config work we could put in place to avoid the soft
> commit issue and the ultra large merge dangers, keeping the replications
> happening quickly?
> - maybe for our initial need we use Master for writing and user access in
> NRT events, but slaves for the heavier backend processing. Thoughts?
> - anyone do consulting on this that would be interested in chatting?
>
> Thanks again!
>
> On Mon, Apr 9, 2018 at 18:18 Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 4/9/2018 12:15 PM, John Blythe wrote:
>> > we're starting to dive into master/slave replication architecture. we'll
>> > have 1 master w 4 slaves behind it. our app is NRT. if user performs an
>> > action in section A's data they may choose to jump to section B which
>> will
>> > be dependent on having the updates from their action in section A. as
>> such,
>> > we're thinking that the replication time should be set to 1-2s (the
>> chances
>> > of them arriving at section B quickly enough to catch the 2s gap is
>> highly
>> > unlikely at best).
>>
>> Once you start talking about master-slave replication, my assumption is
>> that you're not running SolrCloud.  You would NOT want to try and mix
>> SolrCloud with replication.  The features do not play well together.
>> SolrCloud with NRT replicas (this is the only replica type that exists
>> in 6.x and earlier) may be a better option than master-slave replication.
>>
>> > since the replicas will simply be looking for new files it seems like
>> this
>> > would be a lightweight operation even every couple seconds for 4
>> replicas.
>> > that said, i'm going *entirely* off of assumption at this point and
>> wanted
>> > to check in w you all to see any nuances, gotchas, hidden landmines, etc.
>> > that we should be considering before rolling things out.
>>
>> Most of the time, you'd be correct to think that indexing is going to
>> create a new small segment and replication will have little work to do.
>> But as you create more and more segments, eventually Lucene is going to
>> start merging those segments.  For discussion purposes, I'm going to
>> describe a situation where each new segment during indexing is about
>> 100KB in size, and the merge policy is left at the default settings.
>> I'm also going to assume that no documents are getting deleted or
>> reindexed (which will delete the old version).  Deleted documents can
>> have an impact on merging, but it will usually only be a dramatic impact
>> if there are a LOT of deleted documents.
>>
>> The first ten segments created will be this 100KB size.  Then Lucene is
>> going to see that there are enough segments to trigger the merge policy
>> - it's going to combine ten of those segments into one that's
>> approximately one megabyte.  Repeat this ten times, and ten of those 1
>> megabyte segments will be combined into one ten megabyte segment.
>> Repeat all of THAT ten times, and there will be a 100 megabyte segment.
>> And there will eventually be another level creating 1 gigabyte
>> segments.  If the index is below 5GB in size, the entire thing *could*
>> be merged into one segment by this process.
>>
>> The end result of all this:  Replication is not always going to be
>> super-quick.  If merging creates a 1 gigabyte segment, then the amount
>> of time to transfer that new segment is going to depend on how fast your
>> disks are, and how fast your network is.  If you're using commodity SATA
>> drives in the 4 to 10 terabyte range and a gigabit network, the network
>> is probably going to be the bottleneck -- assuming that the system has
>> plenty of memory and isn't under a high load.  If the network is the
>> bottleneck in that situation, it's probably going to take close to ten
>> seconds to transfer a 1GB segment, and the greater part of a minute to
>> transfer a 5GB segment, which is the biggest one that the default merge
>> policy configuration will create without an optimize operation.
>>
>> Also, you should understand something that has come to my attention
>> recently (and is backed up by documentation):  If the master does a soft
>> commit and the segment that was committed remains in memory (not flushed
>> to disk), that segment will NOT be replicated to the slaves.  It has to
>> get flushed to disk before it can be replicated.
>>
>> Thanks,
>> Shawn
>>
>> --
> John Blythe

Re: replication

[John Blythe <jo...@gmail.com>]

Thanks a bunch for the thorough reply, Shawn.

Phew. We’d chosen to go w Master-slave replication instead of SolrCloud per
the sudden need we had encountered and the desire to avoid the nuances and
changes related to moving to SolrCloud. But so much for this being a more
straightforward solution, huh?

Few questions:
- should we try to bite the solrcloud bullet and be done w it?
- is there some more config work we could put in place to avoid the soft
commit issue and the ultra large merge dangers, keeping the replications
happening quickly?
- maybe for our initial need we use Master for writing and user access in
NRT events, but slaves for the heavier backend processing. Thoughts?
- anyone do consulting on this that would be interested in chatting?

Thanks again!

On Mon, Apr 9, 2018 at 18:18 Shawn Heisey <ap...@elyograg.org> wrote:

> On 4/9/2018 12:15 PM, John Blythe wrote:
> > we're starting to dive into master/slave replication architecture. we'll
> > have 1 master w 4 slaves behind it. our app is NRT. if user performs an
> > action in section A's data they may choose to jump to section B which
> will
> > be dependent on having the updates from their action in section A. as
> such,
> > we're thinking that the replication time should be set to 1-2s (the
> chances
> > of them arriving at section B quickly enough to catch the 2s gap is
> highly
> > unlikely at best).
>
> Once you start talking about master-slave replication, my assumption is
> that you're not running SolrCloud.  You would NOT want to try and mix
> SolrCloud with replication.  The features do not play well together.
> SolrCloud with NRT replicas (this is the only replica type that exists
> in 6.x and earlier) may be a better option than master-slave replication.
>
> > since the replicas will simply be looking for new files it seems like
> this
> > would be a lightweight operation even every couple seconds for 4
> replicas.
> > that said, i'm going *entirely* off of assumption at this point and
> wanted
> > to check in w you all to see any nuances, gotchas, hidden landmines, etc.
> > that we should be considering before rolling things out.
>
> Most of the time, you'd be correct to think that indexing is going to
> create a new small segment and replication will have little work to do.
> But as you create more and more segments, eventually Lucene is going to
> start merging those segments.  For discussion purposes, I'm going to
> describe a situation where each new segment during indexing is about
> 100KB in size, and the merge policy is left at the default settings.
> I'm also going to assume that no documents are getting deleted or
> reindexed (which will delete the old version).  Deleted documents can
> have an impact on merging, but it will usually only be a dramatic impact
> if there are a LOT of deleted documents.
>
> The first ten segments created will be this 100KB size.  Then Lucene is
> going to see that there are enough segments to trigger the merge policy
> - it's going to combine ten of those segments into one that's
> approximately one megabyte.  Repeat this ten times, and ten of those 1
> megabyte segments will be combined into one ten megabyte segment.
> Repeat all of THAT ten times, and there will be a 100 megabyte segment.
> And there will eventually be another level creating 1 gigabyte
> segments.  If the index is below 5GB in size, the entire thing *could*
> be merged into one segment by this process.
>
> The end result of all this:  Replication is not always going to be
> super-quick.  If merging creates a 1 gigabyte segment, then the amount
> of time to transfer that new segment is going to depend on how fast your
> disks are, and how fast your network is.  If you're using commodity SATA
> drives in the 4 to 10 terabyte range and a gigabit network, the network
> is probably going to be the bottleneck -- assuming that the system has
> plenty of memory and isn't under a high load.  If the network is the
> bottleneck in that situation, it's probably going to take close to ten
> seconds to transfer a 1GB segment, and the greater part of a minute to
> transfer a 5GB segment, which is the biggest one that the default merge
> policy configuration will create without an optimize operation.
>
> Also, you should understand something that has come to my attention
> recently (and is backed up by documentation):  If the master does a soft
> commit and the segment that was committed remains in memory (not flushed
> to disk), that segment will NOT be replicated to the slaves.  It has to
> get flushed to disk before it can be replicated.
>
> Thanks,
> Shawn
>
> --
John Blythe

Re: replication

[Shawn Heisey <ap...@elyograg.org>]

On 4/9/2018 12:15 PM, John Blythe wrote:
> we're starting to dive into master/slave replication architecture. we'll
> have 1 master w 4 slaves behind it. our app is NRT. if user performs an
> action in section A's data they may choose to jump to section B which will
> be dependent on having the updates from their action in section A. as such,
> we're thinking that the replication time should be set to 1-2s (the chances
> of them arriving at section B quickly enough to catch the 2s gap is highly
> unlikely at best).

Once you start talking about master-slave replication, my assumption is
that you're not running SolrCloud.  You would NOT want to try and mix
SolrCloud with replication.  The features do not play well together. 
SolrCloud with NRT replicas (this is the only replica type that exists
in 6.x and earlier) may be a better option than master-slave replication.

> since the replicas will simply be looking for new files it seems like this
> would be a lightweight operation even every couple seconds for 4 replicas.
> that said, i'm going *entirely* off of assumption at this point and wanted
> to check in w you all to see any nuances, gotchas, hidden landmines, etc.
> that we should be considering before rolling things out.

Most of the time, you'd be correct to think that indexing is going to
create a new small segment and replication will have little work to do. 
But as you create more and more segments, eventually Lucene is going to
start merging those segments.  For discussion purposes, I'm going to
describe a situation where each new segment during indexing is about
100KB in size, and the merge policy is left at the default settings. 
I'm also going to assume that no documents are getting deleted or
reindexed (which will delete the old version).  Deleted documents can
have an impact on merging, but it will usually only be a dramatic impact
if there are a LOT of deleted documents.

The first ten segments created will be this 100KB size.  Then Lucene is
going to see that there are enough segments to trigger the merge policy
- it's going to combine ten of those segments into one that's
approximately one megabyte.  Repeat this ten times, and ten of those 1
megabyte segments will be combined into one ten megabyte segment. 
Repeat all of THAT ten times, and there will be a 100 megabyte segment. 
And there will eventually be another level creating 1 gigabyte
segments.  If the index is below 5GB in size, the entire thing *could*
be merged into one segment by this process.

The end result of all this:  Replication is not always going to be
super-quick.  If merging creates a 1 gigabyte segment, then the amount
of time to transfer that new segment is going to depend on how fast your
disks are, and how fast your network is.  If you're using commodity SATA
drives in the 4 to 10 terabyte range and a gigabit network, the network
is probably going to be the bottleneck -- assuming that the system has
plenty of memory and isn't under a high load.  If the network is the
bottleneck in that situation, it's probably going to take close to ten
seconds to transfer a 1GB segment, and the greater part of a minute to
transfer a 5GB segment, which is the biggest one that the default merge
policy configuration will create without an optimize operation.

Also, you should understand something that has come to my attention
recently (and is backed up by documentation):  If the master does a soft
commit and the segment that was committed remains in memory (not flushed
to disk), that segment will NOT be replicated to the slaves.  It has to
get flushed to disk before it can be replicated.

Thanks,
Shawn