How to replicate config files in master-slave replication without commit on master?

[Georg Sorst <ge...@gmail.com>]

Hi!

According to
https://cwiki.apache.org/confluence/display/solr/Index+Replication#IndexReplication-ReplicatingConfigurationFiles
:

> Solr replicates configuration files only when the index itself is
replicated. That means even if a configuration file is changed on the
master, that file will be replicated only after there is a new
commit/optimize on master's index.

However, the behaviour of some components can be changed by just changing a
configuration file. A good example is the Query Elevation component, which
is configured through conf/elevate.xml. The component explicitly allows
configuration during runtime, ie. without changing the index, by just
modifying the config.

How can I get the Query Elevation component to play nice with Master-Slave
replication? So far I've tried:

* Manually calling commit() after updating the elevate.xml (that's seems to
be no-op when the index hasn't changed: "No uncommitted changes. Skipping
IW.commit.")
* Manually calling update() after updating the elevate.xml (that's seems to
be no-op when the index hasn't changed: "No uncommitted changes. Skipping
IW.commit.")
* Manually calling
http://slave_host:port/solr/core_name/replication?command=fetchindex (doesn't
do anything either)

What can I do, short of inserting some dummy data into the index?

Re: How to replicate config files in master-slave replication without commit on master?

[Shawn Heisey <ap...@elyograg.org>]

On 9/6/2016 1:51 AM, Georg Sorst wrote:
> So if there is a lastmodified timestamp, it would be trivial to see which
> config files have been updated in the meantime. Any idea why the
> replication relies on an update to the index then? 

Master-slave replication was designed to copy an index.  It is designed
entirely around changes to the index.  Later, the ability to copy
changed config files was added ... but the core operation of the feature
is still based on changes to the *index*.

> In effect this means that the Query Elevation component does not work
> with master-slave replication. Are you aware of any efforts to remedy
> this? And finally, would this work with Solr Cloud? 

In SolrCloud, config files are not copied.  They are read from
zookeeper.  If you change the query elevation config and reload the
collection, *all* cores related to that collection are reloaded, and
they will all pick up the changes.

Thanks,
Shawn

Re: How to replicate config files in master-slave replication without commit on master?

[Georg Sorst <ge...@gmail.com>]

Erick Erickson <er...@gmail.com> schrieb am Di., 6. Sep. 2016 um
03:13 Uhr:

> Yes, replicating the ancillary files (e.g. elevate.xml) is contingent
> on the index
> changing. You wouldn't want these files copied down every time the slave
> polled the master, so the replication is part of index replication if
> (and only if) the
> index on the master has changed.
>

In the documentation for the master-slave replication it says:

> The slave issues a filelist command to get the list of the files. This
command returns the names of the files as well as some metadata (for
example, size, a lastmodified timestamp, an alias if any).

So if there is a lastmodified timestamp, it would be trivial to see which
config files have been updated in the meantime. Any idea why the
replication relies on an update to the index then?

In effect this means that the Query Elevation component does not work with
master-slave replication. Are you aware of any efforts to remedy this?

And finally, would this work with Solr Cloud?


>
> You could manually copy the files from the master to the slave perhaps?
>

We're already doing this with an awkward rsync setup. It works, but it's
ugly, and I was hoping for a better way....

Best,
Georg


>
> Best,
> Erick
>
> On Mon, Sep 5, 2016 at 7:49 AM, Georg Sorst <ge...@gmail.com> wrote:
> > Hi!
> >
> > According to
> >
> https://cwiki.apache.org/confluence/display/solr/Index+Replication#IndexReplication-ReplicatingConfigurationFiles
> > :
> >
> >> Solr replicates configuration files only when the index itself is
> > replicated. That means even if a configuration file is changed on the
> > master, that file will be replicated only after there is a new
> > commit/optimize on master's index.
> >
> > However, the behaviour of some components can be changed by just
> changing a
> > configuration file. A good example is the Query Elevation component,
> which
> > is configured through conf/elevate.xml. The component explicitly allows
> > configuration during runtime, ie. without changing the index, by just
> > modifying the config.
> >
> > How can I get the Query Elevation component to play nice with
> Master-Slave
> > replication? So far I've tried:
> >
> > * Manually calling commit() after updating the elevate.xml (that's seems
> to
> > be no-op when the index hasn't changed: "No uncommitted changes. Skipping
> > IW.commit.")
> > * Manually calling update() after updating the elevate.xml (that's seems
> to
> > be no-op when the index hasn't changed: "No uncommitted changes. Skipping
> > IW.commit.")
> > * Manually calling
> > http://slave_host:port/solr/core_name/replication?command=fetchindex
> (doesn't
> > do anything either)
> >
> > What can I do, short of inserting some dummy data into the index?
>

Re: How to replicate config files in master-slave replication without commit on master?

[Erick Erickson <er...@gmail.com>]

Yes, replicating the ancillary files (e.g. elevate.xml) is contingent
on the index
changing. You wouldn't want these files copied down every time the slave
polled the master, so the replication is part of index replication if
(and only if) the
index on the master has changed.

You could manually copy the files from the master to the slave perhaps?

Best,
Erick

On Mon, Sep 5, 2016 at 7:49 AM, Georg Sorst <ge...@gmail.com> wrote:
> Hi!
>
> According to
> https://cwiki.apache.org/confluence/display/solr/Index+Replication#IndexReplication-ReplicatingConfigurationFiles
> :
>
>> Solr replicates configuration files only when the index itself is
> replicated. That means even if a configuration file is changed on the
> master, that file will be replicated only after there is a new
> commit/optimize on master's index.
>
> However, the behaviour of some components can be changed by just changing a
> configuration file. A good example is the Query Elevation component, which
> is configured through conf/elevate.xml. The component explicitly allows
> configuration during runtime, ie. without changing the index, by just
> modifying the config.
>
> How can I get the Query Elevation component to play nice with Master-Slave
> replication? So far I've tried:
>
> * Manually calling commit() after updating the elevate.xml (that's seems to
> be no-op when the index hasn't changed: "No uncommitted changes. Skipping
> IW.commit.")
> * Manually calling update() after updating the elevate.xml (that's seems to
> be no-op when the index hasn't changed: "No uncommitted changes. Skipping
> IW.commit.")
> * Manually calling
> http://slave_host:port/solr/core_name/replication?command=fetchindex (doesn't
> do anything either)
>
> What can I do, short of inserting some dummy data into the index?