You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Fermin Silva <fe...@olx.com> on 2014/03/27 14:35:11 UTC
Block until replication finishes
Hi,
we are moving to native replication with SOLR 3.5.1.
Because we want to control the replication from another program (a cron
job), we decided to curl the slave to issue a fetchIndex command.
The problem we have is that the curl returns immediately, while the
replication still goes in the background.
We need to know when the replication is done, and then resume the cron job.
Is there a way to block on the replication call until it's done similar to
waitForSearcher=true when committing ?
If not, what other possibilities we have?
Just in case, here is the solrconfig part in the slave (we pass masterUrl
in the curl url)
<requestHandler name="/replication" class="solr.ReplicationHandler">
<lst name="slave">
<str name="masterUrl"></str>
</lst>
</requestHandler>
Many thanks in advance
--
Fermin Silva
Re: Block until replication finishes
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
On Tue, Apr 1, 2014 at 5:02 PM, Fermin Silva <fe...@olx.com> wrote:
> Sorry but I have no clue about how to contribute with code. Will check that
> but if someone can point me to the right direction it would be nice.
>
You are welcome http://wiki.apache.org/solr/HowToContribute
Btw, cool finding re wait param! I didn't know it.
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: Block until replication finishes
Posted by Fermin Silva <fe...@olx.com>.
When trying to add the fix to the trunk version, I found that this was
already implemented.
There is a parameter '*wait*' that does exactly that.
if (solrParams.getBool(WAIT, false))
{
puller.join();
}
So the only possible way to do this in SOLR 3.x is to create a plugin with
a new replication handler (which I did) or re-compile SOLR.
On Tue, Apr 1, 2014 at 10:02 AM, Fermin Silva <fe...@olx.com> wrote:
> The ReplicationHandler class is not the most exemplar code to be looking
> at.
> I found however the line that could be changed:
>
> new Thread() {
> @Override
> public void run() {
> doFetch(paramsCopy);
> }
> }.start();
> rsp.add(STATUS, OK_STATUS);
>
> It should be really simple to join on that thread depending on a rest
> parameter.
> I would change that code myself (which I did to my custom SOLR
> installation) but I guess the fix should go for SOLR 4.x and not 3.x.
> Sorry but I have no clue about how to contribute with code. Will check
> that but if someone can point me to the right direction it would be nice.
>
> Thanks
>
>
> On Sat, Mar 29, 2014 at 9:49 AM, Mikhail Khludnev <
> mkhludnev@griddynamics.com> wrote:
>
>> Hello,
>> We did this for our fork, if you are not happy with "RESTful polling", or
>> think that the synchronous replication handler might be useful, please
>> raise a jira.
>> 27.03.2014 17:35 пользователь "Fermin Silva" <fe...@olx.com> написал:
>>
>> > Hi,
>> >
>> > we are moving to native replication with SOLR 3.5.1.
>> > Because we want to control the replication from another program (a cron
>> > job), we decided to curl the slave to issue a fetchIndex command.
>> >
>> > The problem we have is that the curl returns immediately, while the
>> > replication still goes in the background.
>> > We need to know when the replication is done, and then resume the cron
>> job.
>> >
>> > Is there a way to block on the replication call until it's done similar
>> to
>> > waitForSearcher=true when committing ?
>> > If not, what other possibilities we have?
>> >
>> > Just in case, here is the solrconfig part in the slave (we pass
>> masterUrl
>> > in the curl url)
>> >
>> > <requestHandler name="/replication" class="solr.ReplicationHandler">
>> > <lst name="slave">
>> > <str name="masterUrl"></str>
>> > </lst>
>> > </requestHandler>
>> >
>> >
>> > Many thanks in advance
>> >
>> > --
>> > Fermin Silva
>> >
>>
>
>
>
> --
> Fermin Silva
> Speed & Scalability Team
>
--
Fermin Silva
Speed & Scalability Team
Re: Block until replication finishes
Posted by Fermin Silva <fe...@olx.com>.
The ReplicationHandler class is not the most exemplar code to be looking at.
I found however the line that could be changed:
new Thread() {
@Override
public void run() {
doFetch(paramsCopy);
}
}.start();
rsp.add(STATUS, OK_STATUS);
It should be really simple to join on that thread depending on a rest
parameter.
I would change that code myself (which I did to my custom SOLR
installation) but I guess the fix should go for SOLR 4.x and not 3.x.
Sorry but I have no clue about how to contribute with code. Will check that
but if someone can point me to the right direction it would be nice.
Thanks
On Sat, Mar 29, 2014 at 9:49 AM, Mikhail Khludnev <
mkhludnev@griddynamics.com> wrote:
> Hello,
> We did this for our fork, if you are not happy with "RESTful polling", or
> think that the synchronous replication handler might be useful, please
> raise a jira.
> 27.03.2014 17:35 пользователь "Fermin Silva" <fe...@olx.com> написал:
>
> > Hi,
> >
> > we are moving to native replication with SOLR 3.5.1.
> > Because we want to control the replication from another program (a cron
> > job), we decided to curl the slave to issue a fetchIndex command.
> >
> > The problem we have is that the curl returns immediately, while the
> > replication still goes in the background.
> > We need to know when the replication is done, and then resume the cron
> job.
> >
> > Is there a way to block on the replication call until it's done similar
> to
> > waitForSearcher=true when committing ?
> > If not, what other possibilities we have?
> >
> > Just in case, here is the solrconfig part in the slave (we pass masterUrl
> > in the curl url)
> >
> > <requestHandler name="/replication" class="solr.ReplicationHandler">
> > <lst name="slave">
> > <str name="masterUrl"></str>
> > </lst>
> > </requestHandler>
> >
> >
> > Many thanks in advance
> >
> > --
> > Fermin Silva
> >
>
--
Fermin Silva
Speed & Scalability Team
Re: Block until replication finishes
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Hello,
We did this for our fork, if you are not happy with "RESTful polling", or
think that the synchronous replication handler might be useful, please
raise a jira.
27.03.2014 17:35 пользователь "Fermin Silva" <fe...@olx.com> написал:
> Hi,
>
> we are moving to native replication with SOLR 3.5.1.
> Because we want to control the replication from another program (a cron
> job), we decided to curl the slave to issue a fetchIndex command.
>
> The problem we have is that the curl returns immediately, while the
> replication still goes in the background.
> We need to know when the replication is done, and then resume the cron job.
>
> Is there a way to block on the replication call until it's done similar to
> waitForSearcher=true when committing ?
> If not, what other possibilities we have?
>
> Just in case, here is the solrconfig part in the slave (we pass masterUrl
> in the curl url)
>
> <requestHandler name="/replication" class="solr.ReplicationHandler">
> <lst name="slave">
> <str name="masterUrl"></str>
> </lst>
> </requestHandler>
>
>
> Many thanks in advance
>
> --
> Fermin Silva
>
Re: Block until replication finishes
Posted by Chris W <ch...@gmail.com>.
You can use a timeout? In our system we do that same but with a timeout of
around 30 minutes for any replication to slave completes. So far any
replication > 30 minutes (even for cores as big as 10G) indicated a
problem with a core or network on our side
On Fri, Mar 28, 2014 at 6:43 AM, Fermin Silva <fe...@olx.com> wrote:
> Hi,
>
> that's what I'm trying. I'm however really cautious when it comes to a
> while (somethingIsTrue) { doSomething; sleep; }
>
> Is that safe? What if the slave hungs up, the network is slow/fails, etc?
>
> Thanks
>
>
> On Thu, Mar 27, 2014 at 1:40 PM, Chris W <ch...@gmail.com> wrote:
>
> > Hi
> >
> > You can use the "details" command to check the status of replication.
> > http://localhost:8983/solr/core_name/replication?command=details
> >
> > The command returns an xml output and look out for the "isReplicating"
> > field in the output. Keep running the command in a loop until the flag
> > becomes false. Thats when you know its done. I would also recommend you
> to
> > check the # of docs in the output at source/destination after the
> > replication to be sure
> >
> >
> > HTH
> >
> >
> >
> >
> > On Thu, Mar 27, 2014 at 6:35 AM, Fermin Silva <fe...@olx.com> wrote:
> >
> > > Hi,
> > >
> > > we are moving to native replication with SOLR 3.5.1.
> > > Because we want to control the replication from another program (a cron
> > > job), we decided to curl the slave to issue a fetchIndex command.
> > >
> > > The problem we have is that the curl returns immediately, while the
> > > replication still goes in the background.
> > > We need to know when the replication is done, and then resume the cron
> > job.
> > >
> > > Is there a way to block on the replication call until it's done similar
> > to
> > > waitForSearcher=true when committing ?
> > > If not, what other possibilities we have?
> > >
> > > Just in case, here is the solrconfig part in the slave (we pass
> masterUrl
> > > in the curl url)
> > >
> > > <requestHandler name="/replication" class="solr.ReplicationHandler">
> > > <lst name="slave">
> > > <str name="masterUrl"></str>
> > > </lst>
> > > </requestHandler>
> > >
> > >
> > > Many thanks in advance
> > >
> > > --
> > > Fermin Silva
> > >
> >
> >
> >
> > --
> > Best
> > --
> > C
> >
>
>
>
> --
> Fermin Silva
> Speed & Scalability Team
>
--
Best
--
C
Re: Block until replication finishes
Posted by Fermin Silva <fe...@olx.com>.
Hi,
that's what I'm trying. I'm however really cautious when it comes to a
while (somethingIsTrue) { doSomething; sleep; }
Is that safe? What if the slave hungs up, the network is slow/fails, etc?
Thanks
On Thu, Mar 27, 2014 at 1:40 PM, Chris W <ch...@gmail.com> wrote:
> Hi
>
> You can use the "details" command to check the status of replication.
> http://localhost:8983/solr/core_name/replication?command=details
>
> The command returns an xml output and look out for the "isReplicating"
> field in the output. Keep running the command in a loop until the flag
> becomes false. Thats when you know its done. I would also recommend you to
> check the # of docs in the output at source/destination after the
> replication to be sure
>
>
> HTH
>
>
>
>
> On Thu, Mar 27, 2014 at 6:35 AM, Fermin Silva <fe...@olx.com> wrote:
>
> > Hi,
> >
> > we are moving to native replication with SOLR 3.5.1.
> > Because we want to control the replication from another program (a cron
> > job), we decided to curl the slave to issue a fetchIndex command.
> >
> > The problem we have is that the curl returns immediately, while the
> > replication still goes in the background.
> > We need to know when the replication is done, and then resume the cron
> job.
> >
> > Is there a way to block on the replication call until it's done similar
> to
> > waitForSearcher=true when committing ?
> > If not, what other possibilities we have?
> >
> > Just in case, here is the solrconfig part in the slave (we pass masterUrl
> > in the curl url)
> >
> > <requestHandler name="/replication" class="solr.ReplicationHandler">
> > <lst name="slave">
> > <str name="masterUrl"></str>
> > </lst>
> > </requestHandler>
> >
> >
> > Many thanks in advance
> >
> > --
> > Fermin Silva
> >
>
>
>
> --
> Best
> --
> C
>
--
Fermin Silva
Speed & Scalability Team
Re: Block until replication finishes
Posted by Chris W <ch...@gmail.com>.
Hi
You can use the "details" command to check the status of replication.
http://localhost:8983/solr/core_name/replication?command=details
The command returns an xml output and look out for the "isReplicating"
field in the output. Keep running the command in a loop until the flag
becomes false. Thats when you know its done. I would also recommend you to
check the # of docs in the output at source/destination after the
replication to be sure
HTH
On Thu, Mar 27, 2014 at 6:35 AM, Fermin Silva <fe...@olx.com> wrote:
> Hi,
>
> we are moving to native replication with SOLR 3.5.1.
> Because we want to control the replication from another program (a cron
> job), we decided to curl the slave to issue a fetchIndex command.
>
> The problem we have is that the curl returns immediately, while the
> replication still goes in the background.
> We need to know when the replication is done, and then resume the cron job.
>
> Is there a way to block on the replication call until it's done similar to
> waitForSearcher=true when committing ?
> If not, what other possibilities we have?
>
> Just in case, here is the solrconfig part in the slave (we pass masterUrl
> in the curl url)
>
> <requestHandler name="/replication" class="solr.ReplicationHandler">
> <lst name="slave">
> <str name="masterUrl"></str>
> </lst>
> </requestHandler>
>
>
> Many thanks in advance
>
> --
> Fermin Silva
>
--
Best
--
C