You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Uwe Reh <re...@hebis.uni-frankfurt.de> on 2016/06/28 14:36:44 UTC
CDCR (Solr6.x) does not start
Hi,
I'm trying to get CDCR to run, but I can't even trigger any
communication between SOURCE and TARGET.
It seems to be a small but grave misunderstanding. I've tested a lot of
variants but now I'm blind on this point.
If anyone could give me a hint, I would appreciate.
Uwe
Testsetting:
Two nearly identical hosts (open solaris) with:
- a minimal zookeeper ensemble (one local installation (not embedded),
listening on port 2181)
- a minimal cloud (one node, one empty collection, 4 shards)
Initial both installations differ only in solrconfig.xml (snipplets below)
The tcp traffic was observed with 'snoop' (tcpdump). There are no packet
filters or other firewalls between both machines.
Testprocess:
1. Start node for TARGET
2. Create TARGET collection 'tcoll'
http://TARGET:t_port/solr/admin/collections?action=CREATE&name=tcoll&numShards=4&replicationFactor=1&maxShardsPerNode=4&collection.configName=cdcr
3. Get status
http://TARGET:t_port/solr/tcoll/cdcr?action=status&wt=json
> {"responseHeader":{"status":0,"QTime":0},"status":["process","stopped","buffer","enabled"]}
4. Disable buffer
http://TARGET:t_port/solr/tcoll/cdcr?action=disablebuffer&wt=json
> {"responseHeader":{"status":0,"QTime":12},"status":["process","stopped","buffer","disabled"]}
6. Start node for SOURCE
(like expected, no tcp between both hosts)
7. Create SOURCE collection 'scoll'
http://SOURCE:s_port/solr/admin/collections?action=CREATE&name=scoll&numShards=4&replicationFactor=1&maxShardsPerNode=4&collection.configName=cdcr
(no tcp between both hosts)
8. Get status
http://SOURCE:s_port/solr/scoll/cdcr?action=status&wt=json
> {"responseHeader":{"status":0,"QTime":13},"status":["process","stopped","buffer","enabled"]}
(like expected, no tcp between both hosts)
9. Start CDCR
http://SOURCE:s_port/solr/scoll/cdcr?action=start&wt=json
> {"responseHeader":{"status":0,"QTime":13},"status":["process","started","buffer","enabled"]}
! (not even a single query to the target's zookeeper ??)
10. Enter some test data into the SOURCE
11. Explicit commit in SOURCE
http://SOURCE:s_port/solr/scoll/update?commit=true&opensearcher=true
!! (at least now there should be some traffic, or?)
12. Check errors and queues
http://SOURCE:s_port/solr/scoll_shard1_replica1/cdcr?action=queues&wt=json
> {"responseHeader":{"status":0,"QTime":0},"queues":[],"tlogTotalSize":135,"tlogTotalCount":1,"updateLogSynchronizer":"stopped"}
http://SOURCE:s_port/solr/scoll_shard1_replica1/cdcr?action=errors&wt=json
> {"responseHeader":{"status":0,"QTime":0},"errors":[]}
! Why is the element queues is empty
???? where is my stupid bug ????
####################
# solrconfig Source
####################
<updateHandler class="solr.DirectUpdateHandler2">
<updateLog class="solr.CdcrUpdateLog">
<str name="dir">${solr.ulog.dir:}</str>
</updateLog>
</updateHandler>
<requestHandler name="/cdcr" class="solr.CdcrRequestHandler">
<lst name="replica">
<str name="zkHost">TARGET:2181</str>
<str name="Source">scoll</str>
<str name="Target">tcoll</str>
</lst>
<lst name="replicator">
<str name="batchSize">1</str> <!-- only for testing -->
</lst>
</requestHandler>
####################
# solrconfig Target
####################
<updateHandler class="solr.DirectUpdateHandler2">
<updateLog class="solr.CdcrUpdateLog">
<str name="dir">${solr.ulog.dir:}</str>
</updateLog>
<autoCommit>
<maxDocs>${solr.autoCommit.maxdocs:1000}</maxDocs>
<maxTime>${solr.autoCommit.maxTime:300}</maxTime>
<openSearcher>true</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:60}</maxTime>
</autoSoftCommit>
</updateHandler>
<requestHandler name="/cdcr" class="solr.CdcrRequestHandler">
<lst name="buffer">
<str name="defaultState">disabled</str>
</lst> <!-- why is this default ignored? -->
</requestHandler>
<requestHandler name="/update" class="solr.UpdateRequestHandler">
<lst name="defaults">
<str name="update.chain">cdcr-processor-chain</str>
</lst>
</requestHandler>
<updateRequestProcessorChain name="cdcr-processor-chain">
<processor class="solr.CdcrUpdateProcessorFactory"/>
<processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>
######################################
# EOF
#####
Re: CDCR (Solr6.x) does not start (logfile)
Posted by Uwe Reh <re...@hebis.uni-frankfurt.de>.
Hi,
trying to get more information, I restarted the SOURCE node and watched
the log.
For each shard i got following triple:
> WARN org.apache.solr.handler.CdcrRequestHandler - Action LASTPROCESSEDVERSION sent to non-leader replica @ scoll:shard1
> ERROR org.apache.solr.handler.RequestHandlerBase - org.apache.solr.common.SolrException: Action LASTPROCESSEDVERSION sent to non-leader replica
> WARN org.apache.solr.handler.CdcrUpdateLogSynchronizer - Caught unexpected exception
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://SOURCE:s_port/solr/scoll_shard1_replica1: Action LASTPROCESSEDVERSION sent to non-leader replica
Could this the reason, why there is no further action?
The SOURCE cloud has just the replicationfactor '1'.
'scoll_shard1_replica1' should have to be allays the leader, or?
Regards
Uwe
Re: CDCR (Solr6.x) does not start
Posted by Uwe Reh <re...@hebis.uni-frankfurt.de>.
Hi Renaud,
thank you for your response.
You asked for some further information:
1. Log messages at the source cluster:
As mentioned in my addendum "CDCR (Solr6.x) does not start (logfile)". I
changed the log level for all Handlers to TRACE and I got three Messages
for each shard caused by "Action LASTPROCESSEDVERSION sent to non-leader
replica .."
For me this looks like the blocker.
2.
> Replication should start even if no commit has been sent to the source cluster.
Thanks for the clarification. It helps me to understand.
3.
>> The empty queue seems to indicate there is an issue, and that cdcr was unable to instantiate the replicator for the target cluster.
>> Just to be sure, your source cluster has 4 shards, but not replica ? If it has replicas, can you ensure that you execute these command on the shard leader.
At the beginning I tried to replicate 4 shards with an replication
factor of 3. Later on i simplified the environment by omitting the
replicas. (replication factor = 1)
Do you think having no replicas could the reason for the log messages above?
Regards
Uwe
Am 05.07.2016 um 14:55 schrieb Renaud Delbru:
> Hi Uwe,
>
> At first look, your configuration seems correct,
> see my comments below.
>
> On 28/06/16 15:36, Uwe Reh wrote:
>> 9. Start CDCR
>> http://SOURCE:s_port/solr/scoll/cdcr?action=start&wt=json
>>> {"responseHeader":{"status":0,"QTime":13},"status":["process","started","buffer","enabled"]}
>>>
>>>
>> ! (not even a single query to the target's zookeeper ??)
>
> Indeed, you should have observed a communication between the source
> cluster and the target zookeeper. Do you see any errors in the log of
> the source cluster ? Or a log message such as:
> "Unable to instantiate the log reader for target collection ..."
>
>>
>> 10. Enter some test data into the SOURCE
>>
>> 11. Explicit commit in SOURCE
>> http://SOURCE:s_port/solr/scoll/update?commit=true&opensearcher=true
>> !! (at least now there should be some traffic, or?)
>
> Replication should start even if no commit has been sent to the source
> cluster.
>
>>
>> 12. Check errors and queues
>> http://SOURCE:s_port/solr/scoll_shard1_replica1/cdcr?action=queues&wt=json
>>
>>> {"responseHeader":{"status":0,"QTime":0},"queues":[],"tlogTotalSize":135,"tlogTotalCount":1,"updateLogSynchronizer":"stopped"}
>>>
>>>
>> http://SOURCE:s_port/solr/scoll_shard1_replica1/cdcr?action=errors&wt=json
>>
>>> {"responseHeader":{"status":0,"QTime":0},"errors":[]}
>> ! Why is the element queues is empty
>
> The empty queue seems to indicate there is an issue, and that cdcr was
> unable to instantiate the replicator for the target cluster.
> Just to be sure, your source cluster has 4 shards, but not replica ? If
> it has replicas, can you ensure that you execute these command on the
> shard leader.
>
> Kind Regards
Re: CDCR (Solr6.x) does not start
Posted by Renaud Delbru <re...@siren.solutions>.
Hi Uwe,
At first look, your configuration seems correct,
see my comments below.
On 28/06/16 15:36, Uwe Reh wrote:
> 9. Start CDCR
> http://SOURCE:s_port/solr/scoll/cdcr?action=start&wt=json
>> {"responseHeader":{"status":0,"QTime":13},"status":["process","started","buffer","enabled"]}
>>
> ! (not even a single query to the target's zookeeper ??)
Indeed, you should have observed a communication between the source
cluster and the target zookeeper. Do you see any errors in the log of
the source cluster ? Or a log message such as:
"Unable to instantiate the log reader for target collection ..."
>
> 10. Enter some test data into the SOURCE
>
> 11. Explicit commit in SOURCE
> http://SOURCE:s_port/solr/scoll/update?commit=true&opensearcher=true
> !! (at least now there should be some traffic, or?)
Replication should start even if no commit has been sent to the source
cluster.
>
> 12. Check errors and queues
> http://SOURCE:s_port/solr/scoll_shard1_replica1/cdcr?action=queues&wt=json
>> {"responseHeader":{"status":0,"QTime":0},"queues":[],"tlogTotalSize":135,"tlogTotalCount":1,"updateLogSynchronizer":"stopped"}
>>
> http://SOURCE:s_port/solr/scoll_shard1_replica1/cdcr?action=errors&wt=json
>> {"responseHeader":{"status":0,"QTime":0},"errors":[]}
> ! Why is the element queues is empty
The empty queue seems to indicate there is an issue, and that cdcr was
unable to instantiate the replicator for the target cluster.
Just to be sure, your source cluster has 4 shards, but not replica ? If
it has replicas, can you ensure that you execute these command on the
shard leader.
Kind Regards
--
Renaud Delbru
Re: CDCR (Solr6.x) does not start
Posted by neerajbhatt <ne...@gmail.com>.
Hi Uwe
I am facing the same error as you , I am getting error 6538 ERROR
(qtp110456297-20) [c:multi_dc_poc s:shard1 r:core_node3
x:multi_dc_poc_shard1_replica2] o.a.s.h.RequestHandlerBase
org.apache.solr.common.SolrException: Action LASTPROCESSEDVERSION sent to
non-leader replica
at
org.apache.solr.handler.CdcrRequestHandler.handleLastProcessedVersionAction(CdcrRequestHandler.java:498)
- I have 3 shards and 3 replicas in 3 solr machines
Can you please help
Thanks
Neeraj
--
View this message in context: http://lucene.472066.n3.nabble.com/CDCR-Solr6-x-does-not-start-tp4284742p4305205.html
Sent from the Solr - User mailing list archive at Nabble.com.