You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Liu Bo <di...@gmail.com> on 2013/09/30 11:24:44 UTC
documents are not commited distributively in solr cloud tomcat with
core discovery, range is null for shards in clusterstate.json
Hi all
I'm trying out the tutorial about solrcloud, and then I manage to write my
own plugin to import data from our set of databases, I use SolrWriter from
DataImporter package and the docs could be distributed commit to shards.
Every thing works fine using jetty from the solr example, but when I move
to tomcat, solrcloud seems not been configured right. As the documents are
just committed to the shard where update requested goes to.
The cause probably is the range is null for shards in clusterstate.json.
The router is "implicit" instead of "compositeId" as well.
Is there anything missed or configured wrong in the following steps? How
could I fix it. Your help will be much of my appreciation.
PS, solr cloud tomcat wiki page isn't up to 4.4 with core discovery, I'm
trying out after reading SoclrCloud, SolrCloudJboss, and CoreAdmin wiki
pages.
Here's what I've done and some useful logs:
1. start three zookeeper server.
2. upload configuration files to zookeeper, the collection name is
"content_collection"
3. start three tomcat instants on three server with core discovery
a) core file:
name=content
loadOnStartup=true
transient=false
shard=shard1 (differrent on servers)
collection=content_collection
b) solr.xml
<solr>
<solrcloud>
<str name="host">${host:}</str>
<str name="hostContext">${hostContext:solr}</str>
<int name="hostPort">8080</int>
<int name="zkClientTimeout">${zkClientTimeout:15000}</int>
<str name="zkHost">10.199.46.176:2181,10.199.46.165:2181,
10.199.46.158:2181</str>
<bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
</solrcloud>
<shardHandlerFactory name="shardHandlerFactory"
class="HttpShardHandlerFactory">
<int name="socketTimeout">${socketTimeout:0}</int>
<int name="connTimeout">${connTimeout:0}</int>
</shardHandlerFactory>
</solr>
4. In the solr.log, I see the three shards are recognized, and the
solrcloud can see the content_collection has three shards as well.
5. write documents to content_collection using my update request, the
documents only commits to the shard the request goes to, in the log I can
see the DistributedUpdateProcessorFactory is in the processorChain and
disribute commit is triggered:
INFO - 2013-09-30 16:31:43.205;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
updata request processor factories:
INFO - 2013-09-30 16:31:43.206;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.LogUpdateProcessorFactory@4ae7b77
INFO - 2013-09-30 16:31:43.207;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.*DistributedUpdateProcessorFactory*
@5b2bc407
INFO - 2013-09-30 16:31:43.207;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.RunUpdateProcessorFactory@1652d654
INFO - 2013-09-30 16:31:43.283; org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onInit: commits: num=1
commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1}
INFO - 2013-09-30 16:31:43.284; org.apache.solr.core.SolrDeletionPolicy;
newest commit generation = 1
INFO - 2013-09-30 16:31:43.440; *org.apache.solr.update.SolrCmdDistributor;
Distrib commit to*:[StdNode: http://10.199.46.176:8080/solr/content/,
StdNode: http://10.199.46.165:8080/solr/content/]
params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
but the documents won't go to other shards, the other shards only has a
request with not documents:
INFO - 2013-09-30 16:31:43.841;
org.apache.solr.update.DirectUpdateHandler2; start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onInit: commits: num=1
commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1}
INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy;
newest commit generation = 1
INFO - 2013-09-30 16:31:43.856;
org.apache.solr.update.DirectUpdateHandler2; No uncommitted changes.
Skipping IW.commit.
INFO - 2013-09-30 16:31:43.865; org.apache.solr.search.SolrIndexSearcher;
Opening Searcher@3c74c144 main
INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.QuerySenderListener;
QuerySenderListener sending requests to
Searcher@3c74c144main{StandardDirectoryReader(segments_1:1:nrt)}
INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.QuerySenderListener;
QuerySenderListener done.
INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.SolrCore; [content]
Registered new searcher
Searcher@3c74c144main{StandardDirectoryReader(segments_1:1:nrt)}
INFO - 2013-09-30 16:31:43.870;
org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
INFO - 2013-09-30 16:31:43.870;
org.apache.solr.update.processor.LogUpdateProcessor; [content] webapp=/solr
path=/update
params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false}
{commit=} 0 42
6) later I found the range is null in clusterstate.json which might have
caused the document isn't committed distributively
{"content_collection":{
"shards":{
"shard1":{
* "range":null,*
"state":"active",
"replicas":{"core_node1":{
"state":"active",
"core":"content",
"node_name":"10.199.46.176:8080_solr",
"base_url":"http://10.199.46.176:8080/solr",
"leader":"true"}}},
"shard3":{
* "range":null,*
"state":"active",
"replicas":{"core_node2":{
"state":"active",
"core":"content",
"node_name":"10.199.46.202:8080_solr",
"base_url":"http://10.199.46.202:8080/solr",
"leader":"true"}}},
"shard2":{
* "range":null,*
"state":"active",
"replicas":{"core_node3":{
"state":"active",
"core":"content",
"node_name":"10.199.46.165:8080_solr",
"base_url":"http://10.199.46.165:8080/solr",
"leader":"true"}}}},
*"router":"implicit"*}}
--
All the best
Liu Bo