You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Liu Bo <di...@gmail.com> on 2013/10/06 17:53:54 UTC
documents are not commited distributively in solr cloud tomcat with
core discovery, range is null for shards in clusterstate.json
Hi all
I've sent out this mail before, but I only subscribed to lucene-user but
not solr-user at that time. Sorry for repeating if any and your help will
be much of my appreciation.
I'm trying out the tutorial about solrcloud, and then I manage to write my
own plugin to import data from our set of databases, I use SolrWriter from
DataImporter package and the docs could be distributed commit to shards.
Every thing works fine using jetty from the solr example, but when I move
to tomcat, solrcloud seems not been configured right. As the documents are
just committed to the shard where update requested goes to.
The cause probably is the range is null for shards in clusterstate.json.
The router is "implicit" instead of "compositeId" as well.
Is there anything missed or configured wrong in the following steps? How
could I fix it. Your help will be much of my appreciation.
PS, solr cloud tomcat wiki page isn't up to 4.4 with core discovery, I'm
trying out after reading SoclrCloud, SolrCloudJboss, and CoreAdmin wiki
pages.
Here's what I've done and some useful logs:
1. start three zookeeper server.
2. upload configuration files to zookeeper, the collection name is
"content_collection"
3. start three tomcat instants on three server with core discovery
a) core file:
name=content
loadOnStartup=true
transient=false
shard=shard1 (differrent on servers)
collection=content_collection
b) solr.xml
<solr>
<solrcloud>
<str name="host">${host:}</str>
<str name="hostContext">${hostContext:solr}</str>
<int name="hostPort">8080</int>
<int name="zkClientTimeout">${zkClientTimeout:15000}</int>
<str name="zkHost">10.199.46.176:2181,10.199.46.165:2181,
10.199.46.158:2181</str>
<bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
</solrcloud>
<shardHandlerFactory name="shardHandlerFactory"
class="HttpShardHandlerFactory">
<int name="socketTimeout">${socketTimeout:0}</int>
<int name="connTimeout">${connTimeout:0}</int>
</shardHandlerFactory>
</solr>
4. In the solr.log, I see the three shards are recognized, and the
solrcloud can see the content_collection has three shards as well.
5. write documents to content_collection using my update request, the
documents only commits to the shard the request goes to, in the log I can
see the DistributedUpdateProcessorFactory is in the processorChain and
disribute commit is triggered:
INFO - 2013-09-30 16:31:43.205;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
updata request processor factories:
INFO - 2013-09-30 16:31:43.206;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.LogUpdateProcessorFactory@4ae7b77
INFO - 2013-09-30 16:31:43.207;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.*DistributedUpdateProcessorFactory*
@5b2bc407
INFO - 2013-09-30 16:31:43.207;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.RunUpdateProcessorFactory@1652d654
INFO - 2013-09-30 16:31:43.283; org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onInit: commits: num=1
commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1}
INFO - 2013-09-30 16:31:43.284; org.apache.solr.core.SolrDeletionPolicy;
newest commit generation = 1
INFO - 2013-09-30 16:31:43.440; *org.apache.solr.update.SolrCmdDistributor;
Distrib commit to*:[StdNode: http://10.199.46.176:8080/solr/content/,
StdNode: http://10.199.46.165:8080/solr/content/]
params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
but the documents won't go to other shards, the other shards only has a
request with not documents:
INFO - 2013-09-30 16:31:43.841;
org.apache.solr.update.DirectUpdateHandler2; start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onInit: commits: num=1
commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1}
INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy;
newest commit generation = 1
INFO - 2013-09-30 16:31:43.856;
org.apache.solr.update.DirectUpdateHandler2; No uncommitted changes.
Skipping IW.commit.
INFO - 2013-09-30 16:31:43.865; org.apache.solr.search.SolrIndexSearcher;
Opening Searcher@3c74c144 main
INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.QuerySenderListener;
QuerySenderListener sending requests to
Searcher@3c74c144main{StandardDirectoryReader(segments_1:1:nrt)}
INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.QuerySenderListener;
QuerySenderListener done.
INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.SolrCore; [content]
Registered new searcher
Searcher@3c74c144main{StandardDirectoryReader(segments_1:1:nrt)}
INFO - 2013-09-30 16:31:43.870;
org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
INFO - 2013-09-30 16:31:43.870;
org.apache.solr.update.processor.LogUpdateProcessor; [content] webapp=/solr
path=/update
params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false}
{commit=} 0 42
6) later I found the range is null in clusterstate.json which might have
caused the document isn't committed distributively
{"content_collection":{
"shards":{
"shard1":{
* "range":null,*
"state":"active",
"replicas":{"core_node1":{
"state":"active",
"core":"content",
"node_name":"10.199.46.176:8080_solr",
"base_url":"http://10.199.46.176:8080/solr",
"leader":"true"}}},
"shard3":{
* "range":null,*
"state":"active",
"replicas":{"core_node2":{
"state":"active",
"core":"content",
"node_name":"10.199.46.202:8080_solr",
"base_url":"http://10.199.46.202:8080/solr",
"leader":"true"}}},
"shard2":{
* "range":null,*
"state":"active",
"replicas":{"core_node3":{
"state":"active",
"core":"content",
"node_name":"10.199.46.165:8080_solr",
"base_url":"http://10.199.46.165:8080/solr",
"leader":"true"}}}},
*"router":"implicit"*}}
--
All the best
Liu Bo
Re: documents are not commited distributively in solr cloud tomcat
with core discovery, range is null for shards in clusterstate.json
Posted by Liu Bo <di...@gmail.com>.
I've solved this problem myself.
If you use core discovery, you must specify the "numShards" parameter in
core.properties.
or else solr won't be allocate range for each shards and then documents
won't be distributed properly.
Using core discovery to set up solr cloud in tomcat is much easier and
clean than coreAdmin described in the wiki:
http://wiki.apache.org/solr/SolrCloudTomcat.
It costs me some time to move from jetty to tomcat, but I think our IT team
will like this way. :)
On 6 October 2013 23:53, Liu Bo <di...@gmail.com> wrote:
> Hi all
>
> I've sent out this mail before, but I only subscribed to lucene-user but
> not solr-user at that time. Sorry for repeating if any and your help will
> be much of my appreciation.
>
> I'm trying out the tutorial about solrcloud, and then I manage to write my
> own plugin to import data from our set of databases, I use SolrWriter from
> DataImporter package and the docs could be distributed commit to shards.
>
> Every thing works fine using jetty from the solr example, but when I move
> to tomcat, solrcloud seems not been configured right. As the documents are
> just committed to the shard where update requested goes to.
>
> The cause probably is the range is null for shards in clusterstate.json.
> The router is "implicit" instead of "compositeId" as well.
>
> Is there anything missed or configured wrong in the following steps? How
> could I fix it. Your help will be much of my appreciation.
>
> PS, solr cloud tomcat wiki page isn't up to 4.4 with core discovery, I'm
> trying out after reading SoclrCloud, SolrCloudJboss, and CoreAdmin wiki
> pages.
>
> Here's what I've done and some useful logs:
>
> 1. start three zookeeper server.
> 2. upload configuration files to zookeeper, the collection name is
> "content_collection"
> 3. start three tomcat instants on three server with core discovery
>
> a) core file:
> name=content
> loadOnStartup=true
> transient=false
> shard=shard1 (differrent on servers)
> collection=content_collection
> b) solr.xml
>
> <solr>
>
> <solrcloud>
>
> <str name="host">${host:}</str>
>
> <str name="hostContext">${hostContext:solr}</str>
>
> <int name="hostPort">8080</int>
>
> <int name="zkClientTimeout">${zkClientTimeout:15000}</int>
>
> <str name="zkHost">10.199.46.176:2181,10.199.46.165:2181,
> 10.199.46.158:2181</str>
>
> <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
>
> </solrcloud>
>
>
> <shardHandlerFactory name="shardHandlerFactory"
>
> class="HttpShardHandlerFactory">
>
> <int name="socketTimeout">${socketTimeout:0}</int>
>
> <int name="connTimeout">${connTimeout:0}</int>
>
> </shardHandlerFactory>
>
> </solr>
>
> 4. In the solr.log, I see the three shards are recognized, and the
> solrcloud can see the content_collection has three shards as well.
> 5. write documents to content_collection using my update request, the
> documents only commits to the shard the request goes to, in the log I can
> see the DistributedUpdateProcessorFactory is in the processorChain and
> disribute commit is triggered:
>
> INFO - 2013-09-30 16:31:43.205;
> com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
> updata request processor factories:
>
> INFO - 2013-09-30 16:31:43.206;
> com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
> org.apache.solr.update.processor.LogUpdateProcessorFactory@4ae7b77
>
> INFO - 2013-09-30 16:31:43.207;
> com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
> org.apache.solr.update.processor.*DistributedUpdateProcessorFactory*
> @5b2bc407
>
> INFO - 2013-09-30 16:31:43.207;
> com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
> org.apache.solr.update.processor.RunUpdateProcessorFactory@1652d654
>
> INFO - 2013-09-30 16:31:43.283; org.apache.solr.core.SolrDeletionPolicy;
> SolrDeletionPolicy.onInit: commits: num=1
>
>
> commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1}
>
> INFO - 2013-09-30 16:31:43.284; org.apache.solr.core.SolrDeletionPolicy;
> newest commit generation = 1
>
> INFO - 2013-09-30 16:31:43.440; *org.apache.solr.update.SolrCmdDistributor;
> Distrib commit to*:[StdNode: http://10.199.46.176:8080/solr/content/,
> StdNode: http://10.199.46.165:8080/solr/content/]
> params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
>
> but the documents won't go to other shards, the other shards only has a
> request with not documents:
>
> INFO - 2013-09-30 16:31:43.841;
> org.apache.solr.update.DirectUpdateHandler2; start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>
> INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy;
> SolrDeletionPolicy.onInit: commits: num=1
>
>
> commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1}
>
> INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy;
> newest commit generation = 1
>
> INFO - 2013-09-30 16:31:43.856;
> org.apache.solr.update.DirectUpdateHandler2; No uncommitted changes.
> Skipping IW.commit.
>
> INFO - 2013-09-30 16:31:43.865; org.apache.solr.search.SolrIndexSearcher;
> Opening Searcher@3c74c144 main
>
> INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.QuerySenderListener;
> QuerySenderListener sending requests to Searcher@3c74c144main{StandardDirectoryReader(segments_1:1:nrt)}
>
> INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.QuerySenderListener;
> QuerySenderListener done.
>
> INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.SolrCore; [content]
> Registered new searcher Searcher@3c74c144main{StandardDirectoryReader(segments_1:1:nrt)}
>
> INFO - 2013-09-30 16:31:43.870;
> org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
>
> INFO - 2013-09-30 16:31:43.870;
> org.apache.solr.update.processor.LogUpdateProcessor; [content] webapp=/solr
> path=/update
> params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false}
> {commit=} 0 42
>
> 6) later I found the range is null in clusterstate.json which might have
> caused the document isn't committed distributively
>
> {"content_collection":{
>
> "shards":{
>
> "shard1":{
>
> * "range":null,*
>
> "state":"active",
>
> "replicas":{"core_node1":{
>
> "state":"active",
>
> "core":"content",
>
> "node_name":"10.199.46.176:8080_solr",
>
> "base_url":"http://10.199.46.176:8080/solr",
>
> "leader":"true"}}},
>
> "shard3":{
>
> * "range":null,*
>
> "state":"active",
>
> "replicas":{"core_node2":{
>
> "state":"active",
>
> "core":"content",
>
> "node_name":"10.199.46.202:8080_solr",
>
> "base_url":"http://10.199.46.202:8080/solr",
>
> "leader":"true"}}},
>
> "shard2":{
>
> * "range":null,*
>
> "state":"active",
>
> "replicas":{"core_node3":{
>
> "state":"active",
>
> "core":"content",
>
> "node_name":"10.199.46.165:8080_solr",
>
> "base_url":"http://10.199.46.165:8080/solr",
>
> "leader":"true"}}}},
>
> *"router":"implicit"*}}
>
>
>
> --
> All the best
>
> Liu Bo
>
>
--
All the best
Liu Bo