You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Marc Campeau <ca...@gmail.com> on 2014/05/30 15:54:45 UTC
SolrCloud: Understanding Replication
Hi,
I currently have a standalone SOLR 4.5.1deployment on an EC2 instance with
a single collection and core containing an index that's roughly 10G. I've
used this as a proof of concept, prototype and staging during development
phases and I'm about to release to production.
For this release, I've setup 4 EC2 instances with 3 servers in Zookeeper
ensemble and the 4 servers running SolrCloud. My intention is to have my
current collection on a single Shard replicated 4 times based on the high
availability requirements. For that, I'm using an ELB as load balancer to
spread the query load to all 4 instances. For this, I've rsync'ed my
current 10G Collection to all 4 SOLR instances in my SolrCloud and started
them all up. They all come up and do the elections and what nots and all
are queryable, which is great. The idea being to load they current index as
it is and then start updating it instead of reindexing it all from scratch.
BUT...
1) Using zkCLI, I can see that clusterstate shows all instances as down and
this is illustrated on the Solr Admin interface by showing all 4 instances
using the down color. Is that normal? How can I change that? How come if
all 4 instances answer queries just fine?
2) It doesn't seem like the instances are replicating... aka if I add a
document to the collection it doesn't get replicated to the other
instances. Why is that? What should I look for in SOLR logs that would tell
me that replication is happening? I clearly see in there the "/admin/ping"
requests made by the load balancer doing health checks and requests made to
the admin interface but can never find requests made to "/replicate" that
would trigger the replication handler.
There's obviously something I've done wrong put I can't put my finger on
it. I would appreciate your insight on my situation.
Thanks,
Marc Campeau
Re: SolrCloud: Understanding Replication
Posted by Marc Campeau <ca...@gmail.com>.
Hi,
first I'd like to thank those who've spent their time reading this,
especially Erick, Jason and Shawn. Thank you!
I've finally got it working. Shawn was right, I needed to enable updateLog
in solrconfig.xml and create the tlog directory. After doing, so I could
index documents sent on any node, they would go to the leader which indexed
them but they still weren't being replicated to the replicas.
After looking at the logs, I remembered some thing I had read somewhere
(would like to include link here but can't find it) about when using a
Zookeeper ensemble and wanting to reset your Solr Collection you should not
only clear your data folder and tlog folder but you should also clear out
your Zookeeper data folders on all members of ensemble. So I stopped all
Solr Instances in cluster, and then all members of Zookeeper ensemble.
Deleted the Zookeeper data/version_2 folder on all Zookeeper instances.
Deleted the Solr index data on all Solr instances. Restarted the Zookeeper
Ensemble, pushed the SOLR Configs to it and started the remainder of the
Solr cluster. That's when it all started working. Documents I had started
indexing that were solely on the leader got replicated to replicas and then
any document sent to any member of cluster would be indexed and replicated.
So lessons learned:
1) Don't forget to enable updateLog. Needs to be tested: an index that
doesn't have updateLog associated to it cannot be made into replica, or at
least cannot just be copied over to replica nodes.
2) When you want to reset a collection, it's a good idea to reset the
zookeeper ensemble data, especially if your Solr Cluster wasn't replicating.
Marc
2014-06-02 16:31 GMT-04:00 Shawn Heisey <so...@elyograg.org>:
> On 6/2/2014 2:21 PM, Marc Campeau wrote:
> > I notice I have this in the logs when I start SOLR for default example (I
> > had the same with my own connection)
> >
> > 21242 [coreZkRegister-1-thread-1] INFO
> > org.apache.solr.cloud.ShardLeaderElectionContext – Enough replicas
> found
> > to continue.
> > 21242 [coreZkRegister-1-thread-1] INFO
> > org.apache.solr.cloud.ShardLeaderElectionContext – I may be the new
> > leader - try and sync
> > 21242 [coreZkRegister-1-thread-1] INFO
> org.apache.solr.cloud.SyncStrategy
> > – Sync replicas to http://192.168.150.90:8983/solr/collection1/
> > 21243 [coreZkRegister-1-thread-1] ERROR
> org.apache.solr.cloud.SyncStrategy
> > – No UpdateLog found - cannot sync
> > 21243 [coreZkRegister-1-thread-1] INFO
> > org.apache.solr.cloud.ShardLeaderElectionContext – We failed sync, but
> we
> > have no versions - we can't sync in that case - we were active before, so
> > become leader anyway
> > 21243 [coreZkRegister-1-thread-1] INFO
> > org.apache.solr.cloud.ShardLeaderElectionContext – I am the new leader:
> > http://192.168.150.90:8983/solr/collection1/ shard1
> > 21243 [coreZkRegister-1-thread-1] INFO
> > org.apache.solr.common.cloud.SolrZkClient – makePath:
> > /collections/collection1/leaders/shard1
>
> The "No UpdateLog found - cannot sync" message suggests that either you
> have disabled the updateLog feature in solrconfig.xml or that you have
> deleted the tlog directory (or possibly the entire data directory).
> Hopefully it's the latter, because SolrCloud cannot function properly if
> the updateLog is disabled. For data integrity reasons, the updateLog is
> always recommended, whether using SolrCloud or not.
>
> Thanks,
> Shawn
>
>
Re: SolrCloud: Understanding Replication
Posted by Shawn Heisey <so...@elyograg.org>.
On 6/2/2014 2:21 PM, Marc Campeau wrote:
> I notice I have this in the logs when I start SOLR for default example (I
> had the same with my own connection)
>
> 21242 [coreZkRegister-1-thread-1] INFO
> org.apache.solr.cloud.ShardLeaderElectionContext – Enough replicas found
> to continue.
> 21242 [coreZkRegister-1-thread-1] INFO
> org.apache.solr.cloud.ShardLeaderElectionContext – I may be the new
> leader - try and sync
> 21242 [coreZkRegister-1-thread-1] INFO org.apache.solr.cloud.SyncStrategy
> – Sync replicas to http://192.168.150.90:8983/solr/collection1/
> 21243 [coreZkRegister-1-thread-1] ERROR org.apache.solr.cloud.SyncStrategy
> – No UpdateLog found - cannot sync
> 21243 [coreZkRegister-1-thread-1] INFO
> org.apache.solr.cloud.ShardLeaderElectionContext – We failed sync, but we
> have no versions - we can't sync in that case - we were active before, so
> become leader anyway
> 21243 [coreZkRegister-1-thread-1] INFO
> org.apache.solr.cloud.ShardLeaderElectionContext – I am the new leader:
> http://192.168.150.90:8983/solr/collection1/ shard1
> 21243 [coreZkRegister-1-thread-1] INFO
> org.apache.solr.common.cloud.SolrZkClient – makePath:
> /collections/collection1/leaders/shard1
The "No UpdateLog found - cannot sync" message suggests that either you
have disabled the updateLog feature in solrconfig.xml or that you have
deleted the tlog directory (or possibly the entire data directory).
Hopefully it's the latter, because SolrCloud cannot function properly if
the updateLog is disabled. For data integrity reasons, the updateLog is
always recommended, whether using SolrCloud or not.
Thanks,
Shawn
Re: SolrCloud: Understanding Replication
Posted by Marc Campeau <ca...@gmail.com>.
I notice I have this in the logs when I start SOLR for default example (I
had the same with my own connection)
21242 [coreZkRegister-1-thread-1] INFO
org.apache.solr.cloud.ShardLeaderElectionContext – Enough replicas found
to continue.
21242 [coreZkRegister-1-thread-1] INFO
org.apache.solr.cloud.ShardLeaderElectionContext – I may be the new
leader - try and sync
21242 [coreZkRegister-1-thread-1] INFO org.apache.solr.cloud.SyncStrategy
– Sync replicas to http://192.168.150.90:8983/solr/collection1/
21243 [coreZkRegister-1-thread-1] ERROR org.apache.solr.cloud.SyncStrategy
– No UpdateLog found - cannot sync
21243 [coreZkRegister-1-thread-1] INFO
org.apache.solr.cloud.ShardLeaderElectionContext – We failed sync, but we
have no versions - we can't sync in that case - we were active before, so
become leader anyway
21243 [coreZkRegister-1-thread-1] INFO
org.apache.solr.cloud.ShardLeaderElectionContext – I am the new leader:
http://192.168.150.90:8983/solr/collection1/ shard1
21243 [coreZkRegister-1-thread-1] INFO
org.apache.solr.common.cloud.SolrZkClient – makePath:
/collections/collection1/leaders/shard1
2014-05-30 17:54 GMT-04:00 Erick Erickson <er...@gmail.com>:
> bq: it's not getting replicated
>
> This is definitely not what's expected. Are you by chance _configuring_
> replication while at the same time using SolrCloud? Posting your
> solrconfig.xml would help answer that.
>
> This should be all that's in your solrconfig.xml file when running under
> SolrCloud:
> <requestHandler name="/replication" class="solr.ReplicationHandler" />
>
> No parameters. No polling intervals. No URLs for the master (there isn't
> one anyway). Nothing but what's above.
>
> You also need to insure that your soft commit interval (or hard commit if
> you have openSearcher set to true) is exceeded before you can search
> documents that are being indexed.
>
> If none of that is a problem, I'd really consider just doing the example
> code first. Get _that_ set up without doing _any_ changes and get it
> running, _then_ try your stuff. It really sounds like you've got a
> fundamental mis-configuration and are beating your head against the wall to
> no good purpose.
>
> Best
> Erick
>
>
> On Fri, May 30, 2014 at 1:58 PM, Marc Campeau <ca...@gmail.com> wrote:
>
> > As of now I'm trying reindexing everything. Basically I have now an empty
> > collection and when I add a document it's not getting replicated. Not
> > trying anymore to load my old index anymore.
> >
> > Marc
> >
> >
> > 2014-05-30 16:44 GMT-04:00 Jason Hellman <jhellman@innoventsolutions.com
> >:
> >
> > > Marc,
> > >
> > > Fundamentally it’s a good solution design to always be capable of
> > > reposting (reindexing) your data to Solr. You are demonstrating a
> > classic
> > > use case of this, which is upgrade. Is there a critical reason why you
> > are
> > > avoiding this step?
> > >
> > > Jason
> > >
> > > On May 30, 2014, at 10:38 AM, Marc Campeau <ca...@gmail.com> wrote:
> > >
> > > > 2014-05-30 12:24 GMT-04:00 Erick Erickson <er...@gmail.com>:
> > > >
> > > >> Let's back up a bit here. Why are you copying your indexes around?
> > > >> SolrCloud does all this for you. I suspect you've somehow made a
> > > mis-step.
> > > >>
> > > >
> > > > I started by copying the index around because my 4.5.1 instance is
> not
> > > > setup as Cloud and I wanted to avoid reindexing all my data when
> > > migrating
> > > > to my new 4.8.1 SolrCloud setup. I've now put that aside and I'm just
> > > > trying to get replication happening when I populate an empty
> > collection.
> > > >
> > > >
> > > >> So here's what I'd do by preference; Just set up a new collection
> and
> > > >> re-index. Make sure all of the nodes are up and then just go ahead
> and
> > > >> index to any of them. If you're using SolrJ, CloudSolrServer will
> be a
> > > bit
> > > >> more efficient than sending the docs to random nodes, but that's not
> > > >> necessary.
> > > >>
> > > >
> > > > I've been trying that this morning. Stop the instances, deleted the
> > > > contents of /data on all my 4.8.1 instances then started them
> again...
> > > > they all show up in a 1 shard cluster as 4 replicas and one is the
> > > > leader... they're still shown as down in clusterstate. Then I sent a
> > > > document to be added to one of the nodes specifically. Only that node
> > now
> > > > contains the document. It hasn't been replicated to the other
> > instances.
> > > >
> > > > When I issue queries to the collection for that document through my
> > load
> > > > balancer it works roughtly 1/4 times, in accordance with the fact
> that
> > > it's
> > > > only on the instance where it was added.
> > > >
> > > > Must I use the CLI API for collections to create this new collection
> or
> > > can
> > > > I just do it old style by creating subfolder in /solr directory with
> my
> > > > confs?
> > > >
> > > >
> > > > Here's the log of these operations
> > > >
> > > > LOG of Instance where document was added:
> > > > --------------------
> > > > 2758138 [qtp1781256139-14] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> > > > {add=[Listing_3446279]} 0 271
> > > > 2769177 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
> > > > [mycollection] webapp=/solr path=/admin/ping params={} hits=0
> status=0
> > > > QTime=1
> > > >
> > > > [... More Pings ... ]
> > > >
> > > > 2773138 [commitScheduler-7-thread-1] INFO
> > > > org.apache.solr.update.UpdateHandler – start
> > > >
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
> > > > 2773377 [commitScheduler-7-thread-1] INFO
> > > > org.apache.solr.search.SolrIndexSearcher – Opening
> > > > Searcher@175816a5[mycollection]
> > > > main
> > > > 2773389 [searcherExecutor-5-thread-1] INFO
> > org.apache.solr.core.SolrCore
> > > > – QuerySenderListener sending requests to Searcher@175816a5
> > > [mycollection]
> > > > main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
> > > > 2773389 [searcherExecutor-5-thread-1] INFO
> > org.apache.solr.core.SolrCore
> > > > – QuerySenderListener done.
> > > > 2773390 [searcherExecutor-5-thread-1] INFO
> > org.apache.solr.core.SolrCore
> > > > – [mycollection] Registered new searcher Searcher@175816a5
> > [mycollection]
> > > > main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
> > > > 2773390 [commitScheduler-7-thread-1] INFO
> > > > org.apache.solr.update.UpdateHandler – end_commit_flush
> > > >
> > > > [... More Pings ... ]
> > > >
> > > > 2799792 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > start
> > > >
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > > > 2799883 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > > SolrDeletionPolicy.onCommit: commits: num=2
> > > > commit{dir=NRTCachingDirectory(MMapDirectory@
> > > /opt/solr-4.8.0/example/solr/mycollection/data/index
> > > > lockFactory=NativeFSLockFactory@
> > > /opt/solr-4.8.0/example/solr/mycollection/data/index;
> > > > maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
> > > > commit{dir=NRTCachingDirectory(MMapDirectory@
> > > /opt/solr-4.8.0/example/solr/mycollection/data/index
> > > > lockFactory=NativeFSLockFactory@
> > > /opt/solr-4.8.0/example/solr/mycollection/data/index;
> > > > maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
> > > > 2799884 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > newest
> > > > commit generation = 2
> > > > 2799887 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > > SolrIndexSearcher has not changed - not re-opening:
> > > > org.apache.solr.search.SolrIndexSearcher
> > > > 2799887 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > end_commit_flush
> > > > 2799888 [qtp1781256139-18] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update
> > > >
> > >
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> > > >
> > >
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> > > }
> > > > {commit=} 0 96
> > > > 2800051 [qtp1781256139-14] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> > > {commit=}
> > > > 0 611
> > > > 2802085 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > start
> > > >
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > > > 2802085 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > No
> > > > uncommitted changes. Skipping IW.commit.
> > > > 2802086 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > > SolrIndexSearcher has not changed - not re-opening:
> > > > org.apache.solr.search.SolrIndexSearcher
> > > > 2802086 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > end_commit_flush
> > > > 2802087 [qtp1781256139-18] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update
> > > >
> > >
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> > > >
> > >
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> > > }
> > > > {commit=} 0 2
> > > > 2802126 [qtp1781256139-14] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> > > {commit=}
> > > > 0 44
> > > >
> > > > [... More Pings ... ]
> > > > ----------------------
> > > >
> > > > LOG of 3 other instances :
> > > > ----------------------
> > > > 2773666 [qtp1781256139-11] INFO org.apache.solr.core.SolrCore –
> > > > [mycollection] webapp=/solr path=/admin/ping params={} status=0
> QTime=2
> > > > 2774531 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > start
> > > >
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > > > 2774531 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > No
> > > > uncommitted changes. Skipping IW.commit.
> > > > 2774532 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > > SolrIndexSearcher has not changed - not re-opening:
> > > > org.apache.solr.search.SolrIndexSearcher
> > > > 2774533 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > end_commit_flush
> > > > 2774533 [qtp1781256139-18] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update
> > > >
> > >
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> > > >
> > >
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> > > }
> > > > {commit=} 0 23
> > > > 2776603 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > start
> > > >
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > > > 2776603 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > No
> > > > uncommitted changes. Skipping IW.commit.
> > > > 2776604 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > > SolrIndexSearcher has not changed - not re-opening:
> > > > org.apache.solr.search.SolrIndexSearcher
> > > > 2776604 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > end_commit_flush
> > > > 2776604 [qtp1781256139-18] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update
> > > >
> > >
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> > > >
> > >
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> > > }
> > > > {commit=} 0 2
> > > > 2777428 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
> > > > [mycollection] webapp=/solr path=/admin/ping params={} hits=0
> status=0
> > > > QTime=0
> > > > ----------------------
> > > >
> > > >
> > > >> If that isn't feasible, set up a _one_ node "cloud" and get that
> > running
> > > >> and showing as up with your current index. Then use the Collections
> > > >> "ADDREPLICA" command to bring up the other three nodes. All the
> index
> > > >> synching should then "just happen".
> > > >>
> > > >
> > > > Will try that.
> > > >
> > > >
> > > >> You're also confusing master/slave replication with SolrCloud. In
> the
> > > >> normal state, there is no use of the older-style replication, except
> > > when a
> > > >> downed or new node comes online. If "peer synch" isn't possible,
> then
> > > old
> > > >> style replication happens, but that should be the only time it's
> used.
> > > >> Outside those edge cases, updates go to all nodes when sent to any
> > node,
> > > >> otherwise NRT wouldn't work.
> > > >>
> > > >
> > > > That's what I understood from the research I made. I just think that
> I
> > > > missed some config or something that will make replication happen.
> > > >
> > > > Thanks for your time,
> > > >
> > > > Marc
> > >
> > >
> >
>
Re: SolrCloud: Understanding Replication
Posted by Marc Campeau <ca...@gmail.com>.
Here's my solrconfig.xml:
<?xml version="1.0" encoding="UTF-8" ?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!--
For more details about configurations options that may appear in
this file, see http://wiki.apache.org/solr/SolrConfigXml.
-->
<config>
<luceneMatchVersion>4.4</luceneMatchVersion>
<lib dir="../../../contrib/extraction/lib" regex=".*\.jar" />
<lib dir="../../../dist/" regex="solr-cell-\d.*\.jar" />
<lib dir="../../../contrib/clustering/lib/" regex=".*\.jar" />
<lib dir="../../../dist/" regex="solr-clustering-\d.*\.jar" />
<lib dir="../../../contrib/langid/lib/" regex=".*\.jar" />
<lib dir="../../../dist/" regex="solr-langid-\d.*\.jar" />
<lib dir="../../../contrib/velocity/lib" regex=".*\.jar" />
<lib dir="../../../dist/" regex="solr-velocity-\d.*\.jar" />
<dataDir>${solr.data.dir:}</dataDir>
<directoryFactory name="DirectoryFactory"
class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
<codecFactory class="solr.SchemaCodecFactory"/>
<schemaFactory class="ClassicIndexSchemaFactory"/>
<indexConfig>
<lockType>${solr.lock.type:native}</lockType>
<infoStream>true</infoStream>
</indexConfig>
<jmx />
<!-- The default high-performance update handler -->
<updateHandler class="solr.DirectUpdateHandler2">
<autoCommit>
<maxTime>60000</maxTime> <!-- 1 minutes -->
<maxDocs>30000</maxDocs>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime>
</autoSoftCommit>
</updateHandler>
<query>
<maxBooleanClauses>1024</maxBooleanClauses>
<filterCache class="solr.FastLRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<queryResultCache class="solr.LRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<documentCache class="solr.LRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<enableLazyFieldLoading>true</enableLazyFieldLoading>
<queryResultWindowSize>20</queryResultWindowSize>
<queryResultMaxDocsCached>200</queryResultMaxDocsCached>
<listener event="newSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<!--
<lst><str name="q">solr</str><str name="sort">price
asc</str></lst>
<lst><str name="q">rocks</str><str name="sort">weight
asc</str></lst>
-->
</arr>
</listener>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst>
<str name="q">static firstSearcher warming in solrconfig.xml</str>
</lst>
</arr>
</listener>
<useColdSearcher>false</useColdSearcher>
<maxWarmingSearchers>2</maxWarmingSearchers>
</query>
<requestDispatcher handleSelect="false" >
<requestParsers enableRemoteStreaming="true"
multipartUploadLimitInKB="2048000"
formdataUploadLimitInKB="2048"
addHttpRequestToContext="false"/>
<httpCaching never304="true" />
</requestDispatcher>
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rows">10</int>
<str name="df">text</str>
</lst>
</requestHandler>
<requestHandler name="/query" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="wt">json</str>
<str name="indent">true</str>
<str name="df">text</str>
</lst>
</requestHandler>
<requestHandler name="/get" class="solr.RealTimeGetHandler">
<lst name="defaults">
<str name="omitHeader">true</str>
<str name="wt">json</str>
<str name="indent">true</str>
</lst>
</requestHandler>
<requestHandler name="/browse" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<!-- VelocityResponseWriter settings -->
<str name="wt">velocity</str>
<str name="v.template">browse</str>
<str name="v.layout">layout</str>
<str name="title">Solritas</str>
<!-- Query settings -->
<str name="defType">edismax</str>
<str name="qf">
text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4
title^10.0 description^5.0 keywords^5.0 author^2.0
resourcename^1.0
</str>
<str name="df">text</str>
<str name="mm">100%</str>
<str name="q.alt">*:*</str>
<str name="rows">10</str>
<str name="fl">*,score</str>
<str name="mlt.qf">
text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4
title^10.0 description^5.0 keywords^5.0 author^2.0 resourcename^1.0
</str>
<str
name="mlt.fl">text,features,name,sku,id,manu,cat,title,description,keywords,author,resourcename</str>
<int name="mlt.count">3</int>
<!-- Faceting defaults -->
<str name="facet">on</str>
<str name="facet.field">cat</str>
<str name="facet.field">manu_exact</str>
<str name="facet.field">content_type</str>
<str name="facet.field">author_s</str>
<str name="facet.query">ipod</str>
<str name="facet.query">GB</str>
<str name="facet.mincount">1</str>
<str name="facet.pivot">cat,inStock</str>
<str name="facet.range.other">after</str>
<str name="facet.range">price</str>
<int name="f.price.facet.range.start">0</int>
<int name="f.price.facet.range.end">600</int>
<int name="f.price.facet.range.gap">50</int>
<str name="facet.range">popularity</str>
<int name="f.popularity.facet.range.start">0</int>
<int name="f.popularity.facet.range.end">10</int>
<int name="f.popularity.facet.range.gap">3</int>
<str name="facet.range">manufacturedate_dt</str>
<str
name="f.manufacturedate_dt.facet.range.start">NOW/YEAR-10YEARS</str>
<str name="f.manufacturedate_dt.facet.range.end">NOW</str>
<str name="f.manufacturedate_dt.facet.range.gap">+1YEAR</str>
<str name="f.manufacturedate_dt.facet.range.other">before</str>
<str name="f.manufacturedate_dt.facet.range.other">after</str>
<!-- Highlighting defaults -->
<str name="hl">on</str>
<str name="hl.fl">content features title name</str>
<str name="hl.encoder">html</str>
<str name="hl.simple.pre"><b></str>
<str name="hl.simple.post"></b></str>
<str name="f.title.hl.fragsize">0</str>
<str name="f.title.hl.alternateField">title</str>
<str name="f.name.hl.fragsize">0</str>
<str name="f.name.hl.alternateField">name</str>
<str name="f.content.hl.snippets">3</str>
<str name="f.content.hl.fragsize">200</str>
<str name="f.content.hl.alternateField">content</str>
<str name="f.content.hl.maxAlternateFieldLength">750</str>
<!-- Spell checking defaults -->
<str name="spellcheck">on</str>
<str name="spellcheck.extendedResults">false</str>
<str name="spellcheck.count">5</str>
<str name="spellcheck.alternativeTermCount">2</str>
<str name="spellcheck.maxResultsForSuggest">5</str>
<str name="spellcheck.collate">true</str>
<str name="spellcheck.collateExtendedResults">true</str>
<str name="spellcheck.maxCollationTries">5</str>
<str name="spellcheck.maxCollations">3</str>
</lst>
<!-- append spellchecking to our list of components -->
<arr name="last-components">
<str>spellcheck</str>
</arr>
</requestHandler>
<requestHandler name="/update" class="solr.UpdateRequestHandler">
</requestHandler>
<!-- for back compat with clients using /update/json and /update/csv -->
<requestHandler name="/update/json" class="solr.UpdateRequestHandler">
<lst name="defaults">
<str name="stream.contentType">application/json</str>
</lst>
</requestHandler>
<requestHandler name="/update/csv" class="solr.UpdateRequestHandler">
<lst name="defaults">
<str name="stream.contentType">application/csv</str>
</lst>
</requestHandler>
<requestHandler name="/update/extract"
startup="lazy"
class="solr.extraction.ExtractingRequestHandler" >
<lst name="defaults">
<str name="lowernames">true</str>
<str name="uprefix">ignored_</str>
<!-- capture link hrefs but ignore div attributes -->
<str name="captureAttr">true</str>
<str name="fmap.a">links</str>
<str name="fmap.div">ignored_</str>
</lst>
</requestHandler>
<requestHandler name="/analysis/field"
startup="lazy"
class="solr.FieldAnalysisRequestHandler" />
<requestHandler name="/analysis/document"
class="solr.DocumentAnalysisRequestHandler"
startup="lazy" />
<requestHandler name="/admin/"
class="solr.admin.AdminHandlers" />
<!-- ping/healthcheck -->
<requestHandler name="/admin/ping" class="solr.PingRequestHandler">
<lst name="invariants">
<str name="q">solrpingquery</str>
</lst>
<lst name="defaults">
<str name="echoParams">all</str>
</lst>
<!-- An optional feature of the PingRequestHandler is to configure the
handler with a "healthcheckFile" which can be used to
enable/disable
the PingRequestHandler.
relative paths are resolved against the data dir
-->
<!-- <str name="healthcheckFile">server-enabled.txt</str> -->
</requestHandler>
<!-- Echo the request contents back to the client -->
<requestHandler name="/debug/dump" class="solr.DumpRequestHandler" >
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="echoHandler">true</str>
</lst>
</requestHandler>
<!-- Solr Replication
The SolrReplicationHandler supports replicating indexes from a
"master" used for indexing and "slaves" used for queries.
http://wiki.apache.org/solr/SolrReplication
It is also necessary for SolrCloud to function (in Cloud mode, the
replication handler is used to bulk transfer segments when nodes
are added or need to recover).
https://wiki.apache.org/solr/SolrCloud/
-->
<requestHandler name="/replication" class="solr.ReplicationHandler" >
<!-- IMPORTANT: this is for replication without using Zookeeper
To enable simple master/slave replication, uncomment one of the
sections below, depending on whether this solr instance should be
the "master" or a "slave". If this instance is a "slave" you will
also need to fill in the masterUrl to point to a real machine.
-->
<!--
<lst name="master">
<str name="replicateAfter">startup</str>
<str name="replicateAfter">commit</str>
<str name="replicateAfter">optimize</str>
<str
name="confFiles">solrconfig_slave.xml:solrconfig.xml,currency.xml,mapping-FoldToASCII.txt,protwords.txt,solr-data-config.xml,synonyms.txt,
stopwords.txt,elevate.xml,schema.xml,spellings.txt,mapping-ISOLatin1Accent.txt</str>
<str name="backupAfter">optimize</str>
<str name="commitReserveDuration">00:00:10</str>
</lst>
-->
<!--
<lst name="slave">
<str name="masterUrl">http://192.168.150.90:8983/solr</str>
<str name="pollInterval">00:00:20</str>
</lst>
-->
<!-- int name="maxNumberOfBackups">2</int -->
</requestHandler>
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<str name="queryAnalyzerFieldType">text_general</str>
<!-- Multiple "Spell Checkers" can be declared and used by this
component
-->
<!-- a spellchecker built from a field of the main index -->
<lst name="spellchecker">
<str name="name">default</str>
<str name="field">text</str>
<str name="classname">solr.DirectSolrSpellChecker</str>
<!-- the spellcheck distance measure used, the default is the
internal levenshtein -->
<str name="distanceMeasure">internal</str>
<!-- minimum accuracy needed to be considered a valid spellcheck
suggestion -->
<float name="accuracy">0.5</float>
<!-- the maximum #edits we consider when enumerating terms: can be 1
or 2 -->
<int name="maxEdits">2</int>
<!-- the minimum shared prefix when enumerating terms -->
<int name="minPrefix">1</int>
<!-- maximum number of inspections per result. -->
<int name="maxInspections">5</int>
<!-- minimum length of a query term to be considered for correction
-->
<int name="minQueryLength">4</int>
<!-- maximum threshold of documents a query term can appear to be
considered for correction -->
<float name="maxQueryFrequency">0.01</float>
<!-- uncomment this to require suggestions to occur in 1% of the
documents
<float name="thresholdTokenFrequency">.01</float>
-->
</lst>
<!-- a spellchecker that can break or combine words. See "/spell"
handler below for usage -->
<lst name="spellchecker">
<str name="name">wordbreak</str>
<str name="classname">solr.WordBreakSolrSpellChecker</str>
<str name="field">name</str>
<str name="combineWords">true</str>
<str name="breakWords">true</str>
<int name="maxChanges">10</int>
</lst>
</searchComponent>
<requestHandler name="/spell" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="df">text</str>
<!-- Solr will use suggestions from both the 'default' spellchecker
and from the 'wordbreak' spellchecker and combine them.
collations (re-written queries) can include a combination of
corrections from both spellcheckers -->
<str name="spellcheck.dictionary">default</str>
<str name="spellcheck.dictionary">wordbreak</str>
<str name="spellcheck">on</str>
<str name="spellcheck.extendedResults">true</str>
<str name="spellcheck.count">10</str>
<str name="spellcheck.alternativeTermCount">5</str>
<str name="spellcheck.maxResultsForSuggest">5</str>
<str name="spellcheck.collate">true</str>
<str name="spellcheck.collateExtendedResults">true</str>
<str name="spellcheck.maxCollationTries">10</str>
<str name="spellcheck.maxCollations">5</str>
</lst>
<arr name="last-components">
<str>spellcheck</str>
</arr>
</requestHandler>
<searchComponent name="tvComponent" class="solr.TermVectorComponent"/>
<requestHandler name="/tvrh" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="df">text</str>
<bool name="tv">true</bool>
</lst>
<arr name="last-components">
<str>tvComponent</str>
</arr>
</requestHandler>
<searchComponent name="clustering"
enable="${solr.clustering.enabled:false}"
class="solr.clustering.ClusteringComponent" >
<!-- Declare an engine -->
<lst name="engine">
<!-- The name, only one can be named "default" -->
<str name="name">default</str>
<str
name="carrot.algorithm">org.carrot2.clustering.lingo.LingoClusteringAlgorithm</str>
<str name="LingoClusteringAlgorithm.desiredClusterCountBase">20</str>
<str name="carrot.lexicalResourcesDir">clustering/carrot2</str>
<str name="MultilingualClustering.defaultLanguage">ENGLISH</str>
</lst>
<lst name="engine">
<str name="name">stc</str>
<str
name="carrot.algorithm">org.carrot2.clustering.stc.STCClusteringAlgorithm</str>
</lst>
</searchComponent>
<requestHandler name="/clustering"
startup="lazy"
enable="${solr.clustering.enabled:false}"
class="solr.SearchHandler">
<lst name="defaults">
<bool name="clustering">true</bool>
<str name="clustering.engine">default</str>
<bool name="clustering.results">true</bool>
<!-- The title field -->
<str name="carrot.title">name</str>
<str name="carrot.url">id</str>
<!-- The field to cluster on -->
<str name="carrot.snippet">features</str>
<!-- produce summaries -->
<bool name="carrot.produceSummary">true</bool>
<!-- the maximum number of labels per cluster -->
<!--<int name="carrot.numDescriptions">5</int>-->
<!-- produce sub clusters -->
<bool name="carrot.outputSubClusters">false</bool>
<str name="defType">edismax</str>
<str name="qf">
text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4
</str>
<str name="q.alt">*:*</str>
<str name="rows">10</str>
<str name="fl">*,score</str>
</lst>
<arr name="last-components">
<str>clustering</str>
</arr>
</requestHandler>
<searchComponent name="terms" class="solr.TermsComponent"/>
<!-- A request handler for demonstrating the terms component -->
<requestHandler name="/terms" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<bool name="terms">true</bool>
<bool name="distrib">false</bool>
</lst>
<arr name="components">
<str>terms</str>
</arr>
</requestHandler>
<searchComponent name="elevator" class="solr.QueryElevationComponent" >
<!-- pick a fieldType to analyze queries -->
<str name="queryFieldType">string</str>
<str name="config-file">elevate.xml</str>
</searchComponent>
<requestHandler name="/elevate" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="df">text</str>
</lst>
<arr name="last-components">
<str>elevator</str>
</arr>
</requestHandler>
<searchComponent class="solr.HighlightComponent" name="highlight">
<highlighting>
<!-- Configure the standard fragmenter -->
<!-- This could most likely be commented out in the "default" case -->
<fragmenter name="gap"
default="true"
class="solr.highlight.GapFragmenter">
<lst name="defaults">
<int name="hl.fragsize">100</int>
</lst>
</fragmenter>
<fragmenter name="regex"
class="solr.highlight.RegexFragmenter">
<lst name="defaults">
<!-- slightly smaller fragsizes work better because of slop -->
<int name="hl.fragsize">70</int>
<!-- allow 50% slop on fragment sizes -->
<float name="hl.regex.slop">0.5</float>
<!-- a basic sentence pattern -->
<str name="hl.regex.pattern">[-\w ,/\n\"']{20,200}</str>
</lst>
</fragmenter>
<!-- Configure the standard formatter -->
<formatter name="html"
default="true"
class="solr.highlight.HtmlFormatter">
<lst name="defaults">
<str name="hl.simple.pre"><![CDATA[<em>]]></str>
<str name="hl.simple.post"><![CDATA[</em>]]></str>
</lst>
</formatter>
<!-- Configure the standard encoder -->
<encoder name="html"
class="solr.highlight.HtmlEncoder" />
<!-- Configure the standard fragListBuilder -->
<fragListBuilder name="simple"
class="solr.highlight.SimpleFragListBuilder"/>
<!-- Configure the single fragListBuilder -->
<fragListBuilder name="single"
class="solr.highlight.SingleFragListBuilder"/>
<!-- Configure the weighted fragListBuilder -->
<fragListBuilder name="weighted"
default="true"
class="solr.highlight.WeightedFragListBuilder"/>
<!-- default tag FragmentsBuilder -->
<fragmentsBuilder name="default"
default="true"
class="solr.highlight.ScoreOrderFragmentsBuilder">
</fragmentsBuilder>
<!-- multi-colored tag FragmentsBuilder -->
<fragmentsBuilder name="colored"
class="solr.highlight.ScoreOrderFragmentsBuilder">
<lst name="defaults">
<str name="hl.tag.pre"><![CDATA[
<b style="background:yellow">,<b
style="background:lawgreen">,
<b style="background:aquamarine">,<b
style="background:magenta">,
<b style="background:palegreen">,<b
style="background:coral">,
<b style="background:wheat">,<b style="background:khaki">,
<b style="background:lime">,<b
style="background:deepskyblue">]]></str>
<str name="hl.tag.post"><![CDATA[</b>]]></str>
</lst>
</fragmentsBuilder>
<boundaryScanner name="default"
default="true"
class="solr.highlight.SimpleBoundaryScanner">
<lst name="defaults">
<str name="hl.bs.maxScan">10</str>
<str name="hl.bs.chars">.,!? 	 </str>
</lst>
</boundaryScanner>
<boundaryScanner name="breakIterator"
class="solr.highlight.BreakIteratorBoundaryScanner">
<lst name="defaults">
<!-- type should be one of CHARACTER, WORD(default), LINE and
SENTENCE -->
<str name="hl.bs.type">WORD</str>
<!-- language and country are used when constructing Locale
object. -->
<!-- And the Locale object will be used when getting instance of
BreakIterator -->
<str name="hl.bs.language">en</str>
<str name="hl.bs.country">US</str>
</lst>
</boundaryScanner>
</highlighting>
</searchComponent>
<queryResponseWriter name="json" class="solr.JSONResponseWriter">
<str name="content-type">text/plain; charset=UTF-8</str>
</queryResponseWriter>
<queryResponseWriter name="velocity" class="solr.VelocityResponseWriter"
startup="lazy"/>
<queryResponseWriter name="xslt" class="solr.XSLTResponseWriter">
<int name="xsltCacheLifetimeSeconds">5</int>
</queryResponseWriter>
<!-- Legacy config for the admin interface -->
<admin>
<defaultQuery>*:*</defaultQuery>
</admin>
</config>
2014-05-30 17:54 GMT-04:00 Erick Erickson <er...@gmail.com>:
> bq: it's not getting replicated
>
> This is definitely not what's expected. Are you by chance _configuring_
> replication while at the same time using SolrCloud? Posting your
> solrconfig.xml would help answer that.
>
> This should be all that's in your solrconfig.xml file when running under
> SolrCloud:
> <requestHandler name="/replication" class="solr.ReplicationHandler" />
>
> No parameters. No polling intervals. No URLs for the master (there isn't
> one anyway). Nothing but what's above.
>
> You also need to insure that your soft commit interval (or hard commit if
> you have openSearcher set to true) is exceeded before you can search
> documents that are being indexed.
>
> If none of that is a problem, I'd really consider just doing the example
> code first. Get _that_ set up without doing _any_ changes and get it
> running, _then_ try your stuff. It really sounds like you've got a
> fundamental mis-configuration and are beating your head against the wall to
> no good purpose.
>
> Best
> Erick
>
>
> On Fri, May 30, 2014 at 1:58 PM, Marc Campeau <ca...@gmail.com> wrote:
>
> > As of now I'm trying reindexing everything. Basically I have now an empty
> > collection and when I add a document it's not getting replicated. Not
> > trying anymore to load my old index anymore.
> >
> > Marc
> >
> >
> > 2014-05-30 16:44 GMT-04:00 Jason Hellman <jhellman@innoventsolutions.com
> >:
> >
> > > Marc,
> > >
> > > Fundamentally it’s a good solution design to always be capable of
> > > reposting (reindexing) your data to Solr. You are demonstrating a
> > classic
> > > use case of this, which is upgrade. Is there a critical reason why you
> > are
> > > avoiding this step?
> > >
> > > Jason
> > >
> > > On May 30, 2014, at 10:38 AM, Marc Campeau <ca...@gmail.com> wrote:
> > >
> > > > 2014-05-30 12:24 GMT-04:00 Erick Erickson <er...@gmail.com>:
> > > >
> > > >> Let's back up a bit here. Why are you copying your indexes around?
> > > >> SolrCloud does all this for you. I suspect you've somehow made a
> > > mis-step.
> > > >>
> > > >
> > > > I started by copying the index around because my 4.5.1 instance is
> not
> > > > setup as Cloud and I wanted to avoid reindexing all my data when
> > > migrating
> > > > to my new 4.8.1 SolrCloud setup. I've now put that aside and I'm just
> > > > trying to get replication happening when I populate an empty
> > collection.
> > > >
> > > >
> > > >> So here's what I'd do by preference; Just set up a new collection
> and
> > > >> re-index. Make sure all of the nodes are up and then just go ahead
> and
> > > >> index to any of them. If you're using SolrJ, CloudSolrServer will
> be a
> > > bit
> > > >> more efficient than sending the docs to random nodes, but that's not
> > > >> necessary.
> > > >>
> > > >
> > > > I've been trying that this morning. Stop the instances, deleted the
> > > > contents of /data on all my 4.8.1 instances then started them
> again...
> > > > they all show up in a 1 shard cluster as 4 replicas and one is the
> > > > leader... they're still shown as down in clusterstate. Then I sent a
> > > > document to be added to one of the nodes specifically. Only that node
> > now
> > > > contains the document. It hasn't been replicated to the other
> > instances.
> > > >
> > > > When I issue queries to the collection for that document through my
> > load
> > > > balancer it works roughtly 1/4 times, in accordance with the fact
> that
> > > it's
> > > > only on the instance where it was added.
> > > >
> > > > Must I use the CLI API for collections to create this new collection
> or
> > > can
> > > > I just do it old style by creating subfolder in /solr directory with
> my
> > > > confs?
> > > >
> > > >
> > > > Here's the log of these operations
> > > >
> > > > LOG of Instance where document was added:
> > > > --------------------
> > > > 2758138 [qtp1781256139-14] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> > > > {add=[Listing_3446279]} 0 271
> > > > 2769177 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
> > > > [mycollection] webapp=/solr path=/admin/ping params={} hits=0
> status=0
> > > > QTime=1
> > > >
> > > > [... More Pings ... ]
> > > >
> > > > 2773138 [commitScheduler-7-thread-1] INFO
> > > > org.apache.solr.update.UpdateHandler – start
> > > >
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
> > > > 2773377 [commitScheduler-7-thread-1] INFO
> > > > org.apache.solr.search.SolrIndexSearcher – Opening
> > > > Searcher@175816a5[mycollection]
> > > > main
> > > > 2773389 [searcherExecutor-5-thread-1] INFO
> > org.apache.solr.core.SolrCore
> > > > – QuerySenderListener sending requests to Searcher@175816a5
> > > [mycollection]
> > > > main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
> > > > 2773389 [searcherExecutor-5-thread-1] INFO
> > org.apache.solr.core.SolrCore
> > > > – QuerySenderListener done.
> > > > 2773390 [searcherExecutor-5-thread-1] INFO
> > org.apache.solr.core.SolrCore
> > > > – [mycollection] Registered new searcher Searcher@175816a5
> > [mycollection]
> > > > main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
> > > > 2773390 [commitScheduler-7-thread-1] INFO
> > > > org.apache.solr.update.UpdateHandler – end_commit_flush
> > > >
> > > > [... More Pings ... ]
> > > >
> > > > 2799792 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > start
> > > >
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > > > 2799883 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > > SolrDeletionPolicy.onCommit: commits: num=2
> > > > commit{dir=NRTCachingDirectory(MMapDirectory@
> > > /opt/solr-4.8.0/example/solr/mycollection/data/index
> > > > lockFactory=NativeFSLockFactory@
> > > /opt/solr-4.8.0/example/solr/mycollection/data/index;
> > > > maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
> > > > commit{dir=NRTCachingDirectory(MMapDirectory@
> > > /opt/solr-4.8.0/example/solr/mycollection/data/index
> > > > lockFactory=NativeFSLockFactory@
> > > /opt/solr-4.8.0/example/solr/mycollection/data/index;
> > > > maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
> > > > 2799884 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > newest
> > > > commit generation = 2
> > > > 2799887 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > > SolrIndexSearcher has not changed - not re-opening:
> > > > org.apache.solr.search.SolrIndexSearcher
> > > > 2799887 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > end_commit_flush
> > > > 2799888 [qtp1781256139-18] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update
> > > >
> > >
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> > > >
> > >
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> > > }
> > > > {commit=} 0 96
> > > > 2800051 [qtp1781256139-14] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> > > {commit=}
> > > > 0 611
> > > > 2802085 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > start
> > > >
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > > > 2802085 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > No
> > > > uncommitted changes. Skipping IW.commit.
> > > > 2802086 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > > SolrIndexSearcher has not changed - not re-opening:
> > > > org.apache.solr.search.SolrIndexSearcher
> > > > 2802086 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > end_commit_flush
> > > > 2802087 [qtp1781256139-18] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update
> > > >
> > >
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> > > >
> > >
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> > > }
> > > > {commit=} 0 2
> > > > 2802126 [qtp1781256139-14] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> > > {commit=}
> > > > 0 44
> > > >
> > > > [... More Pings ... ]
> > > > ----------------------
> > > >
> > > > LOG of 3 other instances :
> > > > ----------------------
> > > > 2773666 [qtp1781256139-11] INFO org.apache.solr.core.SolrCore –
> > > > [mycollection] webapp=/solr path=/admin/ping params={} status=0
> QTime=2
> > > > 2774531 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > start
> > > >
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > > > 2774531 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > No
> > > > uncommitted changes. Skipping IW.commit.
> > > > 2774532 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > > SolrIndexSearcher has not changed - not re-opening:
> > > > org.apache.solr.search.SolrIndexSearcher
> > > > 2774533 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > end_commit_flush
> > > > 2774533 [qtp1781256139-18] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update
> > > >
> > >
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> > > >
> > >
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> > > }
> > > > {commit=} 0 23
> > > > 2776603 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > start
> > > >
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > > > 2776603 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > No
> > > > uncommitted changes. Skipping IW.commit.
> > > > 2776604 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > > SolrIndexSearcher has not changed - not re-opening:
> > > > org.apache.solr.search.SolrIndexSearcher
> > > > 2776604 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> > –
> > > > end_commit_flush
> > > > 2776604 [qtp1781256139-18] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update
> > > >
> > >
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> > > >
> > >
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> > > }
> > > > {commit=} 0 2
> > > > 2777428 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
> > > > [mycollection] webapp=/solr path=/admin/ping params={} hits=0
> status=0
> > > > QTime=0
> > > > ----------------------
> > > >
> > > >
> > > >> If that isn't feasible, set up a _one_ node "cloud" and get that
> > running
> > > >> and showing as up with your current index. Then use the Collections
> > > >> "ADDREPLICA" command to bring up the other three nodes. All the
> index
> > > >> synching should then "just happen".
> > > >>
> > > >
> > > > Will try that.
> > > >
> > > >
> > > >> You're also confusing master/slave replication with SolrCloud. In
> the
> > > >> normal state, there is no use of the older-style replication, except
> > > when a
> > > >> downed or new node comes online. If "peer synch" isn't possible,
> then
> > > old
> > > >> style replication happens, but that should be the only time it's
> used.
> > > >> Outside those edge cases, updates go to all nodes when sent to any
> > node,
> > > >> otherwise NRT wouldn't work.
> > > >>
> > > >
> > > > That's what I understood from the research I made. I just think that
> I
> > > > missed some config or something that will make replication happen.
> > > >
> > > > Thanks for your time,
> > > >
> > > > Marc
> > >
> > >
> >
>
Re: SolrCloud: Understanding Replication
Posted by Erick Erickson <er...@gmail.com>.
bq: it's not getting replicated
This is definitely not what's expected. Are you by chance _configuring_
replication while at the same time using SolrCloud? Posting your
solrconfig.xml would help answer that.
This should be all that's in your solrconfig.xml file when running under
SolrCloud:
<requestHandler name="/replication" class="solr.ReplicationHandler" />
No parameters. No polling intervals. No URLs for the master (there isn't
one anyway). Nothing but what's above.
You also need to insure that your soft commit interval (or hard commit if
you have openSearcher set to true) is exceeded before you can search
documents that are being indexed.
If none of that is a problem, I'd really consider just doing the example
code first. Get _that_ set up without doing _any_ changes and get it
running, _then_ try your stuff. It really sounds like you've got a
fundamental mis-configuration and are beating your head against the wall to
no good purpose.
Best
Erick
On Fri, May 30, 2014 at 1:58 PM, Marc Campeau <ca...@gmail.com> wrote:
> As of now I'm trying reindexing everything. Basically I have now an empty
> collection and when I add a document it's not getting replicated. Not
> trying anymore to load my old index anymore.
>
> Marc
>
>
> 2014-05-30 16:44 GMT-04:00 Jason Hellman <jh...@innoventsolutions.com>:
>
> > Marc,
> >
> > Fundamentally it’s a good solution design to always be capable of
> > reposting (reindexing) your data to Solr. You are demonstrating a
> classic
> > use case of this, which is upgrade. Is there a critical reason why you
> are
> > avoiding this step?
> >
> > Jason
> >
> > On May 30, 2014, at 10:38 AM, Marc Campeau <ca...@gmail.com> wrote:
> >
> > > 2014-05-30 12:24 GMT-04:00 Erick Erickson <er...@gmail.com>:
> > >
> > >> Let's back up a bit here. Why are you copying your indexes around?
> > >> SolrCloud does all this for you. I suspect you've somehow made a
> > mis-step.
> > >>
> > >
> > > I started by copying the index around because my 4.5.1 instance is not
> > > setup as Cloud and I wanted to avoid reindexing all my data when
> > migrating
> > > to my new 4.8.1 SolrCloud setup. I've now put that aside and I'm just
> > > trying to get replication happening when I populate an empty
> collection.
> > >
> > >
> > >> So here's what I'd do by preference; Just set up a new collection and
> > >> re-index. Make sure all of the nodes are up and then just go ahead and
> > >> index to any of them. If you're using SolrJ, CloudSolrServer will be a
> > bit
> > >> more efficient than sending the docs to random nodes, but that's not
> > >> necessary.
> > >>
> > >
> > > I've been trying that this morning. Stop the instances, deleted the
> > > contents of /data on all my 4.8.1 instances then started them again...
> > > they all show up in a 1 shard cluster as 4 replicas and one is the
> > > leader... they're still shown as down in clusterstate. Then I sent a
> > > document to be added to one of the nodes specifically. Only that node
> now
> > > contains the document. It hasn't been replicated to the other
> instances.
> > >
> > > When I issue queries to the collection for that document through my
> load
> > > balancer it works roughtly 1/4 times, in accordance with the fact that
> > it's
> > > only on the instance where it was added.
> > >
> > > Must I use the CLI API for collections to create this new collection or
> > can
> > > I just do it old style by creating subfolder in /solr directory with my
> > > confs?
> > >
> > >
> > > Here's the log of these operations
> > >
> > > LOG of Instance where document was added:
> > > --------------------
> > > 2758138 [qtp1781256139-14] INFO
> > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> > > {add=[Listing_3446279]} 0 271
> > > 2769177 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
> > > [mycollection] webapp=/solr path=/admin/ping params={} hits=0 status=0
> > > QTime=1
> > >
> > > [... More Pings ... ]
> > >
> > > 2773138 [commitScheduler-7-thread-1] INFO
> > > org.apache.solr.update.UpdateHandler – start
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
> > > 2773377 [commitScheduler-7-thread-1] INFO
> > > org.apache.solr.search.SolrIndexSearcher – Opening
> > > Searcher@175816a5[mycollection]
> > > main
> > > 2773389 [searcherExecutor-5-thread-1] INFO
> org.apache.solr.core.SolrCore
> > > – QuerySenderListener sending requests to Searcher@175816a5
> > [mycollection]
> > > main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
> > > 2773389 [searcherExecutor-5-thread-1] INFO
> org.apache.solr.core.SolrCore
> > > – QuerySenderListener done.
> > > 2773390 [searcherExecutor-5-thread-1] INFO
> org.apache.solr.core.SolrCore
> > > – [mycollection] Registered new searcher Searcher@175816a5
> [mycollection]
> > > main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
> > > 2773390 [commitScheduler-7-thread-1] INFO
> > > org.apache.solr.update.UpdateHandler – end_commit_flush
> > >
> > > [... More Pings ... ]
> > >
> > > 2799792 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> –
> > > start
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > > 2799883 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > SolrDeletionPolicy.onCommit: commits: num=2
> > > commit{dir=NRTCachingDirectory(MMapDirectory@
> > /opt/solr-4.8.0/example/solr/mycollection/data/index
> > > lockFactory=NativeFSLockFactory@
> > /opt/solr-4.8.0/example/solr/mycollection/data/index;
> > > maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
> > > commit{dir=NRTCachingDirectory(MMapDirectory@
> > /opt/solr-4.8.0/example/solr/mycollection/data/index
> > > lockFactory=NativeFSLockFactory@
> > /opt/solr-4.8.0/example/solr/mycollection/data/index;
> > > maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
> > > 2799884 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> newest
> > > commit generation = 2
> > > 2799887 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > SolrIndexSearcher has not changed - not re-opening:
> > > org.apache.solr.search.SolrIndexSearcher
> > > 2799887 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> –
> > > end_commit_flush
> > > 2799888 [qtp1781256139-18] INFO
> > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > webapp=/solr path=/update
> > >
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> > >
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> > }
> > > {commit=} 0 96
> > > 2800051 [qtp1781256139-14] INFO
> > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> > {commit=}
> > > 0 611
> > > 2802085 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> –
> > > start
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > > 2802085 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> –
> > No
> > > uncommitted changes. Skipping IW.commit.
> > > 2802086 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > SolrIndexSearcher has not changed - not re-opening:
> > > org.apache.solr.search.SolrIndexSearcher
> > > 2802086 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> –
> > > end_commit_flush
> > > 2802087 [qtp1781256139-18] INFO
> > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > webapp=/solr path=/update
> > >
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> > >
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> > }
> > > {commit=} 0 2
> > > 2802126 [qtp1781256139-14] INFO
> > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> > {commit=}
> > > 0 44
> > >
> > > [... More Pings ... ]
> > > ----------------------
> > >
> > > LOG of 3 other instances :
> > > ----------------------
> > > 2773666 [qtp1781256139-11] INFO org.apache.solr.core.SolrCore –
> > > [mycollection] webapp=/solr path=/admin/ping params={} status=0 QTime=2
> > > 2774531 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> –
> > > start
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > > 2774531 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> –
> > No
> > > uncommitted changes. Skipping IW.commit.
> > > 2774532 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > SolrIndexSearcher has not changed - not re-opening:
> > > org.apache.solr.search.SolrIndexSearcher
> > > 2774533 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> –
> > > end_commit_flush
> > > 2774533 [qtp1781256139-18] INFO
> > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > webapp=/solr path=/update
> > >
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> > >
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> > }
> > > {commit=} 0 23
> > > 2776603 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> –
> > > start
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > > 2776603 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> –
> > No
> > > uncommitted changes. Skipping IW.commit.
> > > 2776604 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > > SolrIndexSearcher has not changed - not re-opening:
> > > org.apache.solr.search.SolrIndexSearcher
> > > 2776604 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler
> –
> > > end_commit_flush
> > > 2776604 [qtp1781256139-18] INFO
> > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > webapp=/solr path=/update
> > >
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> > >
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> > }
> > > {commit=} 0 2
> > > 2777428 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
> > > [mycollection] webapp=/solr path=/admin/ping params={} hits=0 status=0
> > > QTime=0
> > > ----------------------
> > >
> > >
> > >> If that isn't feasible, set up a _one_ node "cloud" and get that
> running
> > >> and showing as up with your current index. Then use the Collections
> > >> "ADDREPLICA" command to bring up the other three nodes. All the index
> > >> synching should then "just happen".
> > >>
> > >
> > > Will try that.
> > >
> > >
> > >> You're also confusing master/slave replication with SolrCloud. In the
> > >> normal state, there is no use of the older-style replication, except
> > when a
> > >> downed or new node comes online. If "peer synch" isn't possible, then
> > old
> > >> style replication happens, but that should be the only time it's used.
> > >> Outside those edge cases, updates go to all nodes when sent to any
> node,
> > >> otherwise NRT wouldn't work.
> > >>
> > >
> > > That's what I understood from the research I made. I just think that I
> > > missed some config or something that will make replication happen.
> > >
> > > Thanks for your time,
> > >
> > > Marc
> >
> >
>
Re: SolrCloud: Understanding Replication
Posted by Marc Campeau <ca...@gmail.com>.
As of now I'm trying reindexing everything. Basically I have now an empty
collection and when I add a document it's not getting replicated. Not
trying anymore to load my old index anymore.
Marc
2014-05-30 16:44 GMT-04:00 Jason Hellman <jh...@innoventsolutions.com>:
> Marc,
>
> Fundamentally it’s a good solution design to always be capable of
> reposting (reindexing) your data to Solr. You are demonstrating a classic
> use case of this, which is upgrade. Is there a critical reason why you are
> avoiding this step?
>
> Jason
>
> On May 30, 2014, at 10:38 AM, Marc Campeau <ca...@gmail.com> wrote:
>
> > 2014-05-30 12:24 GMT-04:00 Erick Erickson <er...@gmail.com>:
> >
> >> Let's back up a bit here. Why are you copying your indexes around?
> >> SolrCloud does all this for you. I suspect you've somehow made a
> mis-step.
> >>
> >
> > I started by copying the index around because my 4.5.1 instance is not
> > setup as Cloud and I wanted to avoid reindexing all my data when
> migrating
> > to my new 4.8.1 SolrCloud setup. I've now put that aside and I'm just
> > trying to get replication happening when I populate an empty collection.
> >
> >
> >> So here's what I'd do by preference; Just set up a new collection and
> >> re-index. Make sure all of the nodes are up and then just go ahead and
> >> index to any of them. If you're using SolrJ, CloudSolrServer will be a
> bit
> >> more efficient than sending the docs to random nodes, but that's not
> >> necessary.
> >>
> >
> > I've been trying that this morning. Stop the instances, deleted the
> > contents of /data on all my 4.8.1 instances then started them again...
> > they all show up in a 1 shard cluster as 4 replicas and one is the
> > leader... they're still shown as down in clusterstate. Then I sent a
> > document to be added to one of the nodes specifically. Only that node now
> > contains the document. It hasn't been replicated to the other instances.
> >
> > When I issue queries to the collection for that document through my load
> > balancer it works roughtly 1/4 times, in accordance with the fact that
> it's
> > only on the instance where it was added.
> >
> > Must I use the CLI API for collections to create this new collection or
> can
> > I just do it old style by creating subfolder in /solr directory with my
> > confs?
> >
> >
> > Here's the log of these operations
> >
> > LOG of Instance where document was added:
> > --------------------
> > 2758138 [qtp1781256139-14] INFO
> > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> > {add=[Listing_3446279]} 0 271
> > 2769177 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
> > [mycollection] webapp=/solr path=/admin/ping params={} hits=0 status=0
> > QTime=1
> >
> > [... More Pings ... ]
> >
> > 2773138 [commitScheduler-7-thread-1] INFO
> > org.apache.solr.update.UpdateHandler – start
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
> > 2773377 [commitScheduler-7-thread-1] INFO
> > org.apache.solr.search.SolrIndexSearcher – Opening
> > Searcher@175816a5[mycollection]
> > main
> > 2773389 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
> > – QuerySenderListener sending requests to Searcher@175816a5
> [mycollection]
> > main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
> > 2773389 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
> > – QuerySenderListener done.
> > 2773390 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
> > – [mycollection] Registered new searcher Searcher@175816a5[mycollection]
> > main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
> > 2773390 [commitScheduler-7-thread-1] INFO
> > org.apache.solr.update.UpdateHandler – end_commit_flush
> >
> > [... More Pings ... ]
> >
> > 2799792 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> > start
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > 2799883 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > SolrDeletionPolicy.onCommit: commits: num=2
> > commit{dir=NRTCachingDirectory(MMapDirectory@
> /opt/solr-4.8.0/example/solr/mycollection/data/index
> > lockFactory=NativeFSLockFactory@
> /opt/solr-4.8.0/example/solr/mycollection/data/index;
> > maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
> > commit{dir=NRTCachingDirectory(MMapDirectory@
> /opt/solr-4.8.0/example/solr/mycollection/data/index
> > lockFactory=NativeFSLockFactory@
> /opt/solr-4.8.0/example/solr/mycollection/data/index;
> > maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
> > 2799884 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore – newest
> > commit generation = 2
> > 2799887 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > SolrIndexSearcher has not changed - not re-opening:
> > org.apache.solr.search.SolrIndexSearcher
> > 2799887 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> > end_commit_flush
> > 2799888 [qtp1781256139-18] INFO
> > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > webapp=/solr path=/update
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> }
> > {commit=} 0 96
> > 2800051 [qtp1781256139-14] INFO
> > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> {commit=}
> > 0 611
> > 2802085 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> > start
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > 2802085 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> No
> > uncommitted changes. Skipping IW.commit.
> > 2802086 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > SolrIndexSearcher has not changed - not re-opening:
> > org.apache.solr.search.SolrIndexSearcher
> > 2802086 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> > end_commit_flush
> > 2802087 [qtp1781256139-18] INFO
> > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > webapp=/solr path=/update
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> }
> > {commit=} 0 2
> > 2802126 [qtp1781256139-14] INFO
> > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> {commit=}
> > 0 44
> >
> > [... More Pings ... ]
> > ----------------------
> >
> > LOG of 3 other instances :
> > ----------------------
> > 2773666 [qtp1781256139-11] INFO org.apache.solr.core.SolrCore –
> > [mycollection] webapp=/solr path=/admin/ping params={} status=0 QTime=2
> > 2774531 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> > start
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > 2774531 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> No
> > uncommitted changes. Skipping IW.commit.
> > 2774532 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > SolrIndexSearcher has not changed - not re-opening:
> > org.apache.solr.search.SolrIndexSearcher
> > 2774533 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> > end_commit_flush
> > 2774533 [qtp1781256139-18] INFO
> > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > webapp=/solr path=/update
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> }
> > {commit=} 0 23
> > 2776603 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> > start
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > 2776603 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> No
> > uncommitted changes. Skipping IW.commit.
> > 2776604 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > SolrIndexSearcher has not changed - not re-opening:
> > org.apache.solr.search.SolrIndexSearcher
> > 2776604 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> > end_commit_flush
> > 2776604 [qtp1781256139-18] INFO
> > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > webapp=/solr path=/update
> >
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> >
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false
> }
> > {commit=} 0 2
> > 2777428 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
> > [mycollection] webapp=/solr path=/admin/ping params={} hits=0 status=0
> > QTime=0
> > ----------------------
> >
> >
> >> If that isn't feasible, set up a _one_ node "cloud" and get that running
> >> and showing as up with your current index. Then use the Collections
> >> "ADDREPLICA" command to bring up the other three nodes. All the index
> >> synching should then "just happen".
> >>
> >
> > Will try that.
> >
> >
> >> You're also confusing master/slave replication with SolrCloud. In the
> >> normal state, there is no use of the older-style replication, except
> when a
> >> downed or new node comes online. If "peer synch" isn't possible, then
> old
> >> style replication happens, but that should be the only time it's used.
> >> Outside those edge cases, updates go to all nodes when sent to any node,
> >> otherwise NRT wouldn't work.
> >>
> >
> > That's what I understood from the research I made. I just think that I
> > missed some config or something that will make replication happen.
> >
> > Thanks for your time,
> >
> > Marc
>
>
Re: SolrCloud: Understanding Replication
Posted by Jason Hellman <jh...@innoventsolutions.com>.
Marc,
Fundamentally it’s a good solution design to always be capable of reposting (reindexing) your data to Solr. You are demonstrating a classic use case of this, which is upgrade. Is there a critical reason why you are avoiding this step?
Jason
On May 30, 2014, at 10:38 AM, Marc Campeau <ca...@gmail.com> wrote:
> 2014-05-30 12:24 GMT-04:00 Erick Erickson <er...@gmail.com>:
>
>> Let's back up a bit here. Why are you copying your indexes around?
>> SolrCloud does all this for you. I suspect you've somehow made a mis-step.
>>
>
> I started by copying the index around because my 4.5.1 instance is not
> setup as Cloud and I wanted to avoid reindexing all my data when migrating
> to my new 4.8.1 SolrCloud setup. I've now put that aside and I'm just
> trying to get replication happening when I populate an empty collection.
>
>
>> So here's what I'd do by preference; Just set up a new collection and
>> re-index. Make sure all of the nodes are up and then just go ahead and
>> index to any of them. If you're using SolrJ, CloudSolrServer will be a bit
>> more efficient than sending the docs to random nodes, but that's not
>> necessary.
>>
>
> I've been trying that this morning. Stop the instances, deleted the
> contents of /data on all my 4.8.1 instances then started them again...
> they all show up in a 1 shard cluster as 4 replicas and one is the
> leader... they're still shown as down in clusterstate. Then I sent a
> document to be added to one of the nodes specifically. Only that node now
> contains the document. It hasn't been replicated to the other instances.
>
> When I issue queries to the collection for that document through my load
> balancer it works roughtly 1/4 times, in accordance with the fact that it's
> only on the instance where it was added.
>
> Must I use the CLI API for collections to create this new collection or can
> I just do it old style by creating subfolder in /solr directory with my
> confs?
>
>
> Here's the log of these operations
>
> LOG of Instance where document was added:
> --------------------
> 2758138 [qtp1781256139-14] INFO
> org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> {add=[Listing_3446279]} 0 271
> 2769177 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
> [mycollection] webapp=/solr path=/admin/ping params={} hits=0 status=0
> QTime=1
>
> [... More Pings ... ]
>
> 2773138 [commitScheduler-7-thread-1] INFO
> org.apache.solr.update.UpdateHandler – start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
> 2773377 [commitScheduler-7-thread-1] INFO
> org.apache.solr.search.SolrIndexSearcher – Opening
> Searcher@175816a5[mycollection]
> main
> 2773389 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
> – QuerySenderListener sending requests to Searcher@175816a5[mycollection]
> main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
> 2773389 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
> – QuerySenderListener done.
> 2773390 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
> – [mycollection] Registered new searcher Searcher@175816a5[mycollection]
> main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
> 2773390 [commitScheduler-7-thread-1] INFO
> org.apache.solr.update.UpdateHandler – end_commit_flush
>
> [... More Pings ... ]
>
> 2799792 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> 2799883 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> SolrDeletionPolicy.onCommit: commits: num=2
> commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solr-4.8.0/example/solr/mycollection/data/index
> lockFactory=NativeFSLockFactory@/opt/solr-4.8.0/example/solr/mycollection/data/index;
> maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
> commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solr-4.8.0/example/solr/mycollection/data/index
> lockFactory=NativeFSLockFactory@/opt/solr-4.8.0/example/solr/mycollection/data/index;
> maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
> 2799884 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore – newest
> commit generation = 2
> 2799887 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> SolrIndexSearcher has not changed - not re-opening:
> org.apache.solr.search.SolrIndexSearcher
> 2799887 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> end_commit_flush
> 2799888 [qtp1781256139-18] INFO
> org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> webapp=/solr path=/update
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
> {commit=} 0 96
> 2800051 [qtp1781256139-14] INFO
> org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json} {commit=}
> 0 611
> 2802085 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> 2802085 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler – No
> uncommitted changes. Skipping IW.commit.
> 2802086 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> SolrIndexSearcher has not changed - not re-opening:
> org.apache.solr.search.SolrIndexSearcher
> 2802086 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> end_commit_flush
> 2802087 [qtp1781256139-18] INFO
> org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> webapp=/solr path=/update
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
> {commit=} 0 2
> 2802126 [qtp1781256139-14] INFO
> org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json} {commit=}
> 0 44
>
> [... More Pings ... ]
> ----------------------
>
> LOG of 3 other instances :
> ----------------------
> 2773666 [qtp1781256139-11] INFO org.apache.solr.core.SolrCore –
> [mycollection] webapp=/solr path=/admin/ping params={} status=0 QTime=2
> 2774531 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> 2774531 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler – No
> uncommitted changes. Skipping IW.commit.
> 2774532 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> SolrIndexSearcher has not changed - not re-opening:
> org.apache.solr.search.SolrIndexSearcher
> 2774533 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> end_commit_flush
> 2774533 [qtp1781256139-18] INFO
> org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> webapp=/solr path=/update
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
> {commit=} 0 23
> 2776603 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> 2776603 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler – No
> uncommitted changes. Skipping IW.commit.
> 2776604 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> SolrIndexSearcher has not changed - not re-opening:
> org.apache.solr.search.SolrIndexSearcher
> 2776604 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> end_commit_flush
> 2776604 [qtp1781256139-18] INFO
> org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> webapp=/solr path=/update
> params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
> {commit=} 0 2
> 2777428 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
> [mycollection] webapp=/solr path=/admin/ping params={} hits=0 status=0
> QTime=0
> ----------------------
>
>
>> If that isn't feasible, set up a _one_ node "cloud" and get that running
>> and showing as up with your current index. Then use the Collections
>> "ADDREPLICA" command to bring up the other three nodes. All the index
>> synching should then "just happen".
>>
>
> Will try that.
>
>
>> You're also confusing master/slave replication with SolrCloud. In the
>> normal state, there is no use of the older-style replication, except when a
>> downed or new node comes online. If "peer synch" isn't possible, then old
>> style replication happens, but that should be the only time it's used.
>> Outside those edge cases, updates go to all nodes when sent to any node,
>> otherwise NRT wouldn't work.
>>
>
> That's what I understood from the research I made. I just think that I
> missed some config or something that will make replication happen.
>
> Thanks for your time,
>
> Marc
Re: SolrCloud: Understanding Replication
Posted by Marc Campeau <ca...@gmail.com>.
2014-05-30 12:24 GMT-04:00 Erick Erickson <er...@gmail.com>:
> Let's back up a bit here. Why are you copying your indexes around?
> SolrCloud does all this for you. I suspect you've somehow made a mis-step.
>
I started by copying the index around because my 4.5.1 instance is not
setup as Cloud and I wanted to avoid reindexing all my data when migrating
to my new 4.8.1 SolrCloud setup. I've now put that aside and I'm just
trying to get replication happening when I populate an empty collection.
> So here's what I'd do by preference; Just set up a new collection and
> re-index. Make sure all of the nodes are up and then just go ahead and
> index to any of them. If you're using SolrJ, CloudSolrServer will be a bit
> more efficient than sending the docs to random nodes, but that's not
> necessary.
>
I've been trying that this morning. Stop the instances, deleted the
contents of /data on all my 4.8.1 instances then started them again...
they all show up in a 1 shard cluster as 4 replicas and one is the
leader... they're still shown as down in clusterstate. Then I sent a
document to be added to one of the nodes specifically. Only that node now
contains the document. It hasn't been replicated to the other instances.
When I issue queries to the collection for that document through my load
balancer it works roughtly 1/4 times, in accordance with the fact that it's
only on the instance where it was added.
Must I use the CLI API for collections to create this new collection or can
I just do it old style by creating subfolder in /solr directory with my
confs?
Here's the log of these operations
LOG of Instance where document was added:
--------------------
2758138 [qtp1781256139-14] INFO
org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
{add=[Listing_3446279]} 0 271
2769177 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
[mycollection] webapp=/solr path=/admin/ping params={} hits=0 status=0
QTime=1
[... More Pings ... ]
2773138 [commitScheduler-7-thread-1] INFO
org.apache.solr.update.UpdateHandler – start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
2773377 [commitScheduler-7-thread-1] INFO
org.apache.solr.search.SolrIndexSearcher – Opening
Searcher@175816a5[mycollection]
main
2773389 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
– QuerySenderListener sending requests to Searcher@175816a5[mycollection]
main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
2773389 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
– QuerySenderListener done.
2773390 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
– [mycollection] Registered new searcher Searcher@175816a5[mycollection]
main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
2773390 [commitScheduler-7-thread-1] INFO
org.apache.solr.update.UpdateHandler – end_commit_flush
[... More Pings ... ]
2799792 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2799883 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
SolrDeletionPolicy.onCommit: commits: num=2
commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solr-4.8.0/example/solr/mycollection/data/index
lockFactory=NativeFSLockFactory@/opt/solr-4.8.0/example/solr/mycollection/data/index;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solr-4.8.0/example/solr/mycollection/data/index
lockFactory=NativeFSLockFactory@/opt/solr-4.8.0/example/solr/mycollection/data/index;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
2799884 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore – newest
commit generation = 2
2799887 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
SolrIndexSearcher has not changed - not re-opening:
org.apache.solr.search.SolrIndexSearcher
2799887 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
end_commit_flush
2799888 [qtp1781256139-18] INFO
org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
{commit=} 0 96
2800051 [qtp1781256139-14] INFO
org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json} {commit=}
0 611
2802085 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2802085 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler – No
uncommitted changes. Skipping IW.commit.
2802086 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
SolrIndexSearcher has not changed - not re-opening:
org.apache.solr.search.SolrIndexSearcher
2802086 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
end_commit_flush
2802087 [qtp1781256139-18] INFO
org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
{commit=} 0 2
2802126 [qtp1781256139-14] INFO
org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json} {commit=}
0 44
[... More Pings ... ]
----------------------
LOG of 3 other instances :
----------------------
2773666 [qtp1781256139-11] INFO org.apache.solr.core.SolrCore –
[mycollection] webapp=/solr path=/admin/ping params={} status=0 QTime=2
2774531 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2774531 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler – No
uncommitted changes. Skipping IW.commit.
2774532 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
SolrIndexSearcher has not changed - not re-opening:
org.apache.solr.search.SolrIndexSearcher
2774533 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
end_commit_flush
2774533 [qtp1781256139-18] INFO
org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
{commit=} 0 23
2776603 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2776603 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler – No
uncommitted changes. Skipping IW.commit.
2776604 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
SolrIndexSearcher has not changed - not re-opening:
org.apache.solr.search.SolrIndexSearcher
2776604 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
end_commit_flush
2776604 [qtp1781256139-18] INFO
org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
{commit=} 0 2
2777428 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
[mycollection] webapp=/solr path=/admin/ping params={} hits=0 status=0
QTime=0
----------------------
> If that isn't feasible, set up a _one_ node "cloud" and get that running
> and showing as up with your current index. Then use the Collections
> "ADDREPLICA" command to bring up the other three nodes. All the index
> synching should then "just happen".
>
Will try that.
> You're also confusing master/slave replication with SolrCloud. In the
> normal state, there is no use of the older-style replication, except when a
> downed or new node comes online. If "peer synch" isn't possible, then old
> style replication happens, but that should be the only time it's used.
> Outside those edge cases, updates go to all nodes when sent to any node,
> otherwise NRT wouldn't work.
>
That's what I understood from the research I made. I just think that I
missed some config or something that will make replication happen.
Thanks for your time,
Marc
Re: SolrCloud: Understanding Replication
Posted by Erick Erickson <er...@gmail.com>.
Let's back up a bit here. Why are you copying your indexes around?
SolrCloud does all this for you. I suspect you've somehow made a mis-step.
So here's what I'd do by preference; Just set up a new collection and
re-index. Make sure all of the nodes are up and then just go ahead and
index to any of them. If you're using SolrJ, CloudSolrServer will be a bit
more efficient than sending the docs to random nodes, but that's not
necessary.
If that isn't feasible, set up a _one_ node "cloud" and get that running
and showing as up with your current index. Then use the Collections
"ADDREPLICA" command to bring up the other three nodes. All the index
synching should then "just happen".
You're also confusing master/slave replication with SolrCloud. In the
normal state, there is no use of the older-style replication, except when a
downed or new node comes online. If "peer synch" isn't possible, then old
style replication happens, but that should be the only time it's used.
Outside those edge cases, updates go to all nodes when sent to any node,
otherwise NRT wouldn't work.
HTH,
Erick
On Fri, May 30, 2014 at 7:00 AM, Marc Campeau <ca...@gmail.com> wrote:
> Hi, forgot to mention that I'm migrating the index from Solr 4.5.1 to
> 4.8.1.
>
> Thanks,
>
> Marc Campeau
>
>
> 2014-05-30 9:54 GMT-04:00 Marc Campeau <ca...@gmail.com>:
>
> > Hi,
> >
> > I currently have a standalone SOLR 4.5.1deployment on an EC2 instance
> with
> > a single collection and core containing an index that's roughly 10G. I've
> > used this as a proof of concept, prototype and staging during development
> > phases and I'm about to release to production.
> >
> > For this release, I've setup 4 EC2 instances with 3 servers in Zookeeper
> > ensemble and the 4 servers running SolrCloud. My intention is to have my
> > current collection on a single Shard replicated 4 times based on the high
> > availability requirements. For that, I'm using an ELB as load balancer
> to
> > spread the query load to all 4 instances. For this, I've rsync'ed my
> > current 10G Collection to all 4 SOLR instances in my SolrCloud and
> started
> > them all up. They all come up and do the elections and what nots and all
> > are queryable, which is great. The idea being to load they current index
> as
> > it is and then start updating it instead of reindexing it all from
> scratch.
> >
> > BUT...
> >
> > 1) Using zkCLI, I can see that clusterstate shows all instances as down
> > and this is illustrated on the Solr Admin interface by showing all 4
> > instances using the down color. Is that normal? How can I change that?
> How
> > come if all 4 instances answer queries just fine?
> >
> > 2) It doesn't seem like the instances are replicating... aka if I add a
> > document to the collection it doesn't get replicated to the other
> > instances. Why is that? What should I look for in SOLR logs that would
> tell
> > me that replication is happening? I clearly see in there the
> "/admin/ping"
> > requests made by the load balancer doing health checks and requests made
> to
> > the admin interface but can never find requests made to "/replicate" that
> > would trigger the replication handler.
> >
> > There's obviously something I've done wrong put I can't put my finger on
> > it. I would appreciate your insight on my situation.
> >
> > Thanks,
> >
> > Marc Campeau
> >
> >
>
Re: SolrCloud: Understanding Replication
Posted by Marc Campeau <ca...@gmail.com>.
Hi, forgot to mention that I'm migrating the index from Solr 4.5.1 to 4.8.1.
Thanks,
Marc Campeau
2014-05-30 9:54 GMT-04:00 Marc Campeau <ca...@gmail.com>:
> Hi,
>
> I currently have a standalone SOLR 4.5.1deployment on an EC2 instance with
> a single collection and core containing an index that's roughly 10G. I've
> used this as a proof of concept, prototype and staging during development
> phases and I'm about to release to production.
>
> For this release, I've setup 4 EC2 instances with 3 servers in Zookeeper
> ensemble and the 4 servers running SolrCloud. My intention is to have my
> current collection on a single Shard replicated 4 times based on the high
> availability requirements. For that, I'm using an ELB as load balancer to
> spread the query load to all 4 instances. For this, I've rsync'ed my
> current 10G Collection to all 4 SOLR instances in my SolrCloud and started
> them all up. They all come up and do the elections and what nots and all
> are queryable, which is great. The idea being to load they current index as
> it is and then start updating it instead of reindexing it all from scratch.
>
> BUT...
>
> 1) Using zkCLI, I can see that clusterstate shows all instances as down
> and this is illustrated on the Solr Admin interface by showing all 4
> instances using the down color. Is that normal? How can I change that? How
> come if all 4 instances answer queries just fine?
>
> 2) It doesn't seem like the instances are replicating... aka if I add a
> document to the collection it doesn't get replicated to the other
> instances. Why is that? What should I look for in SOLR logs that would tell
> me that replication is happening? I clearly see in there the "/admin/ping"
> requests made by the load balancer doing health checks and requests made to
> the admin interface but can never find requests made to "/replicate" that
> would trigger the replication handler.
>
> There's obviously something I've done wrong put I can't put my finger on
> it. I would appreciate your insight on my situation.
>
> Thanks,
>
> Marc Campeau
>
>