You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Kamal Kishore Aggarwal <kk...@gmail.com> on 2019/08/26 10:37:07 UTC
Re: Recipe for moving to solr cloud without reindexing
Hi Erick,
Is there any workaround for this?
My requirement is to move solr data from solr 6 to solr 7.
I have thought to use /replication handler to do so. But, as per the case,
the solr 6 collection has 1 shard & 1 replica and we wish to have 2 shards
and 1 replica on solr 7.
If I enable replication on collection having 2 shards, the same data from
solr 6 is copied to both the shards. i.e data becomes double.
However, if I enable replication on collection with 1 shard and later try
to SPLITSHARD, it failed with below error even after using Index Upgrade
Tool/Optimization:
*Caused by: java.lang.IllegalArgumentException: Cannot merge a segment that
has been created with major version 6 into this index which has been
created by major version 7*
Regards
Kamal
On Wed, Aug 8, 2018 at 11:58 PM Erick Erickson <er...@gmail.com>
wrote:
> See: https://issues.apache.org/jira/browse/SOLR-12646
>
> On Wed, Aug 8, 2018 at 11:24 AM, Bjarke Buur Mortensen
> <mo...@eluence.com> wrote:
> > OK, thanks.
> >
> > As long as it's my dev box, reindexing is fine.
> > I just hope that my assumption holds, that our prod solr is 7x segments
> > only.
> >
> > Thanks again,
> > Bjarke
> >
> > 2018-08-08 20:03 GMT+02:00 Erick Erickson <er...@gmail.com>:
> >
> >> Bjarke:
> >>
> >> Using SPLITSHARD on an index with 6x segments just seems to not work,
> >> even outside the standalone-> cloud issue. I'll raise a JIRA.
> >> Meanwhile I think you'll have to re-index I'm afraid.
> >>
> >> Thanks for raising the issue.
> >>
> >> Erick
> >>
> >> On Wed, Aug 8, 2018 at 6:34 AM, Bjarke Buur Mortensen
> >> <mo...@eluence.com> wrote:
> >> > Erick,
> >> >
> >> > thanks, that is of course something I left out of the original
> question.
> >> > Our Solr is 7.1, so that should not present a problem (crossing
> fingers).
> >> >
> >> > However, on my dev box I'm trying out the steps, and here I have some
> >> > segments created with version 6 of Solr.
> >> >
> >> > After having copied data from my non-cloud solr into my
> >> > single-shard-single-replica collection and verified that Solr Cloud
> works
> >> > with this collection, I then submit the splitshard command
> >> >
> >> > http://172.17.0.4:8984
> >> > /solr/admin/collections?action=SPLITSHARD&collection=
> >> procurement&shard=shard1
> >> >
> >> > However, this gives me the error:
> >> > org.apache.solr.client.solrj.impl.HttpSolrClient$
> >> RemoteSolrException:Error
> >> > from server at http://172.17.0.4:8984/solr:
> >> > java.lang.IllegalArgumentException: Cannot merge a segment that has
> been
> >> > created with major version 6 into this index which has been created by
> >> > major version 7"}
> >> >
> >> > I have tried running both optimize and IndexUpgrader on the index
> before
> >> > shard splitting, but the same error still occurs.
> >> >
> >> > Any ideas as to why this happens?
> >> >
> >> > Below is an output from running IndexUpgrader, which I cannot
> decipher.
> >> > It both states that "All segments upgraded to version 7.1.0" and ''all
> >> > running merges have aborted" ¯\_(ツ)_/¯
> >> >
> >> > Thanks a lot,
> >> > Bjarke
> >> >
> >> >
> >> > ======================
> >> > java -cp
> >> > /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/lucene-
> >> backward-codecs-7.1.0.jar:/opt/solr/server/solr-webapp/
> >> webapp/WEB-INF/lib/lucene-core-7.1.0.jar
> >> > org.apache.lucene.index.IndexUpgrader -delete-prior-commits -verbose
> >> > /var/solr/cloud/procurement_shard1_replica_n1/data/index
> >> > IFD 0 [2018-08-08T13:00:18.244Z; main]: init: current segments file is
> >> > "segments_4vs";
> >> > deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPoli
> >> cy@721e0f4f
> >> > IFD 0 [2018-08-08T13:00:18.266Z; main]: init: load commit
> "segments_4vs"
> >> > IFD 0 [2018-08-08T13:00:18.270Z; main]: now checkpoint
> >> > "_bhg(7.1.0):C108396" [1 segments ; isCommit = false]
> >> > IFD 0 [2018-08-08T13:00:18.270Z; main]: 0 msec to checkpoint
> >> > IW 0 [2018-08-08T13:00:18.270Z; main]: init: create=false
> >> > IW 0 [2018-08-08T13:00:18.273Z; main]:
> >> > dir=MMapDirectory@/var/solr/cloud/procurement_shard1_
> >> replica_n1/data/index
> >> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@6debcae2
> >> > index=_bhg(7.1.0):C108396
> >> > version=7.1.0
> >> > analyzer=null
> >> > ramBufferSizeMB=16.0
> >> > maxBufferedDocs=-1
> >> > mergedSegmentWarmer=null
> >> > delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
> >> > commit=null
> >> > openMode=CREATE_OR_APPEND
> >> > similarity=org.apache.lucene.search.similarities.BM25Similarity
> >> > mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=-1,
> >> > maxMergeCount=-1, ioThrottle=true
> >> > codec=Lucene70
> >> > infoStream=org.apache.lucene.util.PrintStreamInfoStream
> >> > mergePolicy=UpgradeIndexMergePolicy([TieredMergePolicy:
> >> maxMergeAtOnce=10,
> >> > maxMergeAtOnceExplicit=30, maxMergedSegmentMB=5120.0,
> floorSegmentMB=2.0,
> >> > forceMergeDeletesPctAllowed=10.0, segmentsPerTier=10.0,
> >> > maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1)
> >> >
> indexerThreadPool=org.apache.lucene.index.DocumentsWriterPerThreadPool@
> >> 5ba23b66
> >> > readerPooling=true
> >> > perThreadHardLimitMB=1945
> >> > useCompoundFile=true
> >> > commitOnClose=true
> >> > indexSort=null
> >> > writer=org.apache.lucene.index.IndexWriter@2ff4f00f
> >> >
> >> > IW 0 [2018-08-08T13:00:18.273Z; main]: MMapDirectory.UNMAP_SUPPORTED=
> >> true
> >> > IndexUpgrader 0 [2018-08-08T13:00:18.274Z; main]: Upgrading all
> pre-7.1.0
> >> > segments of index directory
> >> > 'MMapDirectory@
> /var/solr/cloud/procurement_shard1_replica_n1/data/index
> >> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@6debcae2' to
> >> > version 7.1.0...
> >> > IW 0 [2018-08-08T13:00:18.274Z; main]: forceMerge: index now
> >> > _bhg(7.1.0):C108396
> >> > IW 0 [2018-08-08T13:00:18.274Z; main]: now flush at forceMerge
> >> > IW 0 [2018-08-08T13:00:18.274Z; main]: start flush:
> >> applyAllDeletes=true
> >> > IW 0 [2018-08-08T13:00:18.274Z; main]: index before flush
> >> > _bhg(7.1.0):C108396
> >> > DW 0 [2018-08-08T13:00:18.274Z; main]: startFullFlush
> >> > DW 0 [2018-08-08T13:00:18.275Z; main]: main finishFullFlush
> success=true
> >> > IW 0 [2018-08-08T13:00:18.275Z; main]: now apply all deletes for all
> >> > segments buffered updates bytesUsed=0 reader pool bytesUsed=0
> >> > BD 0 [2018-08-08T13:00:18.275Z; main]: waitApply: no deletes to apply
> >> > UPGMP 0 [2018-08-08T13:00:18.276Z; main]: findForcedMerges:
> >> > segmentsToUpgrade={}
> >> > MS 0 [2018-08-08T13:00:18.282Z; main]: initDynamicDefaults spins=true
> >> > maxThreadCount=1 maxMergeCount=6
> >> > MS 0 [2018-08-08T13:00:18.282Z; main]: now merge
> >> > MS 0 [2018-08-08T13:00:18.282Z; main]: index: _bhg(7.1.0):C108396
> >> > MS 0 [2018-08-08T13:00:18.282Z; main]: no more merges pending; now
> >> return
> >> > IndexUpgrader 0 [2018-08-08T13:00:18.282Z; main]: All segments
> upgraded
> >> to
> >> > version 7.1.0
> >> > IndexUpgrader 0 [2018-08-08T13:00:18.283Z; main]: Enforcing commit to
> >> > rewrite all index metadata...
> >> > IW 0 [2018-08-08T13:00:18.283Z; main]: commit: start
> >> > IW 0 [2018-08-08T13:00:18.283Z; main]: commit: enter lock
> >> > IW 0 [2018-08-08T13:00:18.283Z; main]: commit: now prepare
> >> > IW 0 [2018-08-08T13:00:18.283Z; main]: prepareCommit: flush
> >> > IW 0 [2018-08-08T13:00:18.283Z; main]: index before flush
> >> > _bhg(7.1.0):C108396
> >> > DW 0 [2018-08-08T13:00:18.283Z; main]: startFullFlush
> >> > IW 0 [2018-08-08T13:00:18.283Z; main]: now apply all deletes for all
> >> > segments buffered updates bytesUsed=0 reader pool bytesUsed=0
> >> > BD 0 [2018-08-08T13:00:18.283Z; main]: waitApply: no deletes to apply
> >> > DW 0 [2018-08-08T13:00:18.284Z; main]: main finishFullFlush
> success=true
> >> > IW 0 [2018-08-08T13:00:18.284Z; main]: startCommit(): start
> >> > IW 0 [2018-08-08T13:00:18.284Z; main]: startCommit
> >> > index=_bhg(7.1.0):C108396 changeCount=2
> >> > IW 0 [2018-08-08T13:00:18.293Z; main]: startCommit: wrote pending
> >> segments
> >> > file "pending_segments_4vt"
> >> > IW 0 [2018-08-08T13:00:18.295Z; main]: done all syncs:
> >> > [_bhg_Lucene50_0.tip, _bhg.fdx, _bhg.fnm, _bhg.nvm, _bhg.fdt, _bhg.si
> ,
> >> > _bhg_Lucene50_0.pos, _bhg.nvd, _bhg_Lucene50_0.doc,
> _bhg_Lucene50_0.tim]
> >> > IW 0 [2018-08-08T13:00:18.295Z; main]: commit: pendingCommit != null
> >> > IW 0 [2018-08-08T13:00:18.298Z; main]: commit: done writing segments
> file
> >> > "segments_4vt"
> >> > IFD 0 [2018-08-08T13:00:18.298Z; main]: now checkpoint
> >> > "_bhg(7.1.0):C108396" [1 segments ; isCommit = true]
> >> > IFD 0 [2018-08-08T13:00:18.298Z; main]: deleteCommits: now decRef
> commit
> >> > "segments_4vs"
> >> > IFD 0 [2018-08-08T13:00:18.298Z; main]: delete [segments_4vs]
> >> > IFD 0 [2018-08-08T13:00:18.299Z; main]: 0 msec to checkpoint
> >> > IW 0 [2018-08-08T13:00:18.319Z; main]: commit: took 16.0 msec
> >> > IW 0 [2018-08-08T13:00:18.319Z; main]: commit: done
> >> > IndexUpgrader 0 [2018-08-08T13:00:18.319Z; main]: Committed upgraded
> >> > metadata to index.
> >> > IW 0 [2018-08-08T13:00:18.319Z; main]: now flush at close
> >> > IW 0 [2018-08-08T13:00:18.319Z; main]: start flush:
> >> applyAllDeletes=true
> >> > IW 0 [2018-08-08T13:00:18.319Z; main]: index before flush
> >> > _bhg(7.1.0):C108396
> >> > DW 0 [2018-08-08T13:00:18.319Z; main]: startFullFlush
> >> > DW 0 [2018-08-08T13:00:18.320Z; main]: main finishFullFlush
> success=true
> >> > IW 0 [2018-08-08T13:00:18.320Z; main]: now apply all deletes for all
> >> > segments buffered updates bytesUsed=0 reader pool bytesUsed=0
> >> > BD 0 [2018-08-08T13:00:18.320Z; main]: waitApply: no deletes to apply
> >> > MS 0 [2018-08-08T13:00:18.320Z; main]: updateMergeThreads
> ioThrottle=true
> >> > targetMBPerSec=10240.0 MB/sec
> >> > MS 0 [2018-08-08T13:00:18.320Z; main]: now merge
> >> > MS 0 [2018-08-08T13:00:18.321Z; main]: index: _bhg(7.1.0):C108396
> >> > MS 0 [2018-08-08T13:00:18.321Z; main]: no more merges pending; now
> >> return
> >> > IW 0 [2018-08-08T13:00:18.321Z; main]: waitForMerges
> >> > IW 0 [2018-08-08T13:00:18.321Z; main]: waitForMerges done
> >> > IW 0 [2018-08-08T13:00:18.321Z; main]: commit: start
> >> > IW 0 [2018-08-08T13:00:18.321Z; main]: commit: enter lock
> >> > IW 0 [2018-08-08T13:00:18.321Z; main]: commit: now prepare
> >> > IW 0 [2018-08-08T13:00:18.321Z; main]: prepareCommit: flush
> >> > IW 0 [2018-08-08T13:00:18.321Z; main]: index before flush
> >> > _bhg(7.1.0):C108396
> >> > DW 0 [2018-08-08T13:00:18.321Z; main]: startFullFlush
> >> > IW 0 [2018-08-08T13:00:18.321Z; main]: now apply all deletes for all
> >> > segments buffered updates bytesUsed=0 reader pool bytesUsed=0
> >> > BD 0 [2018-08-08T13:00:18.322Z; main]: waitApply: no deletes to apply
> >> > DW 0 [2018-08-08T13:00:18.322Z; main]: main finishFullFlush
> success=true
> >> > IW 0 [2018-08-08T13:00:18.322Z; main]: startCommit(): start
> >> > IW 0 [2018-08-08T13:00:18.322Z; main]: skip startCommit(): no
> changes
> >> > pending
> >> > IW 0 [2018-08-08T13:00:18.322Z; main]: commit: pendingCommit == null;
> >> skip
> >> > IW 0 [2018-08-08T13:00:18.322Z; main]: commit: took 0.9 msec
> >> > IW 0 [2018-08-08T13:00:18.322Z; main]: commit: done
> >> > IW 0 [2018-08-08T13:00:18.322Z; main]: rollback
> >> > IW 0 [2018-08-08T13:00:18.322Z; main]: all running merges have aborted
> >> > IW 0 [2018-08-08T13:00:18.323Z; main]: rollback: done finish merges
> >> > DW 0 [2018-08-08T13:00:18.323Z; main]: abort
> >> > DW 0 [2018-08-08T13:00:18.323Z; main]: done abort success=true
> >> > IW 0 [2018-08-08T13:00:18.323Z; main]: rollback:
> >> infos=_bhg(7.1.0):C108396
> >> > IFD 0 [2018-08-08T13:00:18.323Z; main]: now checkpoint
> >> > "_bhg(7.1.0):C108396" [1 segments ; isCommit = false]
> >> > IFD 0 [2018-08-08T13:00:18.323Z; main]: 0 msec to checkpoint
> >> >
> >> > 2018-08-07 16:38 GMT+02:00 Erick Erickson <er...@gmail.com>:
> >> >
> >> >> Bjarke:
> >> >>
> >> >> One thing, what version of Solr are you moving _from_ and _to_?
> >> >> Solr/Lucene only guarantee one major backward revision so you can
> copy
> >> >> an index created with Solr 6 to another Solr 6 or Solr 7, but you
> >> >> couldn't copy an index created with Solr 5 to Solr 7...
> >> >>
> >> >> Also note that shard splitting is a very expensive operation, so be
> >> >> patient....
> >> >>
> >> >> Best,
> >> >> Erick
> >> >>
> >> >> On Tue, Aug 7, 2018 at 6:17 AM, Rahul Singh
> >> >> <ra...@gmail.com> wrote:
> >> >> > Bjarke,
> >> >> >
> >> >> > I am imagining that at some point you may need to shard that data
> if
> >> it
> >> >> grows. Or do you imagine this data to remain stagnant?
> >> >> >
> >> >> > Generally you want to add solrcloud to do two things : 1. Increase
> >> >> availability with replicas 2. Increase available data via shards 3.
> >> >> Increase fault tolerance with leader and replicas being spread around
> >> the
> >> >> cluster.
> >> >> >
> >> >> > You would be bypassing general High availability / distributed
> >> computing
> >> >> processes by trying to not reindex.
> >> >> >
> >> >> > Rahul
> >> >> > On Aug 7, 2018, 7:06 AM -0400, Bjarke Buur Mortensen <
> >> >> mortensen@eluence.com>, wrote:
> >> >> >> Hi List,
> >> >> >>
> >> >> >> is there a cookbook recipe for moving an existing solr core to a
> solr
> >> >> cloud
> >> >> >> collection.
> >> >> >>
> >> >> >> We currently have a single machine with a large core (~150gb),
> and we
> >> >> would
> >> >> >> like to move to solr cloud.
> >> >> >>
> >> >> >> I haven't been able to find anything that reuses an existing
> index,
> >> so
> >> >> any
> >> >> >> pointers much appreciated.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> Bjarke
> >> >>
> >>
>