You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mugdha Varadkar <mu...@gmail.com> on 2018/04/03 09:22:49 UTC
Re: Trying to Restore older indexes in Solr7.2.1
Hi Shawn Heisey,
Thank you for the reply given here
<https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201803.mbox/%3C4e2ef3ec-65b3-30e0-1274-6edf43bc09ce%40elyograg.org%3E>
.
Please find below answers to your questions,
is the collection using the compositeId router?
Yes collection used of both the versions are using compositeId router, PFA
screenshot of the same.
>
> You would need to look at the information in zookeeper for both versions
Any specific information I can check / look into: shall I check for
uploaded configs for the collection or any specific set of properties in
solrconfig.xml?
From my earlier mail, trying to re-phrase my point #6 - The new documents
available after new collection creation got lost after restoring older
indexes (in step #5).
The difference between schema of solr-5.5.5 and schema that is there for
solr-7.2.1 is that, we are adding docValues on required fields to improvise
indexing of required fields.
Hence before upgrading from solr-5.5.5, I took a backup of documents using
restore-api and then upgraded solr from 5.5.5 to 7.2.1 and tried to restore
the backed-up data by converting the documents to the newer format using
the index-upgrader tool.
And, issue I got after restoring the documents of older version was : all
the documents that were there in collection that was created for solr-7.2.1
were not available at all.
Thanks,
Mugdha Varadkar
On Wed, Mar 28, 2018 at 12:46 PM, Mugdha Varadkar <
mugdha.varadkar010@gmail.com> wrote:
> Hi,
>
> I am trying to restore Solr 5.5.5 indexes into Solr 7.2.1
>
> Performed below steps:
>
> 1. Upgraded the indexes to Solr 6.6.2 indexes using IndexUpgraded Tool
> <https://lucene.apache.org/solr/guide/7_0/indexupgrader-tool.html#indexupgrader-tool>
> Command used : java -cp server/solr-webapp/webapp/WEB-
> INF/lib/lucene-core-6.6.2.jar:server/solr-webapp/webapp/WEB-
> INF/lib/lucene-backward-codecs-6.6.2.jar org.apache.lucene.index.IndexUpgrader
> -verbose /usr/local/solr_5_5_data/index/
> 2. Deleted the old collection used previously, as there is a change in
> schema file.
> 3. Created new collection for new schema. Keeping the replication and
> shards same as old collection.
> 4. Now there is data going into new collection.
> 5. Then I restore the upgraded indexes using replication API.(As the
> backup was taken using that.)
> 6. The indexes got restored but the new data created after new
> collection creation was not available.
>
> Can someone suggest what I missed during upgrading the indexes ?
>
> Thanks,
> Mugdha Varadkar
>
Re: Trying to Restore older indexes in Solr7.2.1
Posted by Shawn Heisey <ap...@elyograg.org>.
On 4/4/2018 4:31 AM, Mugdha Varadkar wrote:
> The hash ranges are available in each collections state.json file. Here is
> the data for the collections:
I thought you said these indexes were sharded? That only shows one
shard. It has the full hash range. If they're all like that, then
don't worry about hash ranges matching up.
> Most schema changes require rebuilding the index from scratch.
>
> So the solution so far is to export the documents of older collection after
> upgrading to solr 7.2.1 import them on to new collection.
If you're using an export/import process, you probably want to perform
the export on the old version, not the new version.
> Are you getting any errors when you query the index? Look in solr.log.
>
> Not yet.
Exactly what are you doing to export your data? What does the export
look like? How are you importing it? Did you commit the changes after
importing?
Thanks,
Shawn
Re: Trying to Restore older indexes in Solr7.2.1
Posted by Mugdha Varadkar <mu...@gmail.com>.
Hi Shawn,
The hash ranges are available in each collections state.json file. Here is
the data for the collections:
*Solr 5.5.5 collection*
{"ranger_audits":{
"replicationFactor":"1",
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false",
"shards":{"shard1":{
"range":"80000000-7fffffff",
"state":"active",
"replicas":{"core_node1":{
"core":"ranger_audits_shard1_replica1",
"base_url":"http://127.0.0.1:8886/solr",
"node_name":"127.0.0.1:8886_solr",
"state":"active",
"leader":"true"}}}}}}
*Solr 7.2.1 collection*
{"ranger_audits":{
"pullReplicas":"0",
"replicationFactor":"1",
"shards":{"shard1":{
"range":"80000000-7fffffff",
"state":"active",
"replicas":{"core_node2":{
"core":"ranger_audits_shard1_replica_n1",
"base_url":"http://localhost:8886/solr",
"node_name":"localhost:8886_solr",
"state":"active",
"type":"NRT",
"leader":"true"}}}},
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false",
"nrtReplicas":"1",
"tlogReplicas":"0"}}
Most schema changes require rebuilding the index from scratch.
So the solution so far is to export the documents of older collection after
upgrading to solr 7.2.1 import them on to new collection.
Are you getting any errors when you query the index? Look in solr.log.
Not yet.
Thanks,
Mugdha Varadkar
On Tue, Apr 3, 2018 at 5:58 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 4/3/2018 3:22 AM, Mugdha Varadkar wrote:
>
>>
>> is the collection using the compositeId router?
>>
>> Yes collection used of both the versions are using compositeId router,
>> PFA screenshot of the same.
>>
>
> If you attached a screenshot, it was lost. The mailing list does not
> allow most attachments through.
>
> You would need to look at the information in zookeeper for both
>> versions
>>
>> Any specific information I can check / look into: shall I check for
>> uploaded configs for the collection or any specific set of properties in
>> solrconfig.xml?
>>
>
> It's not in solrconfig.xml. I think you'll find the hash ranges for your
> shards in the '/collections/NAME/state.json' file for the collection, in
> zookeeper. I'm not 100% sure that this is the correct location, but I
> *think* it is.
>
> The difference between schema of solr-5.5.5 and schema that is there for
>> solr-7.2.1 is that, we are adding docValues on required fields to improvise
>> indexing of required fields.
>> Hence before upgrading from solr-5.5.5, I took a backup of documents
>> using restore-api and then upgraded solr from 5.5.5 to 7.2.1 and tried to
>> restore the backed-up data by converting the documents to the newer format
>> using the index-upgrader tool.
>>
>
> IndexUpgrader is a Lucene tool. The schema is a Solr invention.
> IndexUpgrader does not know anything about the schema. It only knows about
> what's actually in the index.
>
> If you try to use a different schema to access your index than the schema
> it was created with, you run the risk that you won't be able to access your
> data. Most schema changes require rebuilding the index from scratch.
> Adding docValues is a change that requires this. Running indexUpgrader is
> *NOT* a reindex.
>
> And, issue I got after restoring the documents of older version was : all
>> the documents that were there in collection that was created for solr-7.2.1
>> were not available at all.
>>
>
> Are you getting any errors when you query the index? Look in solr.log.
>
> Thanks,
> Shawn
>
>
Re: Trying to Restore older indexes in Solr7.2.1
Posted by Shawn Heisey <ap...@elyograg.org>.
On 4/3/2018 3:22 AM, Mugdha Varadkar wrote:
>
> is the collection using the compositeId router?
>
> Yes collection used of both the versions are using compositeId router,
> PFA screenshot of the same.
If you attached a screenshot, it was lost. The mailing list does not
allow most attachments through.
> You would need to look at the information in zookeeper for both
> versions
>
> Any specific information I can check / look into: shall I check for
> uploaded configs for the collection or any specific set of properties
> in solrconfig.xml?
It's not in solrconfig.xml. I think you'll find the hash ranges for
your shards in the '/collections/NAME/state.json' file for the
collection, in zookeeper. I'm not 100% sure that this is the correct
location, but I *think* it is.
> The difference between schema of solr-5.5.5 and schema that is there
> for solr-7.2.1 is that, we are adding docValues on required fields to
> improvise indexing of required fields.
> Hence before upgrading from solr-5.5.5, I took a backup of documents
> using restore-api and then upgraded solr from 5.5.5 to 7.2.1 and tried
> to restore the backed-up data by converting the documents to the newer
> format using the index-upgrader tool.
IndexUpgrader is a Lucene tool. The schema is a Solr invention.
IndexUpgrader does not know anything about the schema. It only knows
about what's actually in the index.
If you try to use a different schema to access your index than the
schema it was created with, you run the risk that you won't be able to
access your data. Most schema changes require rebuilding the index from
scratch. Adding docValues is a change that requires this. Running
indexUpgrader is *NOT* a reindex.
> And, issue I got after restoring the documents of older version was :
> all the documents that were there in collection that was created for
> solr-7.2.1 were not available at all.
Are you getting any errors when you query the index? Look in solr.log.
Thanks,
Shawn