You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Nimesh Aghera <na...@asite.com> on 2017/12/12 09:15:00 UTC

Implicit routing changes to Composite while re-deploy changes and make Upconfig command

Hello,

I have implemented implicit routing with below configuration.
Created one default collection manually 'AMS_Config' which contains configurations files  schema,solrconfig etc.

Using 'AMS_Config' I have created 2 collections model,workset respectively with below command which created 2 shard for each collection containing 2 node for each shard where nodes are each solr instance where collection created.

Command: /admin/collections?action=CREATE&name=model&replicationFactor=2&router.name=implicit&router.field=dr&shards=shard1,shard2&maxShardsPerNode=2&collection.configName=AMS_Config

Collection Detail:
Model = Shard1,Shard2
Shard1 = node1,node2[leader]
Shard2 =node1[leader],node2

Configuration in admin UI on solr for model collection:
Shard count:2
configName:AMS_Config
replicationFactor:2
maxShardsPerNode:2
router:implicit
autoAddReplicas:false

after this I have index document to particular shard using set value (router.field) dr = shard1

Issue:  After indexing the document made changes in schema files and redeploy using below command  for set latest configs
zkcli.bat -cmd upconfig -confdir ../../solr/AMS_Config/conf -confname AMS_Config -z <zk Ips>

It will changes the router value implicit to compositeId and now my document are index across all shard so why this should happens.How to avoid this.

Please do the needful.

Regards,
Nimesh Aghera

[CC Award Winners!]


Re: Implicit routing changes to Composite while re-deploy changes and make Upconfig command

Posted by Erick Erickson <er...@gmail.com>.
No images come through in e-mail so I can't see what you're actually doing.

On Wed, Dec 13, 2017 at 3:06 AM, Nimesh Aghera <na...@asite.com> wrote:

>
>
> Hi Erik,
>
>
>
> Thanks for the reply.
>
>
>
> Currently using solr cloud 6.1.0 and below is the solr admin UI which
> shows that routing changes after make upconfig command
>
>
>
> Below image are before make  re-upconfig where all data are indexed into
> particular shard.
>
>
>
> AMS_Config:
>
>
>
>
>
> Model Collection:  Created collection using Collection API and also data
> index into particular shard
>
>
>
>
>
> But after that I have make schema changes (Not Mandatory) and do upconfig
> as mention in previsou mail and start the solr instance its  shows my model
> collection as below and also it index data across all shards
>
>
>
>
>
>
>
> Its causes issues that if any how its required to make upconfig than it
> change the routing from implicit to compositeId.
>
>
>
> Please do needful.
>
>
>
> Regards,
>
> Nimesh Aghera
>
>
>
>
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com
> <er...@gmail.com>]
> Sent: Tuesday, December 12, 2017 9:29 PM
> To: solr-user
> Subject: Re: Implicit routing changes to Composite while re-deploy changes
> and make Upconfig command
>
>
>
> bq: It will changes the router value implicit to compositeId
>
>
>
> What evidence do you have of that? This better not happen. The routing
> scheme is part of the collection creation command and is stored in
> ZooKeeper in the collection's znode, it has nothing to do with the
> configset so updating the configs shouldn't change that at all.
>
>
>
> Best,
>
> Erick
>
>
>
> On Tue, Dec 12, 2017 at 1:15 AM, Nimesh Aghera <na...@asite.com> wrote:
>
> > Hello,
>
> >
>
> > I have implemented implicit routing with below configuration.
>
> > Created one default collection manually 'AMS_Config' which contains
> configurations files  schema,solrconfig etc.
>
> >
>
> > Using 'AMS_Config' I have created 2 collections model,workset
> respectively with below command which created 2 shard for each collection
> containing 2 node for each shard where nodes are each solr instance where
> collection created.
>
> >
>
> > Command:
>
> > /admin/collections?action=CREATE&name=model&replicationFactor=2&router
>
> > .name=implicit&router.field=dr&shards=shard1,shard2&maxShardsPerNode=2
>
> > &collection.configName=AMS_Config
>
> >
>
> > Collection Detail:
>
> > Model = Shard1,Shard2
>
> > Shard1 = node1,node2[leader]
>
> > Shard2 =node1[leader],node2
>
> >
>
> > Configuration in admin UI on solr for model collection:
>
> > Shard count:2
>
> > configName:AMS_Config
>
> > replicationFactor:2
>
> > maxShardsPerNode:2
>
> > router:implicit
>
> > autoAddReplicas:false
>
> >
>
> > after this I have index document to particular shard using set value
>
> > (router.field) dr = shard1
>
> >
>
> > Issue:  After indexing the document made changes in schema files and
>
> > redeploy using below command  for set latest configs zkcli.bat -cmd
>
> > upconfig -confdir ../../solr/AMS_Config/conf -confname AMS_Config -z
>
> > <zk Ips>
>
> >
>
> > It will changes the router value implicit to compositeId and now my
> document are index across all shard so why this should happens.How to avoid
> this.
>
> >
>
> > Please do the needful.
>
> >
>
> > Regards,
>
> > Nimesh Aghera
>
> >
>
> > [CC Award Winners!]
>
> >
>

RE: Implicit routing changes to Composite while re-deploy changes and make Upconfig command

Posted by Nimesh Aghera <na...@asite.com>.

Hi Erik,



Thanks for the reply.



Currently using solr cloud 6.1.0 and below is the solr admin UI which shows that routing changes after make upconfig command



Below image are before make  re-upconfig where all data are indexed into particular shard.



AMS_Config:

[cid:image001.png@01D37430.873F2130]





Model Collection:  Created collection using Collection API and also data index into particular shard



[cid:image002.png@01D37430.873F2130]



But after that I have make schema changes (Not Mandatory) and do upconfig as mention in previsou mail and start the solr instance its  shows my model collection as below and also it index data across all shards

[cid:image003.png@01D37430.873F2130]







Its causes issues that if any how its required to make upconfig than it change the routing from implicit to compositeId.



Please do needful.



Regards,

Nimesh Aghera





-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Tuesday, December 12, 2017 9:29 PM
To: solr-user
Subject: Re: Implicit routing changes to Composite while re-deploy changes and make Upconfig command



bq: It will changes the router value implicit to compositeId



What evidence do you have of that? This better not happen. The routing scheme is part of the collection creation command and is stored in ZooKeeper in the collection's znode, it has nothing to do with the configset so updating the configs shouldn't change that at all.



Best,

Erick



On Tue, Dec 12, 2017 at 1:15 AM, Nimesh Aghera <na...@asite.com>> wrote:

> Hello,

>

> I have implemented implicit routing with below configuration.

> Created one default collection manually 'AMS_Config' which contains configurations files  schema,solrconfig etc.

>

> Using 'AMS_Config' I have created 2 collections model,workset respectively with below command which created 2 shard for each collection containing 2 node for each shard where nodes are each solr instance where collection created.

>

> Command:

> /admin/collections?action=CREATE&name=model&replicationFactor=2&router

> .name=implicit&router.field=dr&shards=shard1,shard2&maxShardsPerNode=2

> &collection.configName=AMS_Config

>

> Collection Detail:

> Model = Shard1,Shard2

> Shard1 = node1,node2[leader]

> Shard2 =node1[leader],node2

>

> Configuration in admin UI on solr for model collection:

> Shard count:2

> configName:AMS_Config

> replicationFactor:2

> maxShardsPerNode:2

> router:implicit

> autoAddReplicas:false

>

> after this I have index document to particular shard using set value

> (router.field) dr = shard1

>

> Issue:  After indexing the document made changes in schema files and

> redeploy using below command  for set latest configs zkcli.bat -cmd

> upconfig -confdir ../../solr/AMS_Config/conf -confname AMS_Config -z

> <zk Ips>

>

> It will changes the router value implicit to compositeId and now my document are index across all shard so why this should happens.How to avoid this.

>

> Please do the needful.

>

> Regards,

> Nimesh Aghera

>

> [CC Award Winners!]

>

Re: Implicit routing changes to Composite while re-deploy changes and make Upconfig command

Posted by Shawn Heisey <ap...@elyograg.org>.
On 12/18/2017 7:21 AM, Nimesh Aghera wrote:
> Thanks Shawn for reply,
> Need to do below step.
> 
> After creating collection using the Collection API  clear the version-2 folder from zoo_data and after that I will start server and make upconfig the collection is up but the routing changes from implicit to explicit with solr 6.1.0 while I do same to replicate in solr 7.1.0  while up server it will gives org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Replica core_node5 is not present in cluster state: null So unable to replicate in solr 7.1.0 and also while start server it will create the 'gettingstarted' collection every time which is already exists.

I'm pretty sure that deleting the zoodata/version-2 folder deletes all 
zookeeper data.  Don't do that.

SolrCloud can't recover from all of its ZK data being deleted.  I'm 
surprised anything works at all.

Thanks,
Shawn

RE: Implicit routing changes to Composite while re-deploy changes and make Upconfig command

Posted by Nimesh Aghera <na...@asite.com>.
Hi

Thanks Shawn for reply,
Need to do below step.

After creating collection using the Collection API  clear the version-2 folder from zoo_data and after that I will start server and make upconfig the collection is up but the routing changes from implicit to explicit with solr 6.1.0 while I do same to replicate in solr 7.1.0  while up server it will gives org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Replica core_node5 is not present in cluster state: null So unable to replicate in solr 7.1.0 and also while start server it will create the 'gettingstarted' collection every time which is already exists.

So will you please try above scenario.
  
Please do needful.

Regards,
Nimesh Aghera

-----Original Message-----
From: Shawn Heisey [mailto:apache@elyograg.org]
Sent: Thursday, December 14, 2017 8:55 PM
To: solr-user@lucene.apache.org
Subject: Re: Implicit routing changes to Composite while re-deploy changes and make Upconfig command

On 12/13/2017 10:08 PM, Nimesh Aghera wrote:
> Currently using solr cloud 6.1.0 and below is the solr admin UI which 
> shows that routing changes after make upconfig command
>
> Below values shown in Solr Admin UI  are before make  re-upconfig where all data are indexed into particular shard.

I could not reproduce this on Solr 7.1.0.  If there *is* a problem in version 6.1.0, it's not going to get fixed.  Versions that old will never see an update.

First, I started up the cloud example with "bin\solr start -e cloud -nomprompt".  This was on a Windows 7 system with Oracle Java 8u144.

Then I made a Collections API call like yours to create a collection:

http://localhost:8983/solr/admin/collections?action=CREATE&name=model&replicationFactor=2&router.name=implicit&router.field=title&shards=shard1,shard2&maxShardsPerNode=2&collection.configName=gettingstarted

This is the response from that call:

{
  "responseHeader":{
    "status":0,
    "QTime":4971},
  "success":{
    "192.168.1.4:7574_solr":{
      "responseHeader":{
        "status":0,
        "QTime":2843},
      "core":"model_shard2_replica_n6"},
    "192.168.1.4:7574_solr":{
      "responseHeader":{
        "status":0,
        "QTime":3078},
      "core":"model_shard1_replica_n2"},
    "192.168.1.4:8983_solr":{
      "responseHeader":{
        "status":0,
        "QTime":3382},
      "core":"model_shard2_replica_n4"},
    "192.168.1.4:8983_solr":{
      "responseHeader":{
        "status":0,
        "QTime":3618},
      "core":"model_shard1_replica_n1"}}}

This is a screenshot of the collection overview, showing the implicit
router:

https://www.dropbox.com/s/x5gtpkkl6lv18g8/test1-collection-info-after-create.png?dl=0

Then I made a whitespace-only change to solrconfig.xml in the _default configset, and re-uploaded it to replace the configuration I used for the collection:

bin\solr zk upconfig -d server\solr\configsets\_default -n gettingstarted -z localhost:9983

There was no difference at all in the collection details.  I even did a collection reload, and still did not see anything change.

Thanks,
Shawn


Re: Implicit routing changes to Composite while re-deploy changes and make Upconfig command

Posted by Shawn Heisey <ap...@elyograg.org>.
On 12/13/2017 10:08 PM, Nimesh Aghera wrote:
> Currently using solr cloud 6.1.0 and below is the solr admin UI which shows that routing changes after make upconfig command
>
> Below values shown in Solr Admin UI  are before make  re-upconfig where all data are indexed into particular shard.

I could not reproduce this on Solr 7.1.0.  If there *is* a problem in
version 6.1.0, it's not going to get fixed.  Versions that old will
never see an update.

First, I started up the cloud example with "bin\solr start -e cloud
-nomprompt".  This was on a Windows 7 system with Oracle Java 8u144.

Then I made a Collections API call like yours to create a collection:

http://localhost:8983/solr/admin/collections?action=CREATE&name=model&replicationFactor=2&router.name=implicit&router.field=title&shards=shard1,shard2&maxShardsPerNode=2&collection.configName=gettingstarted

This is the response from that call:

{
  "responseHeader":{
    "status":0,
    "QTime":4971},
  "success":{
    "192.168.1.4:7574_solr":{
      "responseHeader":{
        "status":0,
        "QTime":2843},
      "core":"model_shard2_replica_n6"},
    "192.168.1.4:7574_solr":{
      "responseHeader":{
        "status":0,
        "QTime":3078},
      "core":"model_shard1_replica_n2"},
    "192.168.1.4:8983_solr":{
      "responseHeader":{
        "status":0,
        "QTime":3382},
      "core":"model_shard2_replica_n4"},
    "192.168.1.4:8983_solr":{
      "responseHeader":{
        "status":0,
        "QTime":3618},
      "core":"model_shard1_replica_n1"}}}

This is a screenshot of the collection overview, showing the implicit
router:

https://www.dropbox.com/s/x5gtpkkl6lv18g8/test1-collection-info-after-create.png?dl=0

Then I made a whitespace-only change to solrconfig.xml in the _default
configset, and re-uploaded it to replace the configuration I used for
the collection:

bin\solr zk upconfig -d server\solr\configsets\_default -n
gettingstarted -z localhost:9983

There was no difference at all in the collection details.  I even did a
collection reload, and still did not see anything change.

Thanks,
Shawn


RE: Implicit routing changes to Composite while re-deploy changes and make Upconfig command

Posted by Nimesh Aghera <na...@asite.com>.

Hi Erik,



Thanks for the reply.



Currently using solr cloud 6.1.0 and below is the solr admin UI which shows that routing changes after make upconfig command



Below values shown in Solr Admin UI  are before make  re-upconfig where all data are indexed into particular shard.



AMS_Config:

----------------

Shard count:2

configName:AMS_Config

replicationFactor:1

maxShardsPerNode:1

router:compositeId

autoAddReplicas:false





Model Collection:  Created collection using Collection API and also data index into particular shard

---------------------

Shard count:2

configName:AMS_Config

replicationFactor:2

maxShardsPerNode:2

router:implicit

autoAddReplicas:false





But after that I have make schema changes (Not Mandatory) and do upconfig as mention in previsou mail and start the solr instance its  shows my model collection as below and also it index data across all shards

Shard count:2

configName:AMS_Config

replicationFactor:1

maxShardsPerNode:1

router:compositeId

autoAddReplicas:false



Its causes issues that if any how its required to make upconfig than it change the routing from implicit to compositeId.



Please do needful.



Regards,

Nimesh Aghera



-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Tuesday, December 12, 2017 9:29 PM
To: solr-user
Subject: Re: Implicit routing changes to Composite while re-deploy changes and make Upconfig command



bq: It will changes the router value implicit to compositeId



What evidence do you have of that? This better not happen. The routing scheme is part of the collection creation command and is stored in ZooKeeper in the collection's znode, it has nothing to do with the configset so updating the configs shouldn't change that at all.



Best,

Erick



On Tue, Dec 12, 2017 at 1:15 AM, Nimesh Aghera <na...@asite.com>> wrote:

> Hello,

>

> I have implemented implicit routing with below configuration.

> Created one default collection manually 'AMS_Config' which contains configurations files  schema,solrconfig etc.

>

> Using 'AMS_Config' I have created 2 collections model,workset respectively with below command which created 2 shard for each collection containing 2 node for each shard where nodes are each solr instance where collection created.

>

> Command:

> /admin/collections?action=CREATE&name=model&replicationFactor=2&router

> .name=implicit&router.field=dr&shards=shard1,shard2&maxShardsPerNode=2

> &collection.configName=AMS_Config

>

> Collection Detail:

> Model = Shard1,Shard2

> Shard1 = node1,node2[leader]

> Shard2 =node1[leader],node2

>

> Configuration in admin UI on solr for model collection:

> Shard count:2

> configName:AMS_Config

> replicationFactor:2

> maxShardsPerNode:2

> router:implicit

> autoAddReplicas:false

>

> after this I have index document to particular shard using set value

> (router.field) dr = shard1

>

> Issue:  After indexing the document made changes in schema files and

> redeploy using below command  for set latest configs zkcli.bat -cmd

> upconfig -confdir ../../solr/AMS_Config/conf -confname AMS_Config -z

> <zk Ips>

>

> It will changes the router value implicit to compositeId and now my document are index across all shard so why this should happens.How to avoid this.

>

> Please do the needful.

>

> Regards,

> Nimesh Aghera

>

> [CC Award Winners!]

>

Re: Implicit routing changes to Composite while re-deploy changes and make Upconfig command

Posted by Erick Erickson <er...@gmail.com>.
bq: It will changes the router value implicit to compositeId

What evidence do you have of that? This better not happen. The routing
scheme is part of the collection creation command and is stored in
ZooKeeper in the collection's znode, it has nothing to do with the
configset so updating the configs shouldn't change that at all.

Best,
Erick

On Tue, Dec 12, 2017 at 1:15 AM, Nimesh Aghera <na...@asite.com> wrote:
> Hello,
>
> I have implemented implicit routing with below configuration.
> Created one default collection manually 'AMS_Config' which contains configurations files  schema,solrconfig etc.
>
> Using 'AMS_Config' I have created 2 collections model,workset respectively with below command which created 2 shard for each collection containing 2 node for each shard where nodes are each solr instance where collection created.
>
> Command: /admin/collections?action=CREATE&name=model&replicationFactor=2&router.name=implicit&router.field=dr&shards=shard1,shard2&maxShardsPerNode=2&collection.configName=AMS_Config
>
> Collection Detail:
> Model = Shard1,Shard2
> Shard1 = node1,node2[leader]
> Shard2 =node1[leader],node2
>
> Configuration in admin UI on solr for model collection:
> Shard count:2
> configName:AMS_Config
> replicationFactor:2
> maxShardsPerNode:2
> router:implicit
> autoAddReplicas:false
>
> after this I have index document to particular shard using set value (router.field) dr = shard1
>
> Issue:  After indexing the document made changes in schema files and redeploy using below command  for set latest configs
> zkcli.bat -cmd upconfig -confdir ../../solr/AMS_Config/conf -confname AMS_Config -z <zk Ips>
>
> It will changes the router value implicit to compositeId and now my document are index across all shard so why this should happens.How to avoid this.
>
> Please do the needful.
>
> Regards,
> Nimesh Aghera
>
> [CC Award Winners!]
>