You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Andrew Kettmann <an...@evolve24.com> on 2019/06/25 18:04:21 UTC

Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules

Using docker 7.7.2 image


Solr 7.7.2 on new Znode on ZK. Created the chroot using solr zk mkroot.


Created a policy:

{'set-policy': {'banana': [{'replica': '#ALL',
                            'sysprop.HELM_CHART': 'notbanana'}]}}


No errors on creation of the policy.


I have no nodes that have that value for the system property "HELM_CHART", I have nodes that contain "banana" and "rulesos" for that value only.


I create the collection with a call to the /admin/collections:

{'action': 'CREATE',
 'collection.configName': 'project-solr-7',
 'name': 'banana',
 'numShards': '2',
 'policy': 'banana',
 'replicationFactor': '2'}


and it creates the collection without an error. Which what I expected was the collection creation to fail. This is the behavior I had seen in the past, but after tearing down and recreating the cluster in a higher environment, it does not appear to function.


Is there some prerequisite before policies will be respected? The .system collection is in place as expected, and I am not seeing anything in the logs on the overseer to suggest any problems.

[https://storage.googleapis.com/e24-email-images/e24logonotag.png]<https://www.evolve24.com> Andrew Kettmann
DevOps Engineer
P: 1.314.596.2836
[LinkedIn]<https://linkedin.com/company/evolve24> [Twitter] <https://twitter.com/evolve24>  [Instagram] <https://www.instagram.com/evolve_24>

evolve24 Confidential & Proprietary Statement: This email and any attachments are confidential and may contain information that is privileged, confidential or exempt from disclosure under applicable law. It is intended for the use of the recipients. If you are not the intended recipient, or believe that you have received this communication in error, please do not read, print, copy, retransmit, disseminate, or otherwise use the information. Please delete this email and attachments, without reading, printing, copying, forwarding or saving them, and notify the Sender immediately by reply email. No confidentiality or privilege is waived or lost by any transmission in error.

Re: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules

Posted by Andrew Kettmann <an...@evolve24.com>.
Entered ticket https://issues.apache.org/jira/browse/SOLR-13586


Sadly, no patch attached this time as it is a much more complicated issue than my last one, and a good bit above my paygrade with Java.



________________________________
From: Andrzej Białecki <ab...@getopt.org>
Sent: Friday, June 28, 2019 4:29:49 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules

Andrew, please create a JIRA issue - in my opinion this is a bug not a feature, or at least something that needs clarification.

> On 27 Jun 2019, at 23:56, Andrew Kettmann <an...@evolve24.com> wrote:
>
> I found the issue. Autoscaling seems to silently ignore rules (at least sysprop rules). Example rule:
>
>
> {'set-policy': {'sales-uat': [{'node': '#ANY',
>                               'replica': '<2',
>                               'strict': 'false'},
>                              {'replica': '#ALL',
>                               'strict': 'true',
>                               'sysprop.HELM_CHART': 'foo'}]}}
>
>
> Two cases will get the sysprop rule ignored:
>
>  1.  No nodes have a HELM_CHART system property defined
>  2.  No nodes have the value "foo" for the HELM_CHART system property
>
>
> If you have SOME nodes that have -DHELM_CHART=foo, then it will fail if it cannot satisfy another strict rule. So sysprop autoscaling rules appear to be unable to be strict on their own it appears.
>
>
> Hopefully this can solve some issues for other people as well.
>
> ________________________________
> From: Andrew Kettmann
> Sent: Tuesday, June 25, 2019 1:04:21 PM
> To: solr-user@lucene.apache.org
> Subject: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules
>
>
> Using docker 7.7.2 image
>
>
> Solr 7.7.2 on new Znode on ZK. Created the chroot using solr zk mkroot.
>
>
> Created a policy:
>
> {'set-policy': {'banana': [{'replica': '#ALL',
>                            'sysprop.HELM_CHART': 'notbanana'}]}}
>
>
> No errors on creation of the policy.
>
>
> I have no nodes that have that value for the system property "HELM_CHART", I have nodes that contain "banana" and "rulesos" for that value only.
>
>
> I create the collection with a call to the /admin/collections:
>
> {'action': 'CREATE',
> 'collection.configName': 'project-solr-7',
> 'name': 'banana',
> 'numShards': '2',
> 'policy': 'banana',
> 'replicationFactor': '2'}
>
>
> and it creates the collection without an error. Which what I expected was the collection creation to fail. This is the behavior I had seen in the past, but after tearing down and recreating the cluster in a higher environment, it does not appear to function.
>
>
> Is there some prerequisite before policies will be respected? The .system collection is in place as expected, and I am not seeing anything in the logs on the overseer to suggest any problems.
>
> [https://storage.googleapis.com/e24-email-images/e24logonotag.png]<https://www.evolve24.com> Andrew Kettmann
> DevOps Engineer
> P: 1.314.596.2836
> [LinkedIn]<https://linkedin.com/company/evolve24> [Twitter] <https://twitter.com/evolve24>  [Instagram] <https://www.instagram.com/evolve_24>
>
> evolve24 Confidential & Proprietary Statement: This email and any attachments are confidential and may contain information that is privileged, confidential or exempt from disclosure under applicable law. It is intended for the use of the recipients. If you are not the intended recipient, or believe that you have received this communication in error, please do not read, print, copy, retransmit, disseminate, or otherwise use the information. Please delete this email and attachments, without reading, printing, copying, forwarding or saving them, and notify the Sender immediately by reply email. No confidentiality or privilege is waived or lost by any transmission in error.

Re: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules

Posted by Andrzej Białecki <ab...@getopt.org>.
Andrew, please create a JIRA issue - in my opinion this is a bug not a feature, or at least something that needs clarification.

> On 27 Jun 2019, at 23:56, Andrew Kettmann <an...@evolve24.com> wrote:
> 
> I found the issue. Autoscaling seems to silently ignore rules (at least sysprop rules). Example rule:
> 
> 
> {'set-policy': {'sales-uat': [{'node': '#ANY',
>                               'replica': '<2',
>                               'strict': 'false'},
>                              {'replica': '#ALL',
>                               'strict': 'true',
>                               'sysprop.HELM_CHART': 'foo'}]}}
> 
> 
> Two cases will get the sysprop rule ignored:
> 
>  1.  No nodes have a HELM_CHART system property defined
>  2.  No nodes have the value "foo" for the HELM_CHART system property
> 
> 
> If you have SOME nodes that have -DHELM_CHART=foo, then it will fail if it cannot satisfy another strict rule. So sysprop autoscaling rules appear to be unable to be strict on their own it appears.
> 
> 
> Hopefully this can solve some issues for other people as well.
> 
> ________________________________
> From: Andrew Kettmann
> Sent: Tuesday, June 25, 2019 1:04:21 PM
> To: solr-user@lucene.apache.org
> Subject: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules
> 
> 
> Using docker 7.7.2 image
> 
> 
> Solr 7.7.2 on new Znode on ZK. Created the chroot using solr zk mkroot.
> 
> 
> Created a policy:
> 
> {'set-policy': {'banana': [{'replica': '#ALL',
>                            'sysprop.HELM_CHART': 'notbanana'}]}}
> 
> 
> No errors on creation of the policy.
> 
> 
> I have no nodes that have that value for the system property "HELM_CHART", I have nodes that contain "banana" and "rulesos" for that value only.
> 
> 
> I create the collection with a call to the /admin/collections:
> 
> {'action': 'CREATE',
> 'collection.configName': 'project-solr-7',
> 'name': 'banana',
> 'numShards': '2',
> 'policy': 'banana',
> 'replicationFactor': '2'}
> 
> 
> and it creates the collection without an error. Which what I expected was the collection creation to fail. This is the behavior I had seen in the past, but after tearing down and recreating the cluster in a higher environment, it does not appear to function.
> 
> 
> Is there some prerequisite before policies will be respected? The .system collection is in place as expected, and I am not seeing anything in the logs on the overseer to suggest any problems.
> 
> [https://storage.googleapis.com/e24-email-images/e24logonotag.png]<https://www.evolve24.com> Andrew Kettmann
> DevOps Engineer
> P: 1.314.596.2836
> [LinkedIn]<https://linkedin.com/company/evolve24> [Twitter] <https://twitter.com/evolve24>  [Instagram] <https://www.instagram.com/evolve_24>
> 
> evolve24 Confidential & Proprietary Statement: This email and any attachments are confidential and may contain information that is privileged, confidential or exempt from disclosure under applicable law. It is intended for the use of the recipients. If you are not the intended recipient, or believe that you have received this communication in error, please do not read, print, copy, retransmit, disseminate, or otherwise use the information. Please delete this email and attachments, without reading, printing, copying, forwarding or saving them, and notify the Sender immediately by reply email. No confidentiality or privilege is waived or lost by any transmission in error.

Re: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules

Posted by Andrew Kettmann <an...@evolve24.com>.
I found the issue. Autoscaling seems to silently ignore rules (at least sysprop rules). Example rule:


{'set-policy': {'sales-uat': [{'node': '#ANY',
                               'replica': '<2',
                               'strict': 'false'},
                              {'replica': '#ALL',
                               'strict': 'true',
                               'sysprop.HELM_CHART': 'foo'}]}}


Two cases will get the sysprop rule ignored:

  1.  No nodes have a HELM_CHART system property defined
  2.  No nodes have the value "foo" for the HELM_CHART system property


If you have SOME nodes that have -DHELM_CHART=foo, then it will fail if it cannot satisfy another strict rule. So sysprop autoscaling rules appear to be unable to be strict on their own it appears.


Hopefully this can solve some issues for other people as well.

________________________________
From: Andrew Kettmann
Sent: Tuesday, June 25, 2019 1:04:21 PM
To: solr-user@lucene.apache.org
Subject: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules


Using docker 7.7.2 image


Solr 7.7.2 on new Znode on ZK. Created the chroot using solr zk mkroot.


Created a policy:

{'set-policy': {'banana': [{'replica': '#ALL',
                            'sysprop.HELM_CHART': 'notbanana'}]}}


No errors on creation of the policy.


I have no nodes that have that value for the system property "HELM_CHART", I have nodes that contain "banana" and "rulesos" for that value only.


I create the collection with a call to the /admin/collections:

{'action': 'CREATE',
 'collection.configName': 'project-solr-7',
 'name': 'banana',
 'numShards': '2',
 'policy': 'banana',
 'replicationFactor': '2'}


and it creates the collection without an error. Which what I expected was the collection creation to fail. This is the behavior I had seen in the past, but after tearing down and recreating the cluster in a higher environment, it does not appear to function.


Is there some prerequisite before policies will be respected? The .system collection is in place as expected, and I am not seeing anything in the logs on the overseer to suggest any problems.

[https://storage.googleapis.com/e24-email-images/e24logonotag.png]<https://www.evolve24.com> Andrew Kettmann
DevOps Engineer
P: 1.314.596.2836
[LinkedIn]<https://linkedin.com/company/evolve24> [Twitter] <https://twitter.com/evolve24>  [Instagram] <https://www.instagram.com/evolve_24>

evolve24 Confidential & Proprietary Statement: This email and any attachments are confidential and may contain information that is privileged, confidential or exempt from disclosure under applicable law. It is intended for the use of the recipients. If you are not the intended recipient, or believe that you have received this communication in error, please do not read, print, copy, retransmit, disseminate, or otherwise use the information. Please delete this email and attachments, without reading, printing, copying, forwarding or saving them, and notify the Sender immediately by reply email. No confidentiality or privilege is waived or lost by any transmission in error.

Re: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules

Posted by Andrew Kettmann <an...@evolve24.com>.
Is there some step I am missing here? Policies seem to be entirely ignored in this new cluster and I am at a loss. Is there some default setting that will cause autoscaling to be ignored?

________________________________
From: Andrew Kettmann
Sent: Tuesday, June 25, 2019 1:04:21 PM
To: solr-user@lucene.apache.org
Subject: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules


Using docker 7.7.2 image


Solr 7.7.2 on new Znode on ZK. Created the chroot using solr zk mkroot.


Created a policy:

{'set-policy': {'banana': [{'replica': '#ALL',
                            'sysprop.HELM_CHART': 'notbanana'}]}}


No errors on creation of the policy.


I have no nodes that have that value for the system property "HELM_CHART", I have nodes that contain "banana" and "rulesos" for that value only.


I create the collection with a call to the /admin/collections:

{'action': 'CREATE',
 'collection.configName': 'project-solr-7',
 'name': 'banana',
 'numShards': '2',
 'policy': 'banana',
 'replicationFactor': '2'}


and it creates the collection without an error. Which what I expected was the collection creation to fail. This is the behavior I had seen in the past, but after tearing down and recreating the cluster in a higher environment, it does not appear to function.


Is there some prerequisite before policies will be respected? The .system collection is in place as expected, and I am not seeing anything in the logs on the overseer to suggest any problems.

[https://storage.googleapis.com/e24-email-images/e24logonotag.png]<https://www.evolve24.com> Andrew Kettmann
DevOps Engineer
P: 1.314.596.2836
[LinkedIn]<https://linkedin.com/company/evolve24> [Twitter] <https://twitter.com/evolve24>  [Instagram] <https://www.instagram.com/evolve_24>

evolve24 Confidential & Proprietary Statement: This email and any attachments are confidential and may contain information that is privileged, confidential or exempt from disclosure under applicable law. It is intended for the use of the recipients. If you are not the intended recipient, or believe that you have received this communication in error, please do not read, print, copy, retransmit, disseminate, or otherwise use the information. Please delete this email and attachments, without reading, printing, copying, forwarding or saving them, and notify the Sender immediately by reply email. No confidentiality or privilege is waived or lost by any transmission in error.