Solr 7.7.2 - SolrCloud - Autoscale Triggers - indexSize trigger - Failure isn't sending listener a FAILED message, but a SUCCEEDED message

[Andrew Kettmann <an...@evolve24.com>]

First, pardon any copy/pasted examples of my policies/triggers/etc as they are in Python format as that is my language of choice when working with APIs and the like. So Ignore that they are not JSON exactly as the APIs are getting JSON.


Issue summary: Collection with strict autoscaling rules that cannot be satisfied, when an IndexSize trigger is fired to split the core, it fires over and over, and it sends a SUCCESSFUL message via a configured HTTP listener.


Solr 7.7.2, SolrCloud. Collection with the following policy:


{'set-policy': {'othersolr7': [{'node': '#ANY',
                                'replica': '<2',
                                'strict': 'true'},
                               {'replica': '#ALL',
                                'shard': '#ANY',
                                'sysprop.HELM_CHART': 'othersolr7'}]}}


So one core per node, and strict set to true, There are TWO total nodes that satisfy this.


Collection is 1 shard with 2 total NRT replicas.


Configured a trigger to split at 9999 docs:


{'aboveDocs': '9999',
 'event': 'indexSize',
 'name': 'index_size_trigger_9999_docs',
 'splitMethod': 'link',
 'waitFor': '5s'}


Also a listener configured to send HTTP posts:


{'set-listener': {'afterAction': ['execute_plan'],
                  'class': 'solr.HttpTriggerListener',
                  'header.X-Trigger': '${config.trigger}',
                  'name': 'test-to-flask',
                  'stage': ['ABORTED', 'SUCCEEDED', 'FAILED'],
                  'trigger': 'index_size_trigger_9999_docs',
                  'url': 'http://HOST:5000/post/${config.name:invalidName}/${config.trigger}/${event.id}?STAGE=${stage}'}}


I put 10K docs into the collection to trigger the indexsize trigger and it triggers over and over, sending a post to my listener each time, and sending a SUCCESSFUL message after each one. New event ID each time it triggers and goes round. The message received for the "afterAction" of the execute_plan shows an error:


 'context.operations': '[{\n'
                       '  '
                       '"class":"org.apache.solr.client.solrj.request.CollectionAdminRequest$SplitShard",\n'
                       '  "method":"GET",\n'
                       '  "params.action":"SPLITSHARD",\n'
                       '  '
                       '"params.async":"index_size_trigger_9999_docs/2bbd7de63de68T2eupg9aq3fuuy2lnyi9s1ha0h/0",\n'
                       '  "params.waitForFinalState":"true",\n'
                       '  "params.collection":"othersolr7",\n'
                       '  "params.shard":"shard1",\n'
                       '  "params.splitMethod":"link"}]',
 'context.responses': '[{responseHeader={status=0,QTime=2},Operation '
                      'splitshard caused '
                      'exception:=org.apache.solr.common.SolrException:org.apache.solr.common.SolrException,exception={msg=null,rspCode=500},status={state=failed,msg=found '
                      '[index_size_trigger_9999_docs/2bbd7de63de68T2eupg9aq3fuuy2lnyi9s1ha0h/0] '
                      'in failed tasks}}]',


But then after I get that I still receive a successful message:


{'actionName': '',
 'config.afterActions': 'execute_plan',
 'config.beforeActions': '',
 'config.listenerClass': 'solr.HttpTriggerListener',
 'config.name': 'test-to-flask',
 'config.properties.afterAction': '[execute_plan]',
 'config.properties.beforeAction': '[]',
 'config.properties.class': 'solr.HttpTriggerListener',
 'config.properties.header.X-Trigger': '${config.trigger}',
 'config.properties.stage': '[ABORTED, SUCCEEDED, FAILED]',
 'config.properties.trigger': 'index_size_trigger_9999_docs',
 'config.properties.url': 'http://HOST:5000/post/${config.name:invalidName}/${config.trigger}/${event.id}?STAGE=${stage}',
 'config.stages': 'ABORTED,SUCCEEDED,FAILED',
 'config.trigger': 'index_size_trigger_9999_docs',
 'error': '',
 'event.eventTime': '769485776871016',
 'event.eventType': 'INDEXSIZE',
 'event.id': '2bbd7de63de68T2eupg9aq3fuuy2lnyi9s1ha0h',
 'event.properties.__start__': '1',
 'event.properties._enqueue_time_': '769495912359525',
 'event.properties.aboveSize': '{othersolr7_shard1_replica_n2=docs=10000, '
                               'bytes=9708660}',
 'event.properties.belowSize': '{}',
 'event.properties.requestedOps': '[Op{action=SPLITSHARD, '
                                  'hints={COLL_SHARD=[{\n'
                                  '  "first":"othersolr7",\n'
                                  '  "second":"shard1"}], '
                                  'PARAMS={splitMethod=link}}}]',
 'event.source': 'index_size_trigger_9999_docs',
 'message': '',
 'stage': 'SUCCEEDED'}



And then it continually loops and sends "successful" messages after each failed attempt. The failure, I understand because this is an unfixable situation for Solr, it can't both meet my policies in this situation AND execute the trigger. The problem is the listener sending successes each time. Anyone able to shed some light on this ? Working on setting up some automation so that when we split cores, we automatically create new containers for Solr to use and shuffle cores onto, I was testing failure cases and found this issue. Is this just a ticket I need to open in Jira or is there something I am missing ?



[https://storage.googleapis.com/e24-email-images/e24logonotag.png]<https://www.evolve24.com> Andrew Kettmann
DevOps Engineer
P: 1.314.596.2836
[LinkedIn]<https://linkedin.com/company/evolve24> [Twitter] <https://twitter.com/evolve24>  [Instagram] <https://www.instagram.com/evolve_24>

evolve24 Confidential & Proprietary Statement: This email and any attachments are confidential and may contain information that is privileged, confidential or exempt from disclosure under applicable law. It is intended for the use of the recipients. If you are not the intended recipient, or believe that you have received this communication in error, please do not read, print, copy, retransmit, disseminate, or otherwise use the information. Please delete this email and attachments, without reading, printing, copying, forwarding or saving them, and notify the Sender immediately by reply email. No confidentiality or privilege is waived or lost by any transmission in error.