Artemis operator: MQTT retained messages not consistent across 3 broker cluster

["Shields, Paul Michael" <pa...@hpe.com>]

Hi,
Have a bunch of IoT clients connecting to a broker. Using MQTT Retain Message feature. Using Retain messages feature to get current state of my system when they are brought into system. Works great with single broker.  But need HA so utilized cloud operator in Kubernetes to have a cluster of three brokers, all in same name space.  When I use the three brokers, the clients each round robin connect to the broker instances and each has an incomplete and inconsistent set of retained messages.  The retained messages appear to only be from the local broker the client connects to.  I have set redistribution-delay=0 in the broker.xml file for each broker instance with this stanza:
         <!--default for catch all-->
         <address-setting match="#">
            <redistribution-delay>0</redistribution-delay>
             <other config settings>
         </address-setting>
But, this did not help.  Is there another parameter I need to set in addition to have the redistribution-delay setting?

My activemqartemis.yaml file:

apiVersion: broker.amq.io/v1beta1

kind: ActiveMQArtemis

metadata:

  name: {{ include "my-mqtt.fullname" . }}

spec:

  # console:

  #  expose: true

  deploymentPlan:

    size: {{ .Values.replicaCount }}

    persistenceEnabled: true

    messageMigration: true

    enableMetricsPlugin: true

    {{- if .Values.init.enabled }}

    initImage: {{ .Values.init.repository }}:{{ .Values.init.tag }}

    {{- end }}

    resources:

      limits:

        cpu: "500m"

        memory: "1024Mi"

      requests:

        cpu: "250m"

        memory: "512Mi"

    labels:
      app.kubernetes.io/name: {{ template "my-mqtt.name" . }}
    podSecurity:
     serviceAccountName: {{ include "my-mqtt.fullname" . }}
    # fsGroup 0 is a work around for a long standing bug when using persistence
    # https://github.com/artemiscloud/activemq-artemis-operator/issues/187
    podSecurityContext:
      fsGroup: 0
      fsGroupChangePolicy: "OnRootMismatch"
{{ if .Values.tls.enabled }}
  acceptors:
  - name: tls
    protocols: all
    port: 8883
    sslEnabled: true
    sslSecret: {{ include "my-mqtt.fullname" . }}-tls
{{ else }}
  acceptors:
  - name: insecure
    protocols: all
    port: 1883
    sslEnabled: false
{{ end }}

                                                                                                                            42,1          52%
Thanks in advance,
Paul