You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Bruno Cadonna (Jira)" <ji...@apache.org> on 2022/02/10 09:48:00 UTC
[jira] [Resolved] (KAFKA-8153) Streaming application with state stores takes up to 1 hour to restart
[ https://issues.apache.org/jira/browse/KAFKA-8153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bruno Cadonna resolved KAFKA-8153.
----------------------------------
Resolution: Not A Problem
> Streaming application with state stores takes up to 1 hour to restart
> ---------------------------------------------------------------------
>
> Key: KAFKA-8153
> URL: https://issues.apache.org/jira/browse/KAFKA-8153
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.1.0
> Reporter: Michael Melsen
> Priority: Major
>
> We are using spring cloud stream with Kafka streams 2.0.1 and utilizing the InteractiveQueryService to fetch data from the stores. There are 4 stores that persist data on disk after aggregating data. The code for the topology looks like this:
> {code:java}
> @Slf4j
> @EnableBinding(SensorMeasurementBinding.class)
> public class Consumer {
> public static final String RETENTION_MS = "retention.ms";
> public static final String CLEANUP_POLICY = "cleanup.policy";
> @Value("${windowstore.retention.ms}")
> private String retention;
> /**
> * Process the data flowing in from a Kafka topic. Aggregate the data to:
> * - 2 minute
> * - 15 minutes
> * - one hour
> * - 12 hours
> *
> * @param stream
> */
> @StreamListener(SensorMeasurementBinding.ERROR_SCORE_IN)
> public void process(KStream<String, SensorMeasurement> stream) {
> Map<String, String> topicConfig = new HashMap<>();
> topicConfig.put(RETENTION_MS, retention);
> topicConfig.put(CLEANUP_POLICY, "delete");
> log.info("Changelog and local window store retention.ms: {} and cleanup.policy: {}",
> topicConfig.get(RETENTION_MS),
> topicConfig.get(CLEANUP_POLICY));
> createWindowStore(LocalStore.TWO_MINUTES_STORE, topicConfig, stream);
> createWindowStore(LocalStore.FIFTEEN_MINUTES_STORE, topicConfig, stream);
> createWindowStore(LocalStore.ONE_HOUR_STORE, topicConfig, stream);
> createWindowStore(LocalStore.TWELVE_HOURS_STORE, topicConfig, stream);
> }
> private void createWindowStore(
> LocalStore localStore,
> Map<String, String> topicConfig,
> KStream<String, SensorMeasurement> stream) {
> // Configure how the statestore should be materialized using the provide storeName
> Materialized<String, ErrorScore, WindowStore<Bytes, byte[]>> materialized = Materialized
> .as(localStore.getStoreName());
> // Set retention of changelog topic
> materialized.withLoggingEnabled(topicConfig);
> // Configure how windows looks like and how long data will be retained in local stores
> TimeWindows configuredTimeWindows = getConfiguredTimeWindows(
> localStore.getTimeUnit(), Long.parseLong(topicConfig.get(RETENTION_MS)));
> // Processing description:
> // The input data are 'samples' with key <installationId>:<assetId>:<modelInstanceId>:<algorithmName>
> // 1. With the map we add the Tag to the key and we extract the error score from the data
> // 2. With the groupByKey we group the data on the new key
> // 3. With windowedBy we split up the data in time intervals depending on the provided LocalStore enum
> // 4. With reduce we determine the maximum value in the time window
> // 5. Materialized will make it stored in a table
> stream
> .map(getInstallationAssetModelAlgorithmTagKeyMapper())
> .groupByKey()
> .windowedBy(configuredTimeWindows)
> .reduce((aggValue, newValue) -> getMaxErrorScore(aggValue, newValue), materialized);
> }
> private TimeWindows getConfiguredTimeWindows(long windowSizeMs, long retentionMs) {
> TimeWindows timeWindows = TimeWindows.of(windowSizeMs);
> timeWindows.until(retentionMs);
> return timeWindows;
> }
> /**
> * Determine the max error score to keep by looking at the aggregated error signal and
> * freshly consumed error signal
> *
> * @param aggValue
> * @param newValue
> * @return
> */
> private ErrorScore getMaxErrorScore(ErrorScore aggValue, ErrorScore newValue) {
> if(aggValue.getErrorSignal() > newValue.getErrorSignal()) {
> return aggValue;
> }
> return newValue;
> }
> private KeyValueMapper<String, SensorMeasurement,
> KeyValue<? extends String, ? extends ErrorScore>> getInstallationAssetModelAlgorithmTagKeyMapper() {
> return (s, sensorMeasurement) -> new KeyValue<>(s + "::" + sensorMeasurement.getT(),
> new ErrorScore(sensorMeasurement.getTs(), sensorMeasurement.getE(), sensorMeasurement.getO()));
> }
> }
> {code}
> So we are materializing aggregated data to four different stores after determining the max value within a specific window for a specific key. Please note that retention which is set to two months of data and the clean up policy delete. We don't compact data.
> The size of the individual state stores on disk is between 14 to 20 gb of data.
> We are making use of Interactive Queries: [https://docs.confluent.io/current/streams/developer-guide/interactive-queries.html#interactive-queries]
> On our setup we have 4 instances of our streaming app to be used as one consumer group. So every instance will store a specific part of all data in its store.
> This all seems to work nicely. Until we restart one or more instances and wait for it to become available again. (Restart time only is about 3 minutes max). I would expect that the restart of the app would not take that long but unfortunately it takes op to 1 hour. I guess that the issue is caused by the fact that Streams starts restoring state stores, by first deleting the local state stores. I would have expected that as we persist the state store data on persisted volumes outside of the container that runs on kubernetes, the app would receive the last offset from the broker and only has to continue from that point as the previously consumed data is already there in the state store.
> Restarting our app triggers a restore task:
> {code:java}
> StreamThread-2] Restoring task 4_3's state store twelve-hours-error-score from beginning of the changelog anomaly-timeline-twelve-hours-error-score-changelog-3.{code}
> Streamsconfig:
> {code:java}
> 2019-03-25 09:14:54,352 INFO main org.apache.kafka.common.config.AbstractConfig StreamsConfig values:
> application.id = anomaly-timeline
> application.server = localhost:5000
> bootstrap.servers = [localhost:9095]
> buffered.records.per.partition = 1000
> cache.max.bytes.buffering = 10485760
> client.id = anomaly-timeline-bd6b4a26-9053-4670-9637-ebf4bfe87a96-StreamThread-1-consumer
> commit.interval.ms = 30000
> connections.max.idle.ms = 540000
> default.deserialization.exception.handler = class org.apache.kafka.streams.errors.LogAndFailExceptionHandler
> default.key.serde = class org.apache.kafka.common.serialization.Serdes$ByteArraySerde
> default.production.exception.handler = class org.apache.kafka.streams.errors.DefaultProductionExceptionHandler
> default.timestamp.extractor = class org.apache.kafka.streams.processor.FailOnInvalidTimestamp
> default.value.serde = class org.apache.kafka.common.serialization.Serdes$ByteArraySerde
> metadata.max.age.ms = 300000
> metric.reporters = []
> metrics.num.samples = 2
> metrics.recording.level = INFO
> metrics.sample.window.ms = 30000
> num.standby.replicas = 2
> num.stream.threads = 1
> partition.grouper = class org.apache.kafka.streams.processor.DefaultPartitionGrouper
> poll.ms = 100
> processing.guarantee = at_least_once
> receive.buffer.bytes = 32768
> reconnect.backoff.max.ms = 1000
> reconnect.backoff.ms = 50
> replication.factor = 1
> request.timeout.ms = 40000
> retries = 0
> retry.backoff.ms = 100
> rocksdb.config.setter = null
> security.protocol = PLAINTEXT
> send.buffer.bytes = 131072
> state.cleanup.delay.ms = 600000
> state.dir = /tmp/kafka-streams
> topology.optimization = none
> upgrade.from = null
> windowstore.changelog.additional.retention.ms = 86400000
> {code}
> In addition to figure out what exactly is happening, I've implemented a StateRestoreListener:
> {code:java}
> public class LoggingStateRestoreListener implements StateRestoreListener {
> private static final Logger LOG = LoggerFactory.getLogger(LoggingStateRestoreListener.class);
> private final Map<TopicPartition, Long> totalToRestore = new ConcurrentHashMap<>();
> private final Map<TopicPartition, Long> restoredSoFar = new ConcurrentHashMap<>();
> @Override
> public void onRestoreStart(TopicPartition topicPartition, String store, long start, long end) {
> long toRestore = end - start;
> totalToRestore.put(topicPartition, toRestore);
> LOG.info("Starting restoration for {} on topic-partition {} total to restore {}",
> store, topicPartition, toRestore);
> }
> @Override
> public void onBatchRestored(
> TopicPartition topicPartition,
> String storeName,
> long batchEndOffset,
> long batchCompleted) {
> NumberFormat formatter = new DecimalFormat("#.##");
> long currentProgress = batchCompleted + restoredSoFar.getOrDefault(topicPartition, 0L);
> double percentComplete = (double) currentProgress / totalToRestore.get(topicPartition);
> LOG.info("Completed {} for {}% of total restoration for {} on {}", batchCompleted,
> formatter.format(percentComplete * 100.00), storeName, topicPartition);
> restoredSoFar.put(topicPartition, currentProgress);
> }
> @Override
> public void onRestoreEnd(TopicPartition topicPartition, String store, long totalRestored) {
> LOG.info("Restoration completed for {} on topic-partition {}", store, topicPartition);
> restoredSoFar.put(topicPartition, 0L);
> }
> }
> {code}
> This is invoked for same partition over and over again when starting the app: 2019-03-22 09:35:16,03 INFO atl-737e402a-c5bb-446e-8dad-b81e918d4047-StreamThread-1 c.w.a.a.L Starting restoration for fifteen-min on topic-partition atl-fifteen-min-changelog-2 total to restore 119673932
> I've also enabled Trace logging, not sure if it is of any use:
> {code:java}
> 2019-03-11 15:02:26,476 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StoreChangelogReader stream-thread [anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1] Restored from anomaly-timeline-fifteen-minutes-error-score-changelog-3 to fifteen-minutes-error-score with 1000 records, ending offset is 12114866, next starting position is 12114867
> 2019-03-11 15:02:26,476 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AssignedTasks stream-thread [anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1] stream task changelog partitions that have completed restoring so far: [anomaly-timeline-two-minutes-error-score-changelog-3, anomaly-timeline-twelve-hours-error-score-changelog-0, anomaly-timeline-two-minutes-error-score-changelog-1, anomaly-timeline-one-hour-error-score-changelog-2, anomaly-timeline-fifteen-minutes-error-score-changelog-1, anomaly-timeline-fifteen-minutes-error-score-changelog-0, anomaly-timeline-one-hour-error-score-changelog-0, anomaly-timeline-one-hour-error-score-changelog-1]
> 2019-03-11 15:02:26,476 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AssignedTasks stream-thread [anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1] stream task 1_2 cannot resume processing yet since some of its changelog partitions have not completed restoring: [anomaly-timeline-two-minutes-error-score-changelog-2]
> 2019-03-11 15:02:26,476 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AssignedTasks stream-thread [anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1] stream task 2_3 cannot resume processing yet since some of its changelog partitions have not completed restoring: [anomaly-timeline-fifteen-minutes-error-score-changelog-3]
> 2019-03-11 15:02:26,476 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AssignedTasks stream-thread [anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1] stream task 2_7 cannot resume processing yet since some of its changelog partitions have not completed restoring: [anomaly-timeline-fifteen-minutes-error-score-changelog-7]
> 2019-03-11 15:02:26,476 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamThread stream-thread [anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1] Committing all active tasks [1_1, 2_0, 3_0, 2_1, 1_2, 1_3, 4_0, 3_1, 3_2, 2_3, 2_7] and standby tasks [2_2, 4_1, 1_4, 4_2, 3_3, 2_4, 1_5, 3_4, 1_6, 4_4, 2_6, 4_5, 3_7, 3_8, 4_8] since 504ms has elapsed (commit interval is 500ms)
> 2019-03-11 15:02:26,476 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [1_1] Committing
> 2019-03-11 15:02:26,476 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [1_1] Flushing state and producer
> 2019-03-11 15:02:26,476 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [1_1] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,477 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [1_1] Flushing store two-minutes-error-score
> 2019-03-11 15:02:26,477 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.RecordCollectorImpl task [1_1] Flushing producer
> 2019-03-11 15:02:26,477 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [1_1] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-two-minutes-error-score-changelog-0=112678210, anomaly-timeline-twelve-hours-error-score-changelog-2=4745702, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-6=113137792, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-two-minutes-error-score-changelog-7=113210295, anomaly-timeline-two-minutes-error-score-changelog-5=4464429, anomaly-timeline-twelve-hours-error-score-changelog-7=58001249, anomaly-timeline-twelve-hours-error-score-changelog-5=49765578, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-fifteen-minutes-error-score-changelog-4=46582476, anomaly-timeline-one-hour-error-score-changelog-3=46566107, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-2=57539299, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-scor
> e-changelog-1=121219132, anomaly-timeline-fifteen-minutes-error-score-changelog-8=4826091, anomaly-timeline-one-hour-error-score-changelog-7=112642671, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398, anomaly-timeline-twelve-hours-error-score-changelog-6=26709582}
> 2019-03-11 15:02:26,479 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [2_0] Committing
> 2019-03-11 15:02:26,479 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [2_0] Flushing state and producer
> 2019-03-11 15:02:26,479 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [2_0] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,479 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [2_0] Flushing store fifteen-minutes-error-score
> 2019-03-11 15:02:26,480 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.RecordCollectorImpl task [2_0] Flushing producer
> 2019-03-11 15:02:26,480 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [2_0] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-two-minutes-error-score-changelog-0=112678210, anomaly-timeline-twelve-hours-error-score-changelog-2=4745702, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-6=113137792, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-two-minutes-error-score-changelog-7=113210295, anomaly-timeline-two-minutes-error-score-changelog-5=4464429, anomaly-timeline-twelve-hours-error-score-changelog-7=58001249, anomaly-timeline-twelve-hours-error-score-changelog-5=49765578, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-one-hour-error-score-changelog-3=46566107, anomaly-timeline-fifteen-minutes-error-score-changelog-4=46582476, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-2=57539299, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-scor
> e-changelog-1=121219132, anomaly-timeline-one-hour-error-score-changelog-7=112642671, anomaly-timeline-fifteen-minutes-error-score-changelog-8=4826091, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398, anomaly-timeline-twelve-hours-error-score-changelog-6=26709582}
> 2019-03-11 15:02:26,482 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [3_0] Committing
> 2019-03-11 15:02:26,482 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [3_0] Flushing state and producer
> 2019-03-11 15:02:26,482 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [3_0] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,482 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [3_0] Flushing store one-hour-error-score
> 2019-03-11 15:02:26,482 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.RecordCollectorImpl task [3_0] Flushing producer
> 2019-03-11 15:02:26,482 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [3_0] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-two-minutes-error-score-changelog-0=111734366, anomaly-timeline-twelve-hours-error-score-changelog-2=18521526, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-6=113136853, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-two-minutes-error-score-changelog-7=113207826, anomaly-timeline-two-minutes-error-score-changelog-5=14457271, anomaly-timeline-twelve-hours-error-score-changelog-7=5179040, anomaly-timeline-twelve-hours-error-score-changelog-5=115476239, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-one-hour-error-score-changelog-3=25705510, anomaly-timeline-fifteen-minutes-error-score-changelog-4=25663789, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-2=113452888, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-s
> core-changelog-1=121219132, anomaly-timeline-one-hour-error-score-changelog-7=112077749, anomaly-timeline-fifteen-minutes-error-score-changelog-8=109499390, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398, anomaly-timeline-twelve-hours-error-score-changelog-6=112561985}
> 2019-03-11 15:02:26,484 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [2_1] Committing
> 2019-03-11 15:02:26,484 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [2_1] Flushing state and producer
> 2019-03-11 15:02:26,484 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [2_1] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,484 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [2_1] Flushing store fifteen-minutes-error-score
> 2019-03-11 15:02:26,484 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.RecordCollectorImpl task [2_1] Flushing producer
> 2019-03-11 15:02:26,485 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [2_1] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-two-minutes-error-score-changelog-0=112678210, anomaly-timeline-twelve-hours-error-score-changelog-2=4745702, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-6=113137792, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-two-minutes-error-score-changelog-7=113210295, anomaly-timeline-two-minutes-error-score-changelog-5=4464429, anomaly-timeline-twelve-hours-error-score-changelog-7=58001249, anomaly-timeline-twelve-hours-error-score-changelog-5=49765578, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-one-hour-error-score-changelog-3=46566107, anomaly-timeline-fifteen-minutes-error-score-changelog-4=46582476, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-2=57539299, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-scor
> e-changelog-1=121219132, anomaly-timeline-fifteen-minutes-error-score-changelog-8=4826091, anomaly-timeline-one-hour-error-score-changelog-7=112642671, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398, anomaly-timeline-twelve-hours-error-score-changelog-6=26709582}
> 2019-03-11 15:02:26,487 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [1_3] Committing
> 2019-03-11 15:02:26,487 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [1_3] Flushing state and producer
> 2019-03-11 15:02:26,487 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [1_3] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,487 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [1_3] Flushing store two-minutes-error-score
> 2019-03-11 15:02:26,487 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.RecordCollectorImpl task [1_3] Flushing producer
> 2019-03-11 15:02:26,488 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [1_3] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-6=113136853, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-two-minutes-error-score-changelog-7=113207826, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-score-changelog-1=121219132, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398}
> 2019-03-11 15:02:26,490 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [4_0] Committing
> 2019-03-11 15:02:26,490 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [4_0] Flushing state and producer
> 2019-03-11 15:02:26,490 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [4_0] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,490 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [4_0] Flushing store twelve-hours-error-score
> 2019-03-11 15:02:26,490 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.RecordCollectorImpl task [4_0] Flushing producer
> 2019-03-11 15:02:26,490 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [4_0] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-two-minutes-error-score-changelog-0=112678210, anomaly-timeline-twelve-hours-error-score-changelog-2=4745702, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-6=113137792, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-two-minutes-error-score-changelog-7=113210295, anomaly-timeline-two-minutes-error-score-changelog-5=4464429, anomaly-timeline-twelve-hours-error-score-changelog-7=58001249, anomaly-timeline-twelve-hours-error-score-changelog-5=49765578, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-one-hour-error-score-changelog-3=46566107, anomaly-timeline-fifteen-minutes-error-score-changelog-4=46582476, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-2=57539299, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-scor
> e-changelog-1=121219132, anomaly-timeline-one-hour-error-score-changelog-7=112642671, anomaly-timeline-fifteen-minutes-error-score-changelog-8=4826091, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398, anomaly-timeline-twelve-hours-error-score-changelog-6=26709582}
> 2019-03-11 15:02:26,492 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [3_1] Committing
> 2019-03-11 15:02:26,492 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [3_1] Flushing state and producer
> 2019-03-11 15:02:26,492 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [3_1] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,492 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [3_1] Flushing store one-hour-error-score
> 2019-03-11 15:02:26,492 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.RecordCollectorImpl task [3_1] Flushing producer
> 2019-03-11 15:02:26,493 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [3_1] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-two-minutes-error-score-changelog-0=112678210, anomaly-timeline-twelve-hours-error-score-changelog-2=4745702, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-6=113137792, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-two-minutes-error-score-changelog-7=113210295, anomaly-timeline-two-minutes-error-score-changelog-5=4464429, anomaly-timeline-twelve-hours-error-score-changelog-7=58001249, anomaly-timeline-twelve-hours-error-score-changelog-5=49765578, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-one-hour-error-score-changelog-3=46566107, anomaly-timeline-fifteen-minutes-error-score-changelog-4=46582476, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-2=57539299, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-scor
> e-changelog-1=121219132, anomaly-timeline-one-hour-error-score-changelog-7=112642671, anomaly-timeline-fifteen-minutes-error-score-changelog-8=4826091, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398, anomaly-timeline-twelve-hours-error-score-changelog-6=26709582}
> 2019-03-11 15:02:26,495 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [3_2] Committing
> 2019-03-11 15:02:26,495 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StreamTask task [3_2] Flushing state and producer
> 2019-03-11 15:02:26,495 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [3_2] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,495 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [3_2] Flushing store one-hour-error-score
> 2019-03-11 15:02:26,495 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.RecordCollectorImpl task [3_2] Flushing producer
> 2019-03-11 15:02:26,495 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager task [3_2] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-6=113136853, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-two-minutes-error-score-changelog-7=113207826, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-score-changelog-1=121219132, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398}
> 2019-03-11 15:02:26,497 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [2_2] Committing
> 2019-03-11 15:02:26,497 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [2_2] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,497 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [2_2] Flushing store fifteen-minutes-error-score
> 2019-03-11 15:02:26,497 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [2_2] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-one-hour-error-score-changelog-3=25705510, anomaly-timeline-fifteen-minutes-error-score-changelog-4=25663789, anomaly-timeline-fifteen-minutes-error-score-changelog-2=62148419, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-score-changelog-1=121219132, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398}
> 2019-03-11 15:02:26,500 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [2_2] Updating store offset limit for partition anomaly-timeline-fifteen-minutes-error-score-repartition-2 to 125763438
> 2019-03-11 15:02:26,500 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AbstractTask standby-task [2_2] Updating store offset limits 125763438 for changelog anomaly-timeline-fifteen-minutes-error-score-repartition-2
> 2019-03-11 15:02:26,500 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [4_1] Committing
> 2019-03-11 15:02:26,500 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_1] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,500 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_1] Flushing store twelve-hours-error-score
> 2019-03-11 15:02:26,500 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_1] Writing checkpoint: {anomaly-timeline-two-minutes-error-score-changelog-0=112678210, anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-twelve-hours-error-score-changelog-2=4745702, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-6=113137792, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-two-minutes-error-score-changelog-7=113210295, anomaly-timeline-two-minutes-error-score-changelog-5=4464429, anomaly-timeline-twelve-hours-error-score-changelog-7=58001249, anomaly-timeline-twelve-hours-error-score-changelog-5=49765578, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-fifteen-minutes-error-score-changelog-4=46582476, anomaly-timeline-one-hour-error-score-changelog-3=46566107, anomaly-timeline-fifteen-minutes-error-score-changelog-2=57539299, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-er
> ror-score-changelog-1=121219132, anomaly-timeline-fifteen-minutes-error-score-changelog-8=4826091, anomaly-timeline-one-hour-error-score-changelog-7=112642671, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398, anomaly-timeline-twelve-hours-error-score-changelog-6=26709582}
> 2019-03-11 15:02:26,502 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_1] Updating store offset limit for partition anomaly-timeline-twelve-hours-error-score-repartition-1 to 124006951
> 2019-03-11 15:02:26,502 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AbstractTask standby-task [4_1] Updating store offset limits 124006951 for changelog anomaly-timeline-twelve-hours-error-score-repartition-1
> 2019-03-11 15:02:26,502 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [1_4] Committing
> 2019-03-11 15:02:26,503 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [1_4] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,503 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [1_4] Flushing store two-minutes-error-score
> 2019-03-11 15:02:26,503 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [1_4] Writing checkpoint: {anomaly-timeline-two-minutes-error-score-changelog-0=112678210, anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-twelve-hours-error-score-changelog-2=4745702, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-6=113137792, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-two-minutes-error-score-changelog-7=113210295, anomaly-timeline-two-minutes-error-score-changelog-5=4464429, anomaly-timeline-twelve-hours-error-score-changelog-7=58001249, anomaly-timeline-twelve-hours-error-score-changelog-5=49765578, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-fifteen-minutes-error-score-changelog-4=46582476, anomaly-timeline-one-hour-error-score-changelog-3=46566107, anomaly-timeline-fifteen-minutes-error-score-changelog-2=57539299, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-er
> ror-score-changelog-1=121219132, anomaly-timeline-fifteen-minutes-error-score-changelog-8=4826091, anomaly-timeline-one-hour-error-score-changelog-7=112642671, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398, anomaly-timeline-twelve-hours-error-score-changelog-6=26709582}
> 2019-03-11 15:02:26,505 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [1_4] Updating store offset limit for partition anomaly-timeline-two-minutes-error-score-repartition-4 to 126430132
> 2019-03-11 15:02:26,505 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AbstractTask standby-task [1_4] Updating store offset limits 126430132 for changelog anomaly-timeline-two-minutes-error-score-repartition-4
> 2019-03-11 15:02:26,506 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [4_2] Committing
> 2019-03-11 15:02:26,506 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_2] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,506 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_2] Flushing store twelve-hours-error-score
> 2019-03-11 15:02:26,506 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_2] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-2=9574079, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-1=121219132, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398}
> 2019-03-11 15:02:26,508 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_2] Updating store offset limit for partition anomaly-timeline-twelve-hours-error-score-repartition-2 to 125763449
> 2019-03-11 15:02:26,508 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AbstractTask standby-task [4_2] Updating store offset limits 125763449 for changelog anomaly-timeline-twelve-hours-error-score-repartition-2
> 2019-03-11 15:02:26,508 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [3_3] Committing
> 2019-03-11 15:02:26,508 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_3] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,508 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_3] Flushing store one-hour-error-score
> 2019-03-11 15:02:26,508 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_3] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-two-minutes-error-score-changelog-0=111734366, anomaly-timeline-twelve-hours-error-score-changelog-2=34315444, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-6=113136853, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-two-minutes-error-score-changelog-7=113207826, anomaly-timeline-twelve-hours-error-score-changelog-7=5179040, anomaly-timeline-twelve-hours-error-score-changelog-5=115476239, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-fifteen-minutes-error-score-changelog-4=123200563, anomaly-timeline-one-hour-error-score-changelog-3=51371192, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-2=113452888, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-score-changelog-1=121219132, anomaly-timeline-fifteen-minutes-error-score-changelog-8=109499390, anomaly-timeline-
> fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398, anomaly-timeline-twelve-hours-error-score-changelog-6=112561985}
> 2019-03-11 15:02:26,511 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_3] Updating store offset limit for partition anomaly-timeline-one-hour-error-score-repartition-3 to 118142814
> 2019-03-11 15:02:26,511 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AbstractTask standby-task [3_3] Updating store offset limits 118142814 for changelog anomaly-timeline-one-hour-error-score-repartition-3
> 2019-03-11 15:02:26,511 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [2_4] Committing
> 2019-03-11 15:02:26,511 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [2_4] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,511 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [2_4] Flushing store fifteen-minutes-error-score
> 2019-03-11 15:02:26,511 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [2_4] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-1=121219132, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398}
> 2019-03-11 15:02:26,513 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [2_4] Updating store offset limit for partition anomaly-timeline-fifteen-minutes-error-score-repartition-4 to 126430126
> 2019-03-11 15:02:26,513 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AbstractTask standby-task [2_4] Updating store offset limits 126430126 for changelog anomaly-timeline-fifteen-minutes-error-score-repartition-4
> 2019-03-11 15:02:26,513 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [1_5] Committing
> 2019-03-11 15:02:26,514 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [1_5] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,514 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [1_5] Flushing store two-minutes-error-score
> 2019-03-11 15:02:26,514 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [1_5] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-1=121219132, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-5=9284252, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398}
> 2019-03-11 15:02:26,516 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [1_5] Updating store offset limit for partition anomaly-timeline-two-minutes-error-score-repartition-5 to 118596515
> 2019-03-11 15:02:26,516 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AbstractTask standby-task [1_5] Updating store offset limits 118596515 for changelog anomaly-timeline-two-minutes-error-score-repartition-5
> 2019-03-11 15:02:26,516 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [3_4] Committing
> 2019-03-11 15:02:26,516 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_4] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,517 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_4] Flushing store one-hour-error-score
> 2019-03-11 15:02:26,517 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_4] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-one-hour-error-score-changelog-4=9064711, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-1=121219132, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398}
> 2019-03-11 15:02:26,519 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_4] Updating store offset limit for partition anomaly-timeline-one-hour-error-score-repartition-4 to 126430139
> 2019-03-11 15:02:26,519 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AbstractTask standby-task [3_4] Updating store offset limits 126430139 for changelog anomaly-timeline-one-hour-error-score-repartition-4
> 2019-03-11 15:02:26,519 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [1_6] Committing
> 2019-03-11 15:02:26,519 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [1_6] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,519 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [1_6] Flushing store two-minutes-error-score
> 2019-03-11 15:02:26,519 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [1_6] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124474512, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-1=121219132, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398}
> 2019-03-11 15:02:26,522 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [1_6] Updating store offset limit for partition anomaly-timeline-two-minutes-error-score-repartition-6 to 115826992
> 2019-03-11 15:02:26,522 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AbstractTask standby-task [1_6] Updating store offset limits 115826992 for changelog anomaly-timeline-two-minutes-error-score-repartition-6
> 2019-03-11 15:02:26,522 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [4_4] Committing
> 2019-03-11 15:02:26,522 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_4] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,522 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_4] Flushing store twelve-hours-error-score
> 2019-03-11 15:02:26,522 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_4] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-two-minutes-error-score-changelog-0=112678210, anomaly-timeline-twelve-hours-error-score-changelog-2=4745702, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-4=124386982, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-6=113137792, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-two-minutes-error-score-changelog-7=113210295, anomaly-timeline-two-minutes-error-score-changelog-5=4464429, anomaly-timeline-twelve-hours-error-score-changelog-7=58001249, anomaly-timeline-twelve-hours-error-score-changelog-5=49765578, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-one-hour-error-score-changelog-3=46566107, anomaly-timeline-fifteen-minutes-error-score-changelog-4=46582476, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-2=57539299, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-er
> ror-score-changelog-1=121219132, anomaly-timeline-fifteen-minutes-error-score-changelog-8=4826091, anomaly-timeline-one-hour-error-score-changelog-7=112642671, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398, anomaly-timeline-twelve-hours-error-score-changelog-6=26709582}
> 2019-03-11 15:02:26,525 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_4] Updating store offset limit for partition anomaly-timeline-twelve-hours-error-score-repartition-4 to 126430132
> 2019-03-11 15:02:26,525 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AbstractTask standby-task [4_4] Updating store offset limits 126430132 for changelog anomaly-timeline-twelve-hours-error-score-repartition-4
> 2019-03-11 15:02:26,525 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [2_6] Committing
> 2019-03-11 15:02:26,525 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [2_6] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,525 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [2_6] Flushing store fifteen-minutes-error-score
> 2019-03-11 15:02:26,525 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [2_6] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-1=121219132, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020}
> 2019-03-11 15:02:26,528 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [2_6] Updating store offset limit for partition anomaly-timeline-fifteen-minutes-error-score-repartition-6 to 115827009
> 2019-03-11 15:02:26,528 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AbstractTask standby-task [2_6] Updating store offset limits 115827009 for changelog anomaly-timeline-fifteen-minutes-error-score-repartition-6
> 2019-03-11 15:02:26,528 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [4_5] Committing
> 2019-03-11 15:02:26,528 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_5] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,528 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_5] Flushing store twelve-hours-error-score
> 2019-03-11 15:02:26,528 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_5] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-5=4454556, anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-1=121219132, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398}
> 2019-03-11 15:02:26,530 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [4_5] Updating store offset limit for partition anomaly-timeline-twelve-hours-error-score-repartition-5 to 118596513
> 2019-03-11 15:02:26,530 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AbstractTask standby-task [4_5] Updating store offset limits 118596513 for changelog anomaly-timeline-twelve-hours-error-score-repartition-5
> 2019-03-11 15:02:26,530 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [3_7] Committing
> 2019-03-11 15:02:26,530 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_7] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,531 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_7] Flushing store one-hour-error-score
> 2019-03-11 15:02:26,531 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_7] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-1=121219132, anomaly-timeline-one-hour-error-score-changelog-8=9776082, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398}
> 2019-03-11 15:02:26,533 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_7] Updating store offset limit for partition anomaly-timeline-one-hour-error-score-repartition-7 to 115248401
> 2019-03-11 15:02:26,533 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AbstractTask standby-task [3_7] Updating store offset limits 115248401 for changelog anomaly-timeline-one-hour-error-score-repartition-7
> 2019-03-11 15:02:26,533 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [3_8] Committing
> 2019-03-11 15:02:26,533 DEBUG anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_8] Flushing all stores registered in the state manager
> 2019-03-11 15:02:26,533 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_8] Flushing store one-hour-error-score
> 2019-03-11 15:02:26,533 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_8] Writing checkpoint: {anomaly-timeline-twelve-hours-error-score-changelog-4=123803165, anomaly-timeline-twelve-hours-error-score-changelog-3=5198411, anomaly-timeline-twelve-hours-error-score-changelog-1=121266870, anomaly-timeline-two-minutes-error-score-changelog-3=116262652, anomaly-timeline-twelve-hours-error-score-changelog-0=112141273, anomaly-timeline-two-minutes-error-score-changelog-1=121864699, anomaly-timeline-two-minutes-error-score-changelog-2=33179022, anomaly-timeline-fifteen-minutes-error-score-changelog-5=9496994, anomaly-timeline-one-hour-error-score-changelog-2=122882442, anomaly-timeline-one-hour-error-score-changelog-4=9506063, anomaly-timeline-fifteen-minutes-error-score-changelog-3=12114867, anomaly-timeline-one-hour-error-score-changelog-5=4447988, anomaly-timeline-fifteen-minutes-error-score-changelog-1=122051739, anomaly-timeline-fifteen-minutes-error-score-changelog-0=113124925, anomaly-timeline-one-hour-error-score-changelog-0=112251807, anomaly-timeline-one-hour-error-score-changelog-1=121219132, anomaly-timeline-fifteen-minutes-error-score-changelog-7=6933206, anomaly-timeline-fifteen-minutes-error-score-changelog-6=6288164, anomaly-timeline-two-minutes-error-score-changelog-8=5494256, anomaly-timeline-two-minutes-error-score-changelog-6=5472020, anomaly-timeline-twelve-hours-error-score-changelog-8=4815398}
> 2019-03-11 15:02:26,536 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.ProcessorStateManager standby-task [3_8] Updating store offset limit for partition anomaly-timeline-one-hour-error-score-repartition-8 to 112603023
> 2019-03-11 15:02:26,536 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.AbstractTask standby-task [3_8] Updating store offset limits 112603023 for changelog anomaly-timeline-one-hour-error-score-repartition-8
> 2019-03-11 15:02:26,536 TRACE anomaly-timeline-6f6760cc-c804-4af6-b5a2-6fa93fd6ed02-StreamThread-1 org.apache.kafka.streams.processor.internals.StandbyTask standby-task [4_8] Committing{code}
> I tried to debug it locally and remote, but can't reproduce it locally. I did have a chat with Matthias J. Stax who suggested to increase the state.cleanup.delay.ms but that didn't have an impact. Also because the downtime between restarts is just a couple of minutes. One thing I did notice is that just restarting one container at a time, did improve the time for the state store to become active. Unfortunately only for one instance.
> One last thing to note:
> I did add the following code in order to override the default cleanUp on start and stop where the stores by default are deleted:
> {code:java}
> @Bean public CleanupConfig cleanupConfig() { return new CleanupConfig(false, false); } {code}
> So after this change, the local state dirs are left untouched when starting or stopping an instance.
> *Configuration*
> Confluent kafka 5.1.0 -> Kafka 2.1.0
> Cluster of 3 brokers with 3 zookeepers
> One topic "error-score" with 9 partititions
> Streaming application using Spring Cloud Stream Kafka Binder: Fishtown.RELEASE -> Kafka streams 2.0.1
> Configuration of stream app:
> {code:java}
> server.port = 5000
> # Retention period is two months
> windowstore.retention.ms = 5259600000
> spring.jmx.enabled=false
> spring.kafka.streams.application-id=anomaly-timeline
> spring.kafka.streams.state-dir=./state-store
> spring.cloud.stream.client-id=anomaly-timeline
> spring.cloud.stream.kafka.streams.default.consumer.application-id=anomaly-timeline
> spring.cloud.stream.kafka.streams.default.consumer.configuration.auto.offset.reset=earliest
> spring.cloud.stream.kafka.streams.binder.configuration.commit.interval.ms=500
> spring.cloud.stream.kafka.streams.binder.configuration.cache.max.bytes.buffering=2097152
> spring.cloud.stream.kafka.streams.binder.configuration.num.stream.threads=1
> spring.cloud.stream.kafka.streams.binder.configuration.num.standby.replicas=2
> # used to broadcast the current hosts url in order to participate in interactive queries
> spring.cloud.stream.kafka.streams.binder.configuration.application.server=localhost:5000
> spring.cloud.stream.kafka.streams.binder.configuration.default.key.serde=org.apache.kafka.common.serialization.Serdes$StringSerde
> # use the following timestamp extractor to deal with late arriving events
> spring.cloud.stream.kafka.streams.binder.configuration.default.timestamp.extractor=ErrorScoreTimestampExtractor
> spring.cloud.stream.kafka.streams.binder.configuration.default.value.serde=ErrorScoreSerde
> # Spring cloud kafka defaults
> spring.cloud.stream.kafka.streams.binder.brokers=localhost:9095
> spring.cloud.stream.kafka.binder.auto-create-topics=false
> # error-score-in binding
> spring.cloud.stream.bindings.error-score-in.destination=error-score
> spring.cloud.stream.bindings.error-score-in.consumer.header-mode=raw
> spring.cloud.stream.bindings.error-score-in.group=anomaly-timeline
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)