You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Suresh Rukmangathan <su...@gmail.com> on 2022/08/23 10:08:46 UTC
UnsupportedOperationException: this should not happen: timestamp() is not supported in standby tasks.
Hi,
We have a Kafka streams application with "num.standby.replicas" = 1 and
with replication-factor of 2 for that topic. The application is crashing
with the following stack trace.
java.lang.UnsupportedOperationException: this should not happen:
timestamp() is not supported in standby tasks.\n\n\tat
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.throwUnsupportedOperationExceptionIfStandby(ProcessorContextImpl.java:352)\n\n\tat
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.timestamp(ProcessorContextImpl.java:328)\n\n\tat
org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.log(ChangeLoggingKeyValueBytesStore.java:136)\n\n\tat
org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:78)\n\n\tat
org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:32)\n\n\tat
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$put$4(MeteredKeyValueStore.java:197)\n\n\tat
org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)\n\n\tat
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.put(MeteredKeyValueStore.java:197)\n\n\tat
Key Kafka streams application configuration details are as below:-
{replication.factor=1, num.standby.replicas=1, topology.optimization=all,
producer.partitioner.class=com.hpe.gravity.gravitycommon.utils.Murmur3Partitioner,
max.request.size=1048576, auto.offset.reset=earliest}
Is this a Kafka library issue or something wrong with the application
configuration?
thanks,
-sureshr
Re: UnsupportedOperationException: this should not happen: timestamp() is not supported in standby tasks.
Posted by Suresh Rukmangathan <su...@gmail.com>.
Hi John,
Please find the full stack trace with 3 frames from the app removed that
are process/state-store related functions.
{"log":"2022-08-17 15:21:32,890 [ERROR] Util: Failed saving
key=cx_dev_sys:c3d376b6e62f11ecbf8256eb17370719:SG18KW70HN to stateStore
with Exception={}\n\njava.lang.UnsupportedOperationException: this should
not happen: timestamp() is not supported in standby tasks.\n\n\tat
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.throwUnsupportedOperationExceptionIfStandby(ProcessorContextImpl.java:352)\n\n\tat
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.timestamp(ProcessorContextImpl.java:328)\n\n\tat
org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.log(ChangeLoggingKeyValueBytesStore.java:136)\n\n\tat
org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:78)\n\n\tat
org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:32)\n\n\tat
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$put$4(MeteredKeyValueStore.java:197)\n\n\tat
org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)\n\n\tat
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.put(MeteredKeyValueStore.java:197)\n\n\tat
org.apache.kafka.streams.processor.internals.AbstractReadWriteDecorator$KeyValueStoreReadWriteDecorator.put(AbstractReadWriteDecorator.java:120)\n\n\tat
// app-calls to process & save to state store - 3 frames
org.apache.kafka.streams.processor.internals.ProcessorNode.lambda$process$2(ProcessorNode.java:181)\n\n\tat
org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)\n\n\tat
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:181)\n\n\tat
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:273)\n\n\tat
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:252)\n\n\tat
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:219)\n\n\tat
org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:86)\n\n\tat
org.apache.kafka.streams.processor.internals.StreamTask.lambda$process$1(StreamTask.java:703)\n\n\tat
org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)\n\n\tat
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:703)\n\n\tat
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1105)\n\n\tat
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:647)\n\n\tat
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)\n\n\tat
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:512)\n"
If replication factor = 1 and standby replicas=1, is that an issue? Do we
expect that the replication factor should be at least n+1, if standby
replicas=1 (or) there is no relationship?
Another data point is that once I made the standby replicas to 0, the crash
stopped (or) if I run only one pod (one application instance), even then
the crash is not seen.
So, is there something that is not correct in the way we have configured
the Kafka Streams app and/or Kafka is not handling the standby task
correctly (like calling the put from the standby context, when it is not
supposed to)? Is there a way to debug this better by enabling some logs
from Kafka (like set some env var to add more debug logs) to isolate the
issue?
thanks,
-sureshr
On Tue, Aug 23, 2022 at 3:38 PM Suresh Rukmangathan <su...@gmail.com>
wrote:
> Hi,
>
> We have a Kafka streams application with "num.standby.replicas" = 1 and
> with replication-factor of 2 for that topic. The application is crashing
> with the following stack trace.
>
> java.lang.UnsupportedOperationException: this should not happen:
> timestamp() is not supported in standby tasks.\n\n\tat
>
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.throwUnsupportedOperationExceptionIfStandby(ProcessorContextImpl.java:352)\n\n\tat
>
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.timestamp(ProcessorContextImpl.java:328)\n\n\tat
>
> org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.log(ChangeLoggingKeyValueBytesStore.java:136)\n\n\tat
>
> org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:78)\n\n\tat
>
> org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:32)\n\n\tat
>
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$put$4(MeteredKeyValueStore.java:197)\n\n\tat
>
> org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)\n\n\tat
>
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore.put(MeteredKeyValueStore.java:197)\n\n\tat
>
> Key Kafka streams application configuration details are as below:-
>
> {replication.factor=1, num.standby.replicas=1, topology.optimization=all,
> producer.partitioner.class=com.hpe.gravity.gravitycommon.utils.Murmur3Partitioner,
> max.request.size=1048576, auto.offset.reset=earliest}
>
> Is this a Kafka library issue or something wrong with the application
> configuration?
>
> thanks,
> -sureshr
>
>
Re: UnsupportedOperationException: this should not happen: timestamp() is not supported in standby tasks.
Posted by John Roesler <vv...@apache.org>.
Hi Suresh,
Sorry for the trouble!
Are you able to provide the rest of the stack trace?
It shouldn’t be possible to call put() on a store in a standby task, so we need to see the stack frames that show what is calling it.
Thanks,
John
On Tue, Aug 23, 2022, at 05:08, Suresh Rukmangathan wrote:
> Hi,
>
> We have a Kafka streams application with "num.standby.replicas" = 1 and
> with replication-factor of 2 for that topic. The application is crashing
> with the following stack trace.
>
> java.lang.UnsupportedOperationException: this should not happen:
> timestamp() is not supported in standby tasks.\n\n\tat
>
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.throwUnsupportedOperationExceptionIfStandby(ProcessorContextImpl.java:352)\n\n\tat
>
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.timestamp(ProcessorContextImpl.java:328)\n\n\tat
>
> org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.log(ChangeLoggingKeyValueBytesStore.java:136)\n\n\tat
>
> org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:78)\n\n\tat
>
> org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:32)\n\n\tat
>
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$put$4(MeteredKeyValueStore.java:197)\n\n\tat
>
> org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)\n\n\tat
>
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore.put(MeteredKeyValueStore.java:197)\n\n\tat
>
> Key Kafka streams application configuration details are as below:-
>
> {replication.factor=1, num.standby.replicas=1,
> topology.optimization=all,
>
> producer.partitioner.class=com.hpe.gravity.gravitycommon.utils.Murmur3Partitioner,
> max.request.size=1048576, auto.offset.reset=earliest}
>
> Is this a Kafka library issue or something wrong with the application
> configuration?
>
> thanks,
> -sureshr