You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Ray McDermott (Jira)" <ji...@apache.org> on 2026/04/01 08:53:00 UTC
[jira] [Commented] (KAFKA-19638) NPE in `Processor#init()` accessing state store
[ https://issues.apache.org/jira/browse/KAFKA-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18070273#comment-18070273 ]
Ray McDermott commented on KAFKA-19638:
---------------------------------------
There is no violation of the API: we are doing the same thing as the examples - grabbing the store from the context and using it in the process calls. We use an atom rather than a class variable, that's the only difference and it worked before ... with `transformFlatMap`.
This is what the Processor looks like:
{{(defn processor-with-store}}
{{ "Call the given process-fn with the state store and the event k v.}}
{{ Call the init-fn with the processor context."}}
{{ [store-name process-fn init-fn]}}
{{ (let [state-store (atom nil)}}
{{ context (atom nil)]}}
{{ (reify ProcessorSupplier}}
{{ (get [_]}}
{{ (reify Processor}}
{{ (init [_ processor-context]}}
{{ (reset! context processor-context)}}
{{ (reset! state-store (.getStateStore processor-context store-name))}}
{{ (init-fn processor-context))}}
{{ (process [_ record]}}
{{ (process-fn @state-store (.key record) (.value record))))))))}}
An example injected `process-fn` looks like this:
{{(defn usage-persistence!}}
{{ [registry store k new-usage]}}
{{ (->> (other-usage-persistence-effect k new-usage)}}
{{ (store/apply-db-effect store)))}}
{{Which means that it first calls}}
{{(defn other-usage-persistence-effect}}
{{ [k new-usage]}}
{{ (let [store-key (usage-key->store-key k new-usage)]}}
{{ [:put store-key new-usage]))}}
{{and then uses the result as the final argument to:}}
{{(defn apply-db-effect}}
{{ "Put or delete according to the vector of command and args.}}
{{ - For put, send [:put k v]}}
{{ - For delete, send [:delete k].}}
{{ - To delete a range of keys send [:delete-range k-prefix] where}}
{{ all keys matching k-prefix will be deleted."}}
{{ [^KeyValueStore store [cmd k v]]}}
{{ (condp = cmd}}
{{ :put (.put store k v)}}
{{ :delete (.delete store k)}}
{{ :delete-range (doseq [k' (byte-ordered-prefix-scan store k iterator->keys)]}}
{{ (.delete store k'))))}}
{{On the first put from any input topic we get the NPE.}}
{{We have a topology that has an empty input topic that we use to generate timed events. This works without any issues and can write to the store with NPEs.}}
> NPE in `Processor#init()` accessing state store
> -----------------------------------------------
>
> Key: KAFKA-19638
> URL: https://issues.apache.org/jira/browse/KAFKA-19638
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.2.0
> Reporter: Matthias J. Sax
> Assignee: Eduwer Camacaro
> Priority: Blocker
> Fix For: 4.2.0, 4.3.0, 4.2.1
>
>
> As reported on the dev mailing list, we introduced a regression bug via https://issues.apache.org/jira/browse/KAFKA-13722 in 4.1 branch. We did revert the commit ([https://github.com/apache/kafka/commit/f13a22af0b3a48a4ca1bf2ece5b58f31e3b26b7d]) for 4.1 release, and want to fix-forward for 4.2 release.
> Stacktrace:
> {code:java}
> 15:29:05 ERROR [STREAMS] KafkaStreams - stream-client [app1] Encountered the following exception during processing and the registered exception handler opted to SHUTDOWN_CLIENT. The streams client is going to shut down now.
> org.apache.kafka.streams.errors.StreamsException: failed to initialize processor random-value-processor
> at org.apache.kafka.streams.processor.internals.ProcessorNode.init(ProcessorNode.java:132) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> at org.apache.kafka.streams.processor.internals.ProcessorNode.init(ProcessorNode.java:141) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> at org.apache.kafka.streams.processor.internals.StreamTask.initializeTopology(StreamTask.java:1109) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> at org.apache.kafka.streams.processor.internals.StreamTask.completeRestoration(StreamTask.java:297) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> at org.apache.kafka.streams.processor.internals.TaskManager.tryToCompleteRestoration(TaskManager.java:955) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> at org.apache.kafka.streams.processor.internals.StreamThread.initializeAndRestorePhase(StreamThread.java:1417) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> at org.apache.kafka.streams.processor.internals.StreamThread.runOnceWithoutProcessingThreads(StreamThread.java:1219) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:934) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:894) [kafka-streams-4.2.0-SNAPSHOT.jar:?]
> Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.kafka.streams.processor.internals.ProcessorRecordContext.timestamp()" because the return value of "org.apache.kafka.streams.processor.internals.InternalProcessorContext.recordContext()" is null
> at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:69) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:32) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$put$6(MeteredKeyValueStore.java:303) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> at org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:901) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.put(MeteredKeyValueStore.java:303) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> at org.apache.kafka.streams.processor.internals.AbstractReadWriteDecorator$KeyValueStoreReadWriteDecorator.put(AbstractReadWriteDecorator.java:123) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> at io.littlehorse.simulations.stateful.app.RandomValueProcessor.init(RandomValueProcessor.java:21) ~[kafka-streams-stateful-unspecified.jar:?]
> at org.apache.kafka.streams.processor.internals.ProcessorNode.init(ProcessorNode.java:124) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
> ... 8 more {code}
> Thanks [~eduwerc] for reporting the issue.
> We clearly have a testing gap, not trying to use a state store within `Processor#init()`. – We need to close this gap.
> However, there is also the question if using zero as surrogate ts for this case (as the old code does), is a good solution or not? – We could try to use stream-time, but for the very first startup of an application, we also do not have stream-time established yet, so we kinda push the can down the road.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)