You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Ray McDermott (Jira)" <ji...@apache.org> on 2026/04/01 08:53:00 UTC

[jira] [Commented] (KAFKA-19638) NPE in `Processor#init()` accessing state store

    [ https://issues.apache.org/jira/browse/KAFKA-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18070273#comment-18070273 ] 

Ray McDermott commented on KAFKA-19638:
---------------------------------------

There is no violation of the API: we are doing the same thing as the examples - grabbing the store from the context and using it in the process calls. We use an atom rather than a class variable, that's the only difference and it worked before ... with `transformFlatMap`.

This is what the Processor looks like:

{{(defn processor-with-store}}
{{  "Call the given process-fn with the state store and the event k v.}}
{{  Call the init-fn with the processor context."}}
{{  [store-name process-fn init-fn]}}
{{  (let [state-store (atom nil)}}
{{        context (atom nil)]}}
{{    (reify ProcessorSupplier}}
{{      (get [_]}}
{{        (reify Processor}}
{{          (init [_ processor-context]}}
{{            (reset! context processor-context)}}
{{            (reset! state-store (.getStateStore processor-context store-name))}}
{{            (init-fn processor-context))}}
{{          (process [_ record]}}
{{            (process-fn @state-store (.key record) (.value record))))))))}}

 

An example injected `process-fn` looks like this:

{{(defn usage-persistence!}}
{{  [registry store k new-usage]}}
{{  (->> (other-usage-persistence-effect k new-usage)}}
{{       (store/apply-db-effect store)))}}

 

{{Which means that it first calls}}

{{(defn other-usage-persistence-effect}}
{{  [k new-usage]}}
{{  (let [store-key (usage-key->store-key k new-usage)]}}
{{    [:put store-key new-usage]))}}

 

{{and then uses the result as the final argument to:}}

{{(defn apply-db-effect}}
{{  "Put or delete according to the vector of command and args.}}
{{  - For put, send [:put k v]}}
{{  - For delete, send [:delete k].}}
{{  - To delete a range of keys send [:delete-range k-prefix] where}}
{{  all keys matching k-prefix will be deleted."}}
{{  [^KeyValueStore store [cmd k v]]}}
{{  (condp = cmd}}
{{    :put (.put store k v)}}
{{    :delete (.delete store k)}}
{{    :delete-range (doseq [k' (byte-ordered-prefix-scan store k iterator->keys)]}}
{{                    (.delete store k'))))}}

 

{{On the first put from any input topic we get the NPE.}}

 

{{We have a topology that has an empty input topic that we use to generate timed events. This works without any issues and can write to the store with NPEs.}}

 

> NPE in `Processor#init()` accessing state store
> -----------------------------------------------
>
>                 Key: KAFKA-19638
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19638
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 4.2.0
>            Reporter: Matthias J. Sax
>            Assignee: Eduwer Camacaro
>            Priority: Blocker
>             Fix For: 4.2.0, 4.3.0, 4.2.1
>
>
> As reported on the dev mailing list, we introduced a regression bug via https://issues.apache.org/jira/browse/KAFKA-13722 in 4.1 branch. We did revert the commit ([https://github.com/apache/kafka/commit/f13a22af0b3a48a4ca1bf2ece5b58f31e3b26b7d]) for 4.1 release, and want to fix-forward for 4.2 release.
> Stacktrace:
> {code:java}
> 15:29:05 ERROR [STREAMS] KafkaStreams - stream-client [app1] Encountered the following exception during processing and the registered exception handler opted to SHUTDOWN_CLIENT. The streams client is going to shut down now.
> org.apache.kafka.streams.errors.StreamsException: failed to initialize processor random-value-processor
>         at org.apache.kafka.streams.processor.internals.ProcessorNode.init(ProcessorNode.java:132) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         at org.apache.kafka.streams.processor.internals.ProcessorNode.init(ProcessorNode.java:141) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         at org.apache.kafka.streams.processor.internals.StreamTask.initializeTopology(StreamTask.java:1109) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         at org.apache.kafka.streams.processor.internals.StreamTask.completeRestoration(StreamTask.java:297) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         at org.apache.kafka.streams.processor.internals.TaskManager.tryToCompleteRestoration(TaskManager.java:955) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         at org.apache.kafka.streams.processor.internals.StreamThread.initializeAndRestorePhase(StreamThread.java:1417) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         at org.apache.kafka.streams.processor.internals.StreamThread.runOnceWithoutProcessingThreads(StreamThread.java:1219) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:934) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:894) [kafka-streams-4.2.0-SNAPSHOT.jar:?]
> Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.kafka.streams.processor.internals.ProcessorRecordContext.timestamp()" because the return value of "org.apache.kafka.streams.processor.internals.InternalProcessorContext.recordContext()" is null
>         at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:69) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:32) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$put$6(MeteredKeyValueStore.java:303) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         at org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:901) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.put(MeteredKeyValueStore.java:303) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         at org.apache.kafka.streams.processor.internals.AbstractReadWriteDecorator$KeyValueStoreReadWriteDecorator.put(AbstractReadWriteDecorator.java:123) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         at io.littlehorse.simulations.stateful.app.RandomValueProcessor.init(RandomValueProcessor.java:21) ~[kafka-streams-stateful-unspecified.jar:?]
>         at org.apache.kafka.streams.processor.internals.ProcessorNode.init(ProcessorNode.java:124) ~[kafka-streams-4.2.0-SNAPSHOT.jar:?]
>         ... 8 more {code}
> Thanks [~eduwerc] for reporting the issue.
> We clearly have a testing gap, not trying to use a state store within `Processor#init()`. – We need to close this gap.
> However, there is also the question if using zero as surrogate ts for this case (as the old code does), is a good solution or not? – We could try to use stream-time, but for the very first startup of an application, we also do not have stream-time established yet, so we kinda push the can down the road.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)