You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Bart Vercammen <ba...@cloutrix.com> on 2017/08/08 10:21:47 UTC
[kafka streams] 'null' values in state stores
Hi,
I noticed the following:
When a kafka streams application starts, it will restore its state in its
state-stores (from the log-compacted kafka topic). All good so far, but I
noticed that the 'deleted' entries are actually read in into the store as
'key' with value:`null`
Is this expected behaviour? I would assume that 'null' values are ignored
when restoring the state as this is exactly how the entries are deleted on
the log-compacted kafka-topic.
When the compaction has run on the kafka topic, all is fine, but when the
segment is not compacted yet, these null values are read in.
Greets,
Bart
Re: [kafka streams] 'null' values in state stores
Posted by Bart Vercammen <ba...@cloutrix.com>.
Hi Guy,
Indeed, I referenced the wrong source-code, sorry about that ;-)
I created KAFKA-5717 for this.
Thanks
Bart
On Tue, Aug 8, 2017 at 8:08 PM, Damian Guy <da...@gmail.com> wrote:
> The change logger is not used during restoration of the in-memory-store.
> Restoration is handled
> https://github.com/apache/kafka/blob/0.11.0/streams/src/
> main/java/org/apache/kafka/streams/state/internals/
> InMemoryKeyValueStore.java#L79
>
> But, even then it is just putting `null` when it should be deleting it.
> Feel free to raise a JIRA
> Thanks,
> Damian
>
> On Tue, 8 Aug 2017 at 12:09 Bart Vercammen <ba...@cloutrix.com> wrote:
>
> > That's RocksDB .. I'm using in-memory stores ...
> > here:
> >
> > https://github.com/apache/kafka/blob/0.11.0/streams/src/
> main/java/org/apache/kafka/streams/state/internals/
> ChangeLoggingKeyValueBytesStore.java#L56
> > the 'null' is not checked ...
> >
> > On Tue, Aug 8, 2017 at 12:52 PM, Damian Guy <da...@gmail.com>
> wrote:
> >
> > > Hi,
> > > The null values are treated as deletes when they are written to the
> > store.
> > > You can see here:
> > > https://github.com/apache/kafka/blob/0.11.0/streams/src/
> > > main/java/org/apache/kafka/streams/state/internals/
> RocksDBStore.java#L261
> > >
> > > On Tue, 8 Aug 2017 at 11:22 Bart Vercammen <ba...@cloutrix.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I noticed the following:
> > > > When a kafka streams application starts, it will restore its state in
> > its
> > > > state-stores (from the log-compacted kafka topic). All good so far,
> > but
> > > I
> > > > noticed that the 'deleted' entries are actually read in into the
> store
> > as
> > > > 'key' with value:`null`
> > > >
> > > > Is this expected behaviour? I would assume that 'null' values are
> > > ignored
> > > > when restoring the state as this is exactly how the entries are
> deleted
> > > on
> > > > the log-compacted kafka-topic.
> > > >
> > > > When the compaction has run on the kafka topic, all is fine, but when
> > the
> > > > segment is not compacted yet, these null values are read in.
> > > >
> > > > Greets,
> > > > Bart
> > > >
> > >
>
Re: [kafka streams] 'null' values in state stores
Posted by Damian Guy <da...@gmail.com>.
The change logger is not used during restoration of the in-memory-store.
Restoration is handled
https://github.com/apache/kafka/blob/0.11.0/streams/src/main/java/org/apache/kafka/streams/state/internals/InMemoryKeyValueStore.java#L79
But, even then it is just putting `null` when it should be deleting it.
Feel free to raise a JIRA
Thanks,
Damian
On Tue, 8 Aug 2017 at 12:09 Bart Vercammen <ba...@cloutrix.com> wrote:
> That's RocksDB .. I'm using in-memory stores ...
> here:
>
> https://github.com/apache/kafka/blob/0.11.0/streams/src/main/java/org/apache/kafka/streams/state/internals/ChangeLoggingKeyValueBytesStore.java#L56
> the 'null' is not checked ...
>
> On Tue, Aug 8, 2017 at 12:52 PM, Damian Guy <da...@gmail.com> wrote:
>
> > Hi,
> > The null values are treated as deletes when they are written to the
> store.
> > You can see here:
> > https://github.com/apache/kafka/blob/0.11.0/streams/src/
> > main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L261
> >
> > On Tue, 8 Aug 2017 at 11:22 Bart Vercammen <ba...@cloutrix.com> wrote:
> >
> > > Hi,
> > >
> > > I noticed the following:
> > > When a kafka streams application starts, it will restore its state in
> its
> > > state-stores (from the log-compacted kafka topic). All good so far,
> but
> > I
> > > noticed that the 'deleted' entries are actually read in into the store
> as
> > > 'key' with value:`null`
> > >
> > > Is this expected behaviour? I would assume that 'null' values are
> > ignored
> > > when restoring the state as this is exactly how the entries are deleted
> > on
> > > the log-compacted kafka-topic.
> > >
> > > When the compaction has run on the kafka topic, all is fine, but when
> the
> > > segment is not compacted yet, these null values are read in.
> > >
> > > Greets,
> > > Bart
> > >
> >
>
>
>
> --
> Mvg,
> Bart Vercammen
>
>
> clouTrix BVBA
> +32 486 69 17 68 <+32%20486%2069%2017%2068>
> info@cloutrix.com
>
Re: [kafka streams] 'null' values in state stores
Posted by Bart Vercammen <ba...@cloutrix.com>.
That's RocksDB .. I'm using in-memory stores ...
here:
https://github.com/apache/kafka/blob/0.11.0/streams/src/main/java/org/apache/kafka/streams/state/internals/ChangeLoggingKeyValueBytesStore.java#L56
the 'null' is not checked ...
On Tue, Aug 8, 2017 at 12:52 PM, Damian Guy <da...@gmail.com> wrote:
> Hi,
> The null values are treated as deletes when they are written to the store.
> You can see here:
> https://github.com/apache/kafka/blob/0.11.0/streams/src/
> main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L261
>
> On Tue, 8 Aug 2017 at 11:22 Bart Vercammen <ba...@cloutrix.com> wrote:
>
> > Hi,
> >
> > I noticed the following:
> > When a kafka streams application starts, it will restore its state in its
> > state-stores (from the log-compacted kafka topic). All good so far, but
> I
> > noticed that the 'deleted' entries are actually read in into the store as
> > 'key' with value:`null`
> >
> > Is this expected behaviour? I would assume that 'null' values are
> ignored
> > when restoring the state as this is exactly how the entries are deleted
> on
> > the log-compacted kafka-topic.
> >
> > When the compaction has run on the kafka topic, all is fine, but when the
> > segment is not compacted yet, these null values are read in.
> >
> > Greets,
> > Bart
> >
>
--
Mvg,
Bart Vercammen
clouTrix BVBA
+32 486 69 17 68
info@cloutrix.com
Re: [kafka streams] 'null' values in state stores
Posted by Damian Guy <da...@gmail.com>.
Hi,
The null values are treated as deletes when they are written to the store.
You can see here:
https://github.com/apache/kafka/blob/0.11.0/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L261
On Tue, 8 Aug 2017 at 11:22 Bart Vercammen <ba...@cloutrix.com> wrote:
> Hi,
>
> I noticed the following:
> When a kafka streams application starts, it will restore its state in its
> state-stores (from the log-compacted kafka topic). All good so far, but I
> noticed that the 'deleted' entries are actually read in into the store as
> 'key' with value:`null`
>
> Is this expected behaviour? I would assume that 'null' values are ignored
> when restoring the state as this is exactly how the entries are deleted on
> the log-compacted kafka-topic.
>
> When the compaction has run on the kafka topic, all is fine, but when the
> segment is not compacted yet, these null values are read in.
>
> Greets,
> Bart
>