Session Window emitting null values when converted to stream?

[Liam Clarke <li...@adscale.co.nz>]

Hi all,

Using

I am using a session window of 1 minute to detect and prevent duplicated
events within the window using reduce() to preserve the first seen value
and accumulate how often duplicates were seen, I'm then converting the
windowed KTable into a stream to continue processing events.

However, later in the stream, I've been hitting NPEs because after the
window expiry, the stream is emitting a previously seen key with a null
value when I'd assumed any values in the stream would be non-null. Is this
an idiom expressing that the window for that key has expired and as such
needs to be explicitly handled?

I'm worried that my filtering those values from the ongoing stream is
hiding a problem further upstream, but if it's working as designed, then
sweet as.

Kind regards,

Liam Clarke

Re: Session Window emitting null values when converted to stream?

[Liam Clarke <li...@adscale.co.nz>]

Thanks Matthias,

That's far cleaner :)

Cheers,

Liam Clarke

On Mon, Dec 3, 2018 at 4:59 PM Matthias J. Sax <ma...@confluent.io>
wrote:

> The nulls are expected.
>
> It's not about expired session windows though: sessions window are
> stored as `<(key,start-timestamp,end-timestamp), value>`. If the window
> boundaries changed due to new incoming events (or maybe a merge of two
> windows due to late arriving records), the window is updated via a
> delete of the old window and an create of the new window (because it's
> two different keys for old and new window).
>
> Thus, using session window does not seem to be the best approach for
> your problem. Check out this example on how to do event de-duplication:
>
>
> https://github.com/confluentinc/kafka-streams-examples/blob/5.0.1-post/src/test/java/io/confluent/examples/streams/EventDeduplicationLambdaIntegrationTest.java
>
> Hope this helps.
>
>
> -Matthias
>
>
> On 12/2/18 4:59 PM, Liam Clarke wrote:
> > Hi all,
> >
> > Using
> >
> > I am using a session window of 1 minute to detect and prevent duplicated
> > events within the window using reduce() to preserve the first seen value
> > and accumulate how often duplicates were seen, I'm then converting the
> > windowed KTable into a stream to continue processing events.
> >
> > However, later in the stream, I've been hitting NPEs because after the
> > window expiry, the stream is emitting a previously seen key with a null
> > value when I'd assumed any values in the stream would be non-null. Is
> this
> > an idiom expressing that the window for that key has expired and as such
> > needs to be explicitly handled?
> >
> > I'm worried that my filtering those values from the ongoing stream is
> > hiding a problem further upstream, but if it's working as designed, then
> > sweet as.
> >
> > Kind regards,
> >
> > Liam Clarke
> >
>
>

Re: Session Window emitting null values when converted to stream?

["Matthias J. Sax" <ma...@confluent.io>]

The nulls are expected.

It's not about expired session windows though: sessions window are
stored as `<(key,start-timestamp,end-timestamp), value>`. If the window
boundaries changed due to new incoming events (or maybe a merge of two
windows due to late arriving records), the window is updated via a
delete of the old window and an create of the new window (because it's
two different keys for old and new window).

Thus, using session window does not seem to be the best approach for
your problem. Check out this example on how to do event de-duplication:

https://github.com/confluentinc/kafka-streams-examples/blob/5.0.1-post/src/test/java/io/confluent/examples/streams/EventDeduplicationLambdaIntegrationTest.java

Hope this helps.


-Matthias


On 12/2/18 4:59 PM, Liam Clarke wrote:
> Hi all,
> 
> Using
> 
> I am using a session window of 1 minute to detect and prevent duplicated
> events within the window using reduce() to preserve the first seen value
> and accumulate how often duplicates were seen, I'm then converting the
> windowed KTable into a stream to continue processing events.
> 
> However, later in the stream, I've been hitting NPEs because after the
> window expiry, the stream is emitting a previously seen key with a null
> value when I'd assumed any values in the stream would be non-null. Is this
> an idiom expressing that the window for that key has expired and as such
> needs to be explicitly handled?
> 
> I'm worried that my filtering those values from the ongoing stream is
> hiding a problem further upstream, but if it's working as designed, then
> sweet as.
> 
> Kind regards,
> 
> Liam Clarke
>