update of GlobalKTable

[Imre Gábor <Im...@aut.bme.hu>]

Hi,

I am new to Kafka and observed a strange behavior. (I am using Kafka through the spring-kafka library)

I think I misunderstand something about the "liveness" of the data in a global table.
I thought if I define and materialize a GlobalKTable at the startup of my application, and messages arrive later to the underlying topic, the data in the store are automatically updated.
i.e. if I have a method that queries the store of the table, it will see the most up-to-date data. (With a bit of latency of course.)
However, it seems not to be the case:


1.       My application defines a GlobalKTable over a topic, materialized with a name and serializers passed.

2.       I have a business logic method that calls store() method with the name of this global table, and tries to get a value by key.

3.       Next, I wrote a test case, that starts the application, and I send a message with a producer to the appropriate topic, and I expect I can get it by its key from the global table's store.

However, it returns null. I can see the message in the topic so it is surely delivered.

4.       If I iterate through the store via store.all() before calling store.get(key), the KeyValue is there with the appropriate key! However, store.get(key) still returns null.

5.       I implemented hashCode and equals for my key class appropriately, so it cannot be the problem

6.       Also, it is a not about a small latency, because if I wait about a minute, still cannot get the data.

7.       If I modify my test case so that I do not send a test message (because there is one in the topic already), and rerun the test (which starts the application and defined the global table),

the value is correctly found by key. (I guess the message from the topic is read when the table is created.)

Do you have any idea what am I missing?

Best regards

Gabor

Re: update of GlobalKTable

["Matthias J. Sax" <ma...@confluent.io>]

Your expectation and understanding how GlobalKTables work is correct.

If you can reproduce the issue in an IDE, set a breakpoint in
`GlobalStreamThread#pollAndUpdate()` to see if data is fetched from the
brokers.


-Matthias


On 6/26/19 7:09 AM, Imre Gábor wrote:
> Hi,
> 
> I am new to Kafka and observed a strange behavior. (I am using Kafka through the spring-kafka library)
> 
> I think I misunderstand something about the "liveness" of the data in a global table.
> I thought if I define and materialize a GlobalKTable at the startup of my application, and messages arrive later to the underlying topic, the data in the store are automatically updated.
> i.e. if I have a method that queries the store of the table, it will see the most up-to-date data. (With a bit of latency of course.)
> However, it seems not to be the case:
> 
> 
> 1.       My application defines a GlobalKTable over a topic, materialized with a name and serializers passed.
> 
> 2.       I have a business logic method that calls store() method with the name of this global table, and tries to get a value by key.
> 
> 3.       Next, I wrote a test case, that starts the application, and I send a message with a producer to the appropriate topic, and I expect I can get it by its key from the global table's store.
> 
> However, it returns null. I can see the message in the topic so it is surely delivered.
> 
> 4.       If I iterate through the store via store.all() before calling store.get(key), the KeyValue is there with the appropriate key! However, store.get(key) still returns null.
> 
> 5.       I implemented hashCode and equals for my key class appropriately, so it cannot be the problem
> 
> 6.       Also, it is a not about a small latency, because if I wait about a minute, still cannot get the data.
> 
> 7.       If I modify my test case so that I do not send a test message (because there is one in the topic already), and rerun the test (which starts the application and defined the global table),
> 
> the value is correctly found by key. (I guess the message from the topic is read when the table is created.)
> 
> Do you have any idea what am I missing?
> 
> Best regards
> 
> Gabor
>