Kafka order info to MySQL discard middle status and guarantee final correctness

[Lei Wang <le...@gmail.com>]

Receive order message from kafka, every message has a status field, the
schema is just like this:  orderId,    status

The message will be inserted to MySQL.

For each order  there's too much status and changes very frequently. In
order to reduce stress to the database, we can discard some middle status,
but must guarantee the final status is correct. In addition, the final
status is not a definite value.

How  can I implement this?

Thanks,
Lei

Re: Kafka order info to MySQL discard middle status and guarantee final correctness

[胡伟华 <hu...@gmail.com>]

Hi, Lei Wang

I think you could use a custom filter, the logic is as follows:
1. If the status is a final status, then return true
2. If the status is a final status, perform random sampling return true


> 2022年2月28日 上午10:16，Lei Wang <le...@gmail.com> 写道：
> 
> Receive order message from kafka, every message has a status field, the schema is just like this:  orderId,    status    
> 
> The message will be inserted to MySQL. 
> 
> For each order  there's too much status and changes very frequently. In order to reduce stress to the database, we can discard some middle status, but must guarantee the final status is correct. In addition, the final status is not a definite value. 
> 
> How  can I implement this?
> 
> Thanks,
> Lei