You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Sachin Mittal <sj...@gmail.com> on 2024/10/25 11:09:37 UTC

Kafka sink producing record at event timestamp

Hi,
I am having a pipeline where source and sink are two Kafka topics.
The pipeline uses event time semantics, where event time is extracted from
the record.

What I notice is that when producing records at the sink side, it produces
them such that the record's time in the kafka topic is the same as its
event time.

Ideally I was expecting that the record's time should be the timestamp when
the record was inserted in the Kafka topic and not the same as its event
time.

How can I control this behaviour?

Thanks
Sachin

Re: Kafka sink producing record at event timestamp

Posted by Sebastian Zapata <se...@gmail.com>.
there is one option of configuring this in kafka at the topic level
"""
<https://stackoverflow.com/posts/55473742/timeline>

By default Kafka will use the timestamp provided by the producer.

However, you can also make Kafka update the timestamp when it writes the
record to the log by setting message.timestamp.type to LogAppendTime on
your topic.

See the topic configurations section
<http://kafka.apache.org/documentation/#topicconfigs> in the documentation.
""""
https://stackoverflow.com/questions/55472723/get-the-time-the-message-reached-kafka




On Fri, Oct 25, 2024 at 6:14 AM Sachin Mittal <sj...@gmail.com> wrote:

> Hi,
> I am having a pipeline where source and sink are two Kafka topics.
> The pipeline uses event time semantics, where event time is extracted from
> the record.
>
> What I notice is that when producing records at the sink side, it produces
> them such that the record's time in the kafka topic is the same as its
> event time.
>
> Ideally I was expecting that the record's time should be the timestamp
> when the record was inserted in the Kafka topic and not the same as its
> event time.
>
> How can I control this behaviour?
>
> Thanks
> Sachin
>
>