You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jake Yoon <ja...@eggbun-edu.com> on 2018/01/19 03:46:23 UTC
What's the use of timestamp in ProducerRecord?
Hi, I am very new to Kafka.
And I have a very basic question.
Kafka doc says,
*ProducerRecord
<https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html#ProducerRecord(java.lang.String,%20java.lang.Integer,%20java.lang.Long,%20K,%20V)>*
(String
<http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true>
topic, Integer
<http://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html?is-external=true>
partition, Long
<http://docs.oracle.com/javase/7/docs/api/java/lang/Long.html?is-external=true>
timestamp, K
<https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html>
key, V
<https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html>
value)
and I know the default timestamp is the current time. But I am not sure
what's the use of it.
- Is it just to log when the record is added?
- How Kafka use it for?
- Are there any other uses of it?
- Can "Consumer" retrieves the timestamp of the "ProducerRecord"?
Thanks,
Re: What's the use of timestamp in ProducerRecord?
Posted by "1095193290@qq.com" <10...@qq.com>.
kafka does not delete message when message is consumed, it will purge message when this message is expired. I guess this timeStamp is for checking whether message is expired.
1095193290@qq.com
From: Jake Yoon
Date: 2018-01-19 11:46
To: users
Subject: What's the use of timestamp in ProducerRecord?
Hi, I am very new to Kafka.
And I have a very basic question.
Kafka doc says,
*ProducerRecord
<https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html#ProducerRecord(java.lang.String,%20java.lang.Integer,%20java.lang.Long,%20K,%20V)>*
(String
<http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true>
topic, Integer
<http://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html?is-external=true>
partition, Long
<http://docs.oracle.com/javase/7/docs/api/java/lang/Long.html?is-external=true>
timestamp, K
<https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html>
key, V
<https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html>
value)
and I know the default timestamp is the current time. But I am not sure
what's the use of it.
- Is it just to log when the record is added?
- How Kafka use it for?
- Are there any other uses of it?
- Can "Consumer" retrieves the timestamp of the "ProducerRecord"?
Thanks,
Re: What's the use of timestamp in ProducerRecord?
Posted by Dan Markhasin <mi...@gmail.com>.
It is also used for rewinding consumer offsets.
On 19 January 2018 at 06:25, Matthias J. Sax <ma...@confluent.io> wrote:
> The timestamp has many different purposes. As mentioned already, it used
> to expired data via retention time. It's also used for stream processing
> via Streams API. All processing is based on those timestamps for
> windowing and joining of streams.
>
> -Matthias
>
> On 1/18/18 7:57 PM, Stephen Powis wrote:
> > A common use case is for calculating a "processing lag time" comparing
> the
> > record's timestamp (when it was published/persisted) against the current
> > time that the consumer 'consumed' it. In theory, the delta is 'how far
> > behind' the consumer is, tho you can come up with a variety of reasons
> why
> > this wouldn't actually be true.
> >
> > On Fri, Jan 19, 2018 at 12:52 PM, 1095193290@qq.com <10...@qq.com>
> > wrote:
> >
> >> kafka does not delete message when message is consumed, it will purge
> >> message when this message is expired. I guess this timeStamp is for
> >> checking whether message is expired.
> >>
> >>
> >>
> >> 1095193290@qq.com
> >>
> >> From: Jake Yoon
> >> Date: 2018-01-19 11:46
> >> To: users
> >> Subject: What's the use of timestamp in ProducerRecord?
> >> Hi, I am very new to Kafka.
> >> And I have a very basic question.
> >>
> >> Kafka doc says,
> >>
> >> *ProducerRecord
> >> <https://kafka.apache.org/0100/javadoc/org/apache/kafka/
> clients/producer/
> >> ProducerRecord.html#ProducerRecord(java.lang.
> String,%20java.lang.Integer,%
> >> 20java.lang.Long,%20K,%20V)>*
> >> (String
> >> <http://docs.oracle.com/javase/7/docs/api/java/lang/
> >> String.html?is-external=true>
> >> topic, Integer
> >> <http://docs.oracle.com/javase/7/docs/api/java/lang/
> >> Integer.html?is-external=true>
> >> partition, Long
> >> <http://docs.oracle.com/javase/7/docs/api/java/lang/
> >> Long.html?is-external=true>
> >> timestamp, K
> >> <https://kafka.apache.org/0100/javadoc/org/apache/kafka/
> clients/producer/
> >> ProducerRecord.html>
> >> key, V
> >> <https://kafka.apache.org/0100/javadoc/org/apache/kafka/
> clients/producer/
> >> ProducerRecord.html>
> >> value)
> >>
> >> and I know the default timestamp is the current time. But I am not sure
> >> what's the use of it.
> >>
> >> - Is it just to log when the record is added?
> >> - How Kafka use it for?
> >> - Are there any other uses of it?
> >> - Can "Consumer" retrieves the timestamp of the "ProducerRecord"?
> >>
> >> Thanks,
> >>
> >
>
>
Re: What's the use of timestamp in ProducerRecord?
Posted by "Matthias J. Sax" <ma...@confluent.io>.
The timestamp has many different purposes. As mentioned already, it used
to expired data via retention time. It's also used for stream processing
via Streams API. All processing is based on those timestamps for
windowing and joining of streams.
-Matthias
On 1/18/18 7:57 PM, Stephen Powis wrote:
> A common use case is for calculating a "processing lag time" comparing the
> record's timestamp (when it was published/persisted) against the current
> time that the consumer 'consumed' it. In theory, the delta is 'how far
> behind' the consumer is, tho you can come up with a variety of reasons why
> this wouldn't actually be true.
>
> On Fri, Jan 19, 2018 at 12:52 PM, 1095193290@qq.com <10...@qq.com>
> wrote:
>
>> kafka does not delete message when message is consumed, it will purge
>> message when this message is expired. I guess this timeStamp is for
>> checking whether message is expired.
>>
>>
>>
>> 1095193290@qq.com
>>
>> From: Jake Yoon
>> Date: 2018-01-19 11:46
>> To: users
>> Subject: What's the use of timestamp in ProducerRecord?
>> Hi, I am very new to Kafka.
>> And I have a very basic question.
>>
>> Kafka doc says,
>>
>> *ProducerRecord
>> <https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/
>> ProducerRecord.html#ProducerRecord(java.lang.String,%20java.lang.Integer,%
>> 20java.lang.Long,%20K,%20V)>*
>> (String
>> <http://docs.oracle.com/javase/7/docs/api/java/lang/
>> String.html?is-external=true>
>> topic, Integer
>> <http://docs.oracle.com/javase/7/docs/api/java/lang/
>> Integer.html?is-external=true>
>> partition, Long
>> <http://docs.oracle.com/javase/7/docs/api/java/lang/
>> Long.html?is-external=true>
>> timestamp, K
>> <https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/
>> ProducerRecord.html>
>> key, V
>> <https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/
>> ProducerRecord.html>
>> value)
>>
>> and I know the default timestamp is the current time. But I am not sure
>> what's the use of it.
>>
>> - Is it just to log when the record is added?
>> - How Kafka use it for?
>> - Are there any other uses of it?
>> - Can "Consumer" retrieves the timestamp of the "ProducerRecord"?
>>
>> Thanks,
>>
>
Re: What's the use of timestamp in ProducerRecord?
Posted by Stephen Powis <sp...@salesforce.com>.
A common use case is for calculating a "processing lag time" comparing the
record's timestamp (when it was published/persisted) against the current
time that the consumer 'consumed' it. In theory, the delta is 'how far
behind' the consumer is, tho you can come up with a variety of reasons why
this wouldn't actually be true.
On Fri, Jan 19, 2018 at 12:52 PM, 1095193290@qq.com <10...@qq.com>
wrote:
> kafka does not delete message when message is consumed, it will purge
> message when this message is expired. I guess this timeStamp is for
> checking whether message is expired.
>
>
>
> 1095193290@qq.com
>
> From: Jake Yoon
> Date: 2018-01-19 11:46
> To: users
> Subject: What's the use of timestamp in ProducerRecord?
> Hi, I am very new to Kafka.
> And I have a very basic question.
>
> Kafka doc says,
>
> *ProducerRecord
> <https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/
> ProducerRecord.html#ProducerRecord(java.lang.String,%20java.lang.Integer,%
> 20java.lang.Long,%20K,%20V)>*
> (String
> <http://docs.oracle.com/javase/7/docs/api/java/lang/
> String.html?is-external=true>
> topic, Integer
> <http://docs.oracle.com/javase/7/docs/api/java/lang/
> Integer.html?is-external=true>
> partition, Long
> <http://docs.oracle.com/javase/7/docs/api/java/lang/
> Long.html?is-external=true>
> timestamp, K
> <https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/
> ProducerRecord.html>
> key, V
> <https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/
> ProducerRecord.html>
> value)
>
> and I know the default timestamp is the current time. But I am not sure
> what's the use of it.
>
> - Is it just to log when the record is added?
> - How Kafka use it for?
> - Are there any other uses of it?
> - Can "Consumer" retrieves the timestamp of the "ProducerRecord"?
>
> Thanks,
>