You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Andrew Otto <ot...@wikimedia.org> on 2020/04/16 16:39:39 UTC
log.message.timestamp.difference.max.ms and future timestamps?
log.message.timestamp.difference.max.ms is the "The maximum difference
allowed between the timestamp when a broker receives a message and the
timestamp specified in the message.".
What happens if the message timestamp is in the future? We just
encountered a problem where a producer set a timestamp a year in the
future, which is keeping log segments with that message to not be deleted,
causing disks to fill. I'd like to reject messages with timestamps (too
far) in the future. Based on the way the docs are worded, I'd assume the
difference will be negative in this case.
log.message.timestamp.difference.max.ms - futureTimestamp == -bigNumber
Will the message be rejected or accepted in this case?
Thanks!
-Andrew Otto
@Wikimedia Foundation
Re: log.message.timestamp.difference.max.ms and future timestamps?
Posted by Andrew Otto <ot...@wikimedia.org>.
Thank you, good to know!
On Fri, May 8, 2020 at 2:53 PM Matthias J. Sax <mj...@apache.org> wrote:
> >> What happens if the message timestamp is in the future?
>
> If the difference if larger than
> `log.message.timestamp.difference.max.ms` the write will be rejected.
>
> This timestamp difference works both ways.
>
>
> -Matthias
>
>
> On 4/16/20 9:39 AM, Andrew Otto wrote:
> > log.message.timestamp.difference.max.ms is the "The maximum difference
> > allowed between the timestamp when a broker receives a message and the
> > timestamp specified in the message.".
> >
> > What happens if the message timestamp is in the future? We just
> > encountered a problem where a producer set a timestamp a year in the
> > future, which is keeping log segments with that message to not be
> deleted,
> > causing disks to fill. I'd like to reject messages with timestamps (too
> > far) in the future. Based on the way the docs are worded, I'd assume the
> > difference will be negative in this case.
> >
> > log.message.timestamp.difference.max.ms - futureTimestamp ==
> -bigNumber
> >
> > Will the message be rejected or accepted in this case?
> >
> > Thanks!
> > -Andrew Otto
> > @Wikimedia Foundation
> >
>
>
Re: log.message.timestamp.difference.max.ms and future timestamps?
Posted by "Matthias J. Sax" <mj...@apache.org>.
>> What happens if the message timestamp is in the future?
If the difference if larger than
`log.message.timestamp.difference.max.ms` the write will be rejected.
This timestamp difference works both ways.
-Matthias
On 4/16/20 9:39 AM, Andrew Otto wrote:
> log.message.timestamp.difference.max.ms is the "The maximum difference
> allowed between the timestamp when a broker receives a message and the
> timestamp specified in the message.".
>
> What happens if the message timestamp is in the future? We just
> encountered a problem where a producer set a timestamp a year in the
> future, which is keeping log segments with that message to not be deleted,
> causing disks to fill. I'd like to reject messages with timestamps (too
> far) in the future. Based on the way the docs are worded, I'd assume the
> difference will be negative in this case.
>
> log.message.timestamp.difference.max.ms - futureTimestamp == -bigNumber
>
> Will the message be rejected or accepted in this case?
>
> Thanks!
> -Andrew Otto
> @Wikimedia Foundation
>