[jira] [Assigned] (KAFKA-6048) Support negative record timestamps

["Matthias J. Sax (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias J. Sax reassigned KAFKA-6048:
--------------------------------------

    Assignee: Konstantin Chukhlomin  (was: james chien)

> Support negative record timestamps
> ----------------------------------
>
>                 Key: KAFKA-6048
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6048
>             Project: Kafka
>          Issue Type: Improvement
>          Components: clients, core, streams
>    Affects Versions: 1.0.0
>            Reporter: Matthias J. Sax
>            Assignee: Konstantin Chukhlomin
>              Labels: needs-kip
>
> Kafka does not support negative record timestamps, and this prevents the storage of historical data in Kafka. In general, negative timestamps are supported by UNIX system time stamps: 
> From https://en.wikipedia.org/wiki/Unix_time
> {quote}
> The Unix time number is zero at the Unix epoch, and increases by exactly 86,400 per day since the epoch. Thus 2004-09-16T00:00:00Z, 12,677 days after the epoch, is represented by the Unix time number 12,677 × 86,400 = 1095292800. This can be extended backwards from the epoch too, using negative numbers; thus 1957-10-04T00:00:00Z, 4,472 days before the epoch, is represented by the Unix time number −4,472 × 86,400 = −386380800.
> {quote}
> Allowing for negative timestamps would require multiple changes:
>  - while brokers in general do support negative timestamps, broker use {{-1}} as default value if a producer uses an old message format (this would not be compatible with supporting negative timestamps "end-to-end" as {{-1}} cannot be used as "unknown" anymore): we could introduce a message flag indicating a missing timestamp (and let producer throw an exception if {{ConsumerRecord#timestamp()}} is called. Another possible solution might be, to require topics that are used by old producers to be configured with {{LogAppendTime}} semantics and rejecting writes to topics with {{CreateTime}} semantics for older message formats
>  - {{KafkaProducer}} does not allow to send records with negative timestamp and thus this would need to be fixed
>  - Streams API does drop records with negative timestamps (or fails by default) -- also, some internal store implementation for windowed stores assume that there are not negative timestamps to do range queries
> There might be other gaps we need to address. This is just a summary of issue coming to my mind atm.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)