[jira] [Created] (KAFKA-6048) Support negative record timestamps

["Matthias J. Sax (JIRA)" <ji...@apache.org>]

Matthias J. Sax created KAFKA-6048:
--------------------------------------

             Summary: Support negative record timestamps
                 Key: KAFKA-6048
                 URL: https://issues.apache.org/jira/browse/KAFKA-6048
             Project: Kafka
          Issue Type: Improvement
          Components: clients, core, streams
    Affects Versions: 1.0.0
            Reporter: Matthias J. Sax


Kafka does not support negative record timestamps, and this prevents the storage of historical data in Kafka. In general, negative timestamps are supported by UNIX system time stamps: 

From https://en.wikipedia.org/wiki/Unix_time
{quote}
The Unix time number is zero at the Unix epoch, and increases by exactly 86,400 per day since the epoch. Thus 2004-09-16T00:00:00Z, 12,677 days after the epoch, is represented by the Unix time number 12,677 × 86,400 = 1095292800. This can be extended backwards from the epoch too, using negative numbers; thus 1957-10-04T00:00:00Z, 4,472 days before the epoch, is represented by the Unix time number −4,472 × 86,400 = −386380800.
{quote}

Allowing for negative timestamps would require multiple changes:

 - while brokers in general do support negative timestamps, broker use {{-1}} as default value if a producer uses an old message format (this would not be compatible with supporting negative timestamps "end-to-end" as {{-1}} cannot be used as "unknown" anymore): we could introduce a message flag indicating a missing timestamp (and let producer throw an exception if {{ConsumerRecord#timestamp()}} is called. Another possible solution might be, to require topics that are used by old producers to be configured with {{LogAppendTime}} semantics and rejecting writes to topics with {{CreateTime}} semantics for older message formats
 - {{KafkaProducer}} does not allow to send records with negative timestamp and thus this would need to be fixed
 - Streams API does drop records with negative timestamps (or fails by default) -- also, some internal store implementation for windowed stores assume that there are not negative timestamps to do range queries

There might be other gaps we need to address. This is just a summary of issue coming to my mind atm.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)