You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by pritam kumar <ku...@gmail.com> on 2025/04/04 18:44:52 UTC
[DISCUSS] KIP-1153: Extending support for Microsecond Precision for Kafka Connect
Hi Kafka Community,
I’d like to start a discussion on KIP-1153: Extending Support for
Microsecond Precision for Kafka Connect
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect>
.
The primary motivation behind this KIP is to enhance the precision of
timestamp handling in Kafka Connect. Currently, Kafka Connect is limited to
millisecond-level precision for timestamps. However, many modern data
formats and platforms have moved beyond this limitation:
-
Formats such as *Avro*, *Parquet*, and *ORC* support microsecond (and
even nanosecond) precision. For example, Avro specifies support for
timestamp-micros (spec link
<https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>).
-
Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi*
offer *microsecond and nanosecond precision* for time-based fields,
making millisecond precision inadequate for accurate data replication and
analytics in some use cases.
This gap can lead to *loss of precision* when transferring data through
Kafka Connect, especially when interacting with systems that depend on
high-resolution timestamps for change tracking, event ordering, or
deduplication.
The goal of this KIP is to:
-
Introduce microsecond-level timestamp handling in Kafka Connect schema
and data representation.
-
Ensure connectors (both source and sink) can leverage this precision
when supported by the underlying data systems.
-
Maintain backward compatibility with existing millisecond-based
configurations and data.
We welcome community feedback on:
-
Potential implementation concerns or edge cases we should address
-
Suggestions for schema enhancements or conversion strategies
-
Impacts on connector compatibility and testing
Looking forward to your thoughts and input on this proposal!
Thanks.
Link to the KIP.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API>
Re: [DISCUSS] KIP-1153: Extending support for Microsecond Precision for Kafka Connect
Posted by pritam kumar <ku...@gmail.com>.
Hi Mickael,
Thank you for the detailed review! And really sorry for the absence and
late reply. Also I am extremely sorry for the discrepancy between the KIP
number in the discussion thread and the actual KIP number. Actually I took
KIP 1153 but forgot to update the number on the main page. I've updated the
KIP to address all your feedback points.
1. Removed full implementation code. The KIP now explicitly lists every
public API change (method signatures, types, return values) in tables and
describes the associated behavior
changes for each—without embedding the full logic.
2. *TimestampConverter* is now clearly called out. It has its own
dedicated section ("5. Updated SMT: TimestampConverter") that describes the
new target.type values (TimestampMicros,
TimestampNanos), precision-preserving unix conversion behavior, and
updates to timestampTypeFromSchema().
3. Compatibility section updated. I've replaced the blanket "No breaking
changes" statement with a nuanced breakdown:
- A "Backward Compatibility" subsection confirming existing
configurations are unaffected.
- A "Behavioral Changes" subsection listing what changes when the new
types are actively used (opt-in via connector/converter configuration).
- A "Rolling Upgrades" subsection covering the scenario where workers
are at different versions.
4. Test plan expanded. Now covers TimestampMicros, TimestampNanos,
TimestampConverter (including precision-preserving conversions, upscaling,
truncation, schema-based and schemaless
modes), JsonConverter, ConnectSchema validation, and the Cast SMT.
I've also added a summary table of all files changed across modules for
easier review.
Updated KIP: KIP-1154: Extending support for Microsecond/Nanosecond
Precision for Kafka Connect
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=347933282#KIP1154:ExtendingsupportforMicrosecond/NanosecondPrecisionforKafkaConnect-1.NewLogicalType:TimestampMicros>
Looking forward to your thoughts.
Thanks,
Pritam
On Mon, Jan 5, 2026 at 10:00 PM Mickael Maison <mi...@gmail.com>
wrote:
> Hi,
>
> Thanks for the KIP. I think overall this would be a useful improvement.
>
> However, the KIP is kind of messy and hard to read.
> It shouldn't include the full logic but instead it should explicitly
> list the changes to public APIs as well as other major required
> changes. For each it should highlight the associated behavior changes.
>
> It seems the code following the "After introduction of these logical
> types, a fix will be added to KIP-808 in the following way" sentence
> refers to the TimestampConverter transformation. If so, can you make
> it clear that this transformation is being updated as part of the KIP?
>
> In the Compatibility, Deprecation, and Migration Plan, you state "No
> breaking changes" and "Backward-compatible". It seems after this KIP,
> there would be some behavioral changes as there are no way to disable
> the feature.
>
> In test plan you will need to also test TimestampNanos and
> TimestampConverter (if applicable).
>
> Thanks,
> Mickael
>
>
> On Wed, May 21, 2025 at 2:56 AM pritam kumar <ku...@gmail.com>
> wrote:
> >
> > Hi Kafka Community,
> > If there is no other feedback. I would like to start a VOTE.
> >
> > On Mon, May 19, 2025 at 6:51 PM pritam kumar <ku...@gmail.com>
> > wrote:
> >
> > > Hi Kafka Community,
> > > Here is the KIP LInk ->
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=347933282
> > >
> > > On Mon, May 19, 2025 at 6:47 PM pritam kumar <
> kumarpritammsd7@gmail.com>
> > > wrote:
> > >
> > >> Hi Kafka Community,
> > >>
> > >> Could you please review this KIP proposal when you have a chance?
> > >>
> > >> Thank you,
> > >> Pritam
> > >>
> > >>
> > >> On Sat, May 3, 2025 at 10:53 PM pritam kumar <
> kumarpritammsd7@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi Chris,
> > >>> Can you please review this one too?
> > >>>
> > >>> On Sat, Apr 5, 2025 at 7:00 AM pritam kumar <
> kumarpritammsd7@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi Kafka Community,
> > >>>> Sorry due to some changes I had to change the link to the kip.
> > >>>> Here is the updated KIP link:
> > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1154%3A+Extending+support+for+Microsecond+Precision+for+Kafka+Connect
> > >>>>
> > >>>> On Sat, Apr 5, 2025 at 12:14 AM pritam kumar <
> kumarpritammsd7@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi Kafka Community,
> > >>>>>
> > >>>>> I’d like to start a discussion on KIP-1153: Extending Support for
> > >>>>> Microsecond Precision for Kafka Connect
> > >>>>> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect
> >
> > >>>>> .
> > >>>>>
> > >>>>> The primary motivation behind this KIP is to enhance the precision
> of
> > >>>>> timestamp handling in Kafka Connect. Currently, Kafka Connect is
> limited to
> > >>>>> millisecond-level precision for timestamps. However, many modern
> data
> > >>>>> formats and platforms have moved beyond this limitation:
> > >>>>>
> > >>>>> -
> > >>>>>
> > >>>>> Formats such as *Avro*, *Parquet*, and *ORC* support microsecond
> > >>>>> (and even nanosecond) precision. For example, Avro specifies
> support for
> > >>>>> timestamp-micros (spec link
> > >>>>> <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros
> >).
> > >>>>> -
> > >>>>>
> > >>>>> Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache
> Hudi*
> > >>>>> offer *microsecond and nanosecond precision* for time-based
> > >>>>> fields, making millisecond precision inadequate for accurate
> data
> > >>>>> replication and analytics in some use cases.
> > >>>>>
> > >>>>> This gap can lead to *loss of precision* when transferring data
> > >>>>> through Kafka Connect, especially when interacting with systems
> that depend
> > >>>>> on high-resolution timestamps for change tracking, event ordering,
> or
> > >>>>> deduplication.
> > >>>>>
> > >>>>> The goal of this KIP is to:
> > >>>>>
> > >>>>> -
> > >>>>>
> > >>>>> Introduce microsecond-level timestamp handling in Kafka Connect
> > >>>>> schema and data representation.
> > >>>>> -
> > >>>>>
> > >>>>> Ensure connectors (both source and sink) can leverage this
> > >>>>> precision when supported by the underlying data systems.
> > >>>>> -
> > >>>>>
> > >>>>> Maintain backward compatibility with existing millisecond-based
> > >>>>> configurations and data.
> > >>>>>
> > >>>>> We welcome community feedback on:
> > >>>>>
> > >>>>> -
> > >>>>>
> > >>>>> Potential implementation concerns or edge cases we should
> address
> > >>>>> -
> > >>>>>
> > >>>>> Suggestions for schema enhancements or conversion strategies
> > >>>>> -
> > >>>>>
> > >>>>> Impacts on connector compatibility and testing
> > >>>>>
> > >>>>> Looking forward to your thoughts and input on this proposal!
> > >>>>>
> > >>>>> Thanks.
> > >>>>> Link to the KIP.
> > >>>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect
> > >>>>> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API
> >
> > >>>>>
> > >>>>>
>
Re: [DISCUSS] KIP-1153: Extending support for Microsecond Precision for Kafka Connect
Posted by Mickael Maison <mi...@gmail.com>.
Hi,
Thanks for the KIP. I think overall this would be a useful improvement.
However, the KIP is kind of messy and hard to read.
It shouldn't include the full logic but instead it should explicitly
list the changes to public APIs as well as other major required
changes. For each it should highlight the associated behavior changes.
It seems the code following the "After introduction of these logical
types, a fix will be added to KIP-808 in the following way" sentence
refers to the TimestampConverter transformation. If so, can you make
it clear that this transformation is being updated as part of the KIP?
In the Compatibility, Deprecation, and Migration Plan, you state "No
breaking changes" and "Backward-compatible". It seems after this KIP,
there would be some behavioral changes as there are no way to disable
the feature.
In test plan you will need to also test TimestampNanos and
TimestampConverter (if applicable).
Thanks,
Mickael
On Wed, May 21, 2025 at 2:56 AM pritam kumar <ku...@gmail.com> wrote:
>
> Hi Kafka Community,
> If there is no other feedback. I would like to start a VOTE.
>
> On Mon, May 19, 2025 at 6:51 PM pritam kumar <ku...@gmail.com>
> wrote:
>
> > Hi Kafka Community,
> > Here is the KIP LInk ->
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=347933282
> >
> > On Mon, May 19, 2025 at 6:47 PM pritam kumar <ku...@gmail.com>
> > wrote:
> >
> >> Hi Kafka Community,
> >>
> >> Could you please review this KIP proposal when you have a chance?
> >>
> >> Thank you,
> >> Pritam
> >>
> >>
> >> On Sat, May 3, 2025 at 10:53 PM pritam kumar <ku...@gmail.com>
> >> wrote:
> >>
> >>> Hi Chris,
> >>> Can you please review this one too?
> >>>
> >>> On Sat, Apr 5, 2025 at 7:00 AM pritam kumar <ku...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi Kafka Community,
> >>>> Sorry due to some changes I had to change the link to the kip.
> >>>> Here is the updated KIP link:
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1154%3A+Extending+support+for+Microsecond+Precision+for+Kafka+Connect
> >>>>
> >>>> On Sat, Apr 5, 2025 at 12:14 AM pritam kumar <ku...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Kafka Community,
> >>>>>
> >>>>> I’d like to start a discussion on KIP-1153: Extending Support for
> >>>>> Microsecond Precision for Kafka Connect
> >>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect>
> >>>>> .
> >>>>>
> >>>>> The primary motivation behind this KIP is to enhance the precision of
> >>>>> timestamp handling in Kafka Connect. Currently, Kafka Connect is limited to
> >>>>> millisecond-level precision for timestamps. However, many modern data
> >>>>> formats and platforms have moved beyond this limitation:
> >>>>>
> >>>>> -
> >>>>>
> >>>>> Formats such as *Avro*, *Parquet*, and *ORC* support microsecond
> >>>>> (and even nanosecond) precision. For example, Avro specifies support for
> >>>>> timestamp-micros (spec link
> >>>>> <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>).
> >>>>> -
> >>>>>
> >>>>> Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi*
> >>>>> offer *microsecond and nanosecond precision* for time-based
> >>>>> fields, making millisecond precision inadequate for accurate data
> >>>>> replication and analytics in some use cases.
> >>>>>
> >>>>> This gap can lead to *loss of precision* when transferring data
> >>>>> through Kafka Connect, especially when interacting with systems that depend
> >>>>> on high-resolution timestamps for change tracking, event ordering, or
> >>>>> deduplication.
> >>>>>
> >>>>> The goal of this KIP is to:
> >>>>>
> >>>>> -
> >>>>>
> >>>>> Introduce microsecond-level timestamp handling in Kafka Connect
> >>>>> schema and data representation.
> >>>>> -
> >>>>>
> >>>>> Ensure connectors (both source and sink) can leverage this
> >>>>> precision when supported by the underlying data systems.
> >>>>> -
> >>>>>
> >>>>> Maintain backward compatibility with existing millisecond-based
> >>>>> configurations and data.
> >>>>>
> >>>>> We welcome community feedback on:
> >>>>>
> >>>>> -
> >>>>>
> >>>>> Potential implementation concerns or edge cases we should address
> >>>>> -
> >>>>>
> >>>>> Suggestions for schema enhancements or conversion strategies
> >>>>> -
> >>>>>
> >>>>> Impacts on connector compatibility and testing
> >>>>>
> >>>>> Looking forward to your thoughts and input on this proposal!
> >>>>>
> >>>>> Thanks.
> >>>>> Link to the KIP.
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect
> >>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API>
> >>>>>
> >>>>>
Re: [DISCUSS] KIP-1153: Extending support for Microsecond Precision for Kafka Connect
Posted by pritam kumar <ku...@gmail.com>.
Hi Kafka Community,
If there is no other feedback. I would like to start a VOTE.
On Mon, May 19, 2025 at 6:51 PM pritam kumar <ku...@gmail.com>
wrote:
> Hi Kafka Community,
> Here is the KIP LInk ->
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=347933282
>
> On Mon, May 19, 2025 at 6:47 PM pritam kumar <ku...@gmail.com>
> wrote:
>
>> Hi Kafka Community,
>>
>> Could you please review this KIP proposal when you have a chance?
>>
>> Thank you,
>> Pritam
>>
>>
>> On Sat, May 3, 2025 at 10:53 PM pritam kumar <ku...@gmail.com>
>> wrote:
>>
>>> Hi Chris,
>>> Can you please review this one too?
>>>
>>> On Sat, Apr 5, 2025 at 7:00 AM pritam kumar <ku...@gmail.com>
>>> wrote:
>>>
>>>> Hi Kafka Community,
>>>> Sorry due to some changes I had to change the link to the kip.
>>>> Here is the updated KIP link:
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1154%3A+Extending+support+for+Microsecond+Precision+for+Kafka+Connect
>>>>
>>>> On Sat, Apr 5, 2025 at 12:14 AM pritam kumar <ku...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Kafka Community,
>>>>>
>>>>> I’d like to start a discussion on KIP-1153: Extending Support for
>>>>> Microsecond Precision for Kafka Connect
>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect>
>>>>> .
>>>>>
>>>>> The primary motivation behind this KIP is to enhance the precision of
>>>>> timestamp handling in Kafka Connect. Currently, Kafka Connect is limited to
>>>>> millisecond-level precision for timestamps. However, many modern data
>>>>> formats and platforms have moved beyond this limitation:
>>>>>
>>>>> -
>>>>>
>>>>> Formats such as *Avro*, *Parquet*, and *ORC* support microsecond
>>>>> (and even nanosecond) precision. For example, Avro specifies support for
>>>>> timestamp-micros (spec link
>>>>> <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>).
>>>>> -
>>>>>
>>>>> Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi*
>>>>> offer *microsecond and nanosecond precision* for time-based
>>>>> fields, making millisecond precision inadequate for accurate data
>>>>> replication and analytics in some use cases.
>>>>>
>>>>> This gap can lead to *loss of precision* when transferring data
>>>>> through Kafka Connect, especially when interacting with systems that depend
>>>>> on high-resolution timestamps for change tracking, event ordering, or
>>>>> deduplication.
>>>>>
>>>>> The goal of this KIP is to:
>>>>>
>>>>> -
>>>>>
>>>>> Introduce microsecond-level timestamp handling in Kafka Connect
>>>>> schema and data representation.
>>>>> -
>>>>>
>>>>> Ensure connectors (both source and sink) can leverage this
>>>>> precision when supported by the underlying data systems.
>>>>> -
>>>>>
>>>>> Maintain backward compatibility with existing millisecond-based
>>>>> configurations and data.
>>>>>
>>>>> We welcome community feedback on:
>>>>>
>>>>> -
>>>>>
>>>>> Potential implementation concerns or edge cases we should address
>>>>> -
>>>>>
>>>>> Suggestions for schema enhancements or conversion strategies
>>>>> -
>>>>>
>>>>> Impacts on connector compatibility and testing
>>>>>
>>>>> Looking forward to your thoughts and input on this proposal!
>>>>>
>>>>> Thanks.
>>>>> Link to the KIP.
>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect
>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API>
>>>>>
>>>>>
Re: [DISCUSS] KIP-1153: Extending support for Microsecond Precision for Kafka Connect
Posted by pritam kumar <ku...@gmail.com>.
Hi Kafka Community,
Here is the KIP LInk ->
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=347933282
On Mon, May 19, 2025 at 6:47 PM pritam kumar <ku...@gmail.com>
wrote:
> Hi Kafka Community,
>
> Could you please review this KIP proposal when you have a chance?
>
> Thank you,
> Pritam
>
>
> On Sat, May 3, 2025 at 10:53 PM pritam kumar <ku...@gmail.com>
> wrote:
>
>> Hi Chris,
>> Can you please review this one too?
>>
>> On Sat, Apr 5, 2025 at 7:00 AM pritam kumar <ku...@gmail.com>
>> wrote:
>>
>>> Hi Kafka Community,
>>> Sorry due to some changes I had to change the link to the kip.
>>> Here is the updated KIP link:
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1154%3A+Extending+support+for+Microsecond+Precision+for+Kafka+Connect
>>>
>>> On Sat, Apr 5, 2025 at 12:14 AM pritam kumar <ku...@gmail.com>
>>> wrote:
>>>
>>>> Hi Kafka Community,
>>>>
>>>> I’d like to start a discussion on KIP-1153: Extending Support for
>>>> Microsecond Precision for Kafka Connect
>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect>
>>>> .
>>>>
>>>> The primary motivation behind this KIP is to enhance the precision of
>>>> timestamp handling in Kafka Connect. Currently, Kafka Connect is limited to
>>>> millisecond-level precision for timestamps. However, many modern data
>>>> formats and platforms have moved beyond this limitation:
>>>>
>>>> -
>>>>
>>>> Formats such as *Avro*, *Parquet*, and *ORC* support microsecond
>>>> (and even nanosecond) precision. For example, Avro specifies support for
>>>> timestamp-micros (spec link
>>>> <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>).
>>>> -
>>>>
>>>> Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi*
>>>> offer *microsecond and nanosecond precision* for time-based
>>>> fields, making millisecond precision inadequate for accurate data
>>>> replication and analytics in some use cases.
>>>>
>>>> This gap can lead to *loss of precision* when transferring data
>>>> through Kafka Connect, especially when interacting with systems that depend
>>>> on high-resolution timestamps for change tracking, event ordering, or
>>>> deduplication.
>>>>
>>>> The goal of this KIP is to:
>>>>
>>>> -
>>>>
>>>> Introduce microsecond-level timestamp handling in Kafka Connect
>>>> schema and data representation.
>>>> -
>>>>
>>>> Ensure connectors (both source and sink) can leverage this
>>>> precision when supported by the underlying data systems.
>>>> -
>>>>
>>>> Maintain backward compatibility with existing millisecond-based
>>>> configurations and data.
>>>>
>>>> We welcome community feedback on:
>>>>
>>>> -
>>>>
>>>> Potential implementation concerns or edge cases we should address
>>>> -
>>>>
>>>> Suggestions for schema enhancements or conversion strategies
>>>> -
>>>>
>>>> Impacts on connector compatibility and testing
>>>>
>>>> Looking forward to your thoughts and input on this proposal!
>>>>
>>>> Thanks.
>>>> Link to the KIP.
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect
>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API>
>>>>
>>>>
Re: [DISCUSS] KIP-1153: Extending support for Microsecond Precision for Kafka Connect
Posted by pritam kumar <ku...@gmail.com>.
Hi Kafka Community,
Could you please review this KIP proposal when you have a chance?
Thank you,
Pritam
On Sat, May 3, 2025 at 10:53 PM pritam kumar <ku...@gmail.com>
wrote:
> Hi Chris,
> Can you please review this one too?
>
> On Sat, Apr 5, 2025 at 7:00 AM pritam kumar <ku...@gmail.com>
> wrote:
>
>> Hi Kafka Community,
>> Sorry due to some changes I had to change the link to the kip.
>> Here is the updated KIP link:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1154%3A+Extending+support+for+Microsecond+Precision+for+Kafka+Connect
>>
>> On Sat, Apr 5, 2025 at 12:14 AM pritam kumar <ku...@gmail.com>
>> wrote:
>>
>>> Hi Kafka Community,
>>>
>>> I’d like to start a discussion on KIP-1153: Extending Support for
>>> Microsecond Precision for Kafka Connect
>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect>
>>> .
>>>
>>> The primary motivation behind this KIP is to enhance the precision of
>>> timestamp handling in Kafka Connect. Currently, Kafka Connect is limited to
>>> millisecond-level precision for timestamps. However, many modern data
>>> formats and platforms have moved beyond this limitation:
>>>
>>> -
>>>
>>> Formats such as *Avro*, *Parquet*, and *ORC* support microsecond
>>> (and even nanosecond) precision. For example, Avro specifies support for
>>> timestamp-micros (spec link
>>> <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>).
>>> -
>>>
>>> Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi*
>>> offer *microsecond and nanosecond precision* for time-based fields,
>>> making millisecond precision inadequate for accurate data replication and
>>> analytics in some use cases.
>>>
>>> This gap can lead to *loss of precision* when transferring data through
>>> Kafka Connect, especially when interacting with systems that depend on
>>> high-resolution timestamps for change tracking, event ordering, or
>>> deduplication.
>>>
>>> The goal of this KIP is to:
>>>
>>> -
>>>
>>> Introduce microsecond-level timestamp handling in Kafka Connect
>>> schema and data representation.
>>> -
>>>
>>> Ensure connectors (both source and sink) can leverage this precision
>>> when supported by the underlying data systems.
>>> -
>>>
>>> Maintain backward compatibility with existing millisecond-based
>>> configurations and data.
>>>
>>> We welcome community feedback on:
>>>
>>> -
>>>
>>> Potential implementation concerns or edge cases we should address
>>> -
>>>
>>> Suggestions for schema enhancements or conversion strategies
>>> -
>>>
>>> Impacts on connector compatibility and testing
>>>
>>> Looking forward to your thoughts and input on this proposal!
>>>
>>> Thanks.
>>> Link to the KIP.
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect
>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API>
>>>
>>>
Re: [DISCUSS] KIP-1153: Extending support for Microsecond Precision for Kafka Connect
Posted by pritam kumar <ku...@gmail.com>.
Hi Chris,
Can you please review this one too?
On Sat, Apr 5, 2025 at 7:00 AM pritam kumar <ku...@gmail.com>
wrote:
> Hi Kafka Community,
> Sorry due to some changes I had to change the link to the kip.
> Here is the updated KIP link:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1154%3A+Extending+support+for+Microsecond+Precision+for+Kafka+Connect
>
> On Sat, Apr 5, 2025 at 12:14 AM pritam kumar <ku...@gmail.com>
> wrote:
>
>> Hi Kafka Community,
>>
>> I’d like to start a discussion on KIP-1153: Extending Support for
>> Microsecond Precision for Kafka Connect
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect>
>> .
>>
>> The primary motivation behind this KIP is to enhance the precision of
>> timestamp handling in Kafka Connect. Currently, Kafka Connect is limited to
>> millisecond-level precision for timestamps. However, many modern data
>> formats and platforms have moved beyond this limitation:
>>
>> -
>>
>> Formats such as *Avro*, *Parquet*, and *ORC* support microsecond (and
>> even nanosecond) precision. For example, Avro specifies support for
>> timestamp-micros (spec link
>> <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>).
>> -
>>
>> Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi*
>> offer *microsecond and nanosecond precision* for time-based fields,
>> making millisecond precision inadequate for accurate data replication and
>> analytics in some use cases.
>>
>> This gap can lead to *loss of precision* when transferring data through
>> Kafka Connect, especially when interacting with systems that depend on
>> high-resolution timestamps for change tracking, event ordering, or
>> deduplication.
>>
>> The goal of this KIP is to:
>>
>> -
>>
>> Introduce microsecond-level timestamp handling in Kafka Connect
>> schema and data representation.
>> -
>>
>> Ensure connectors (both source and sink) can leverage this precision
>> when supported by the underlying data systems.
>> -
>>
>> Maintain backward compatibility with existing millisecond-based
>> configurations and data.
>>
>> We welcome community feedback on:
>>
>> -
>>
>> Potential implementation concerns or edge cases we should address
>> -
>>
>> Suggestions for schema enhancements or conversion strategies
>> -
>>
>> Impacts on connector compatibility and testing
>>
>> Looking forward to your thoughts and input on this proposal!
>>
>> Thanks.
>> Link to the KIP.
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API>
>>
>>
Re: [DISCUSS] KIP-1153: Extending support for Microsecond Precision for Kafka Connect
Posted by pritam kumar <ku...@gmail.com>.
Hi Kafka Community,
Sorry due to some changes I had to change the link to the kip.
Here is the updated KIP link:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1154%3A+Extending+support+for+Microsecond+Precision+for+Kafka+Connect
On Sat, Apr 5, 2025 at 12:14 AM pritam kumar <ku...@gmail.com>
wrote:
> Hi Kafka Community,
>
> I’d like to start a discussion on KIP-1153: Extending Support for
> Microsecond Precision for Kafka Connect
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect>
> .
>
> The primary motivation behind this KIP is to enhance the precision of
> timestamp handling in Kafka Connect. Currently, Kafka Connect is limited to
> millisecond-level precision for timestamps. However, many modern data
> formats and platforms have moved beyond this limitation:
>
> -
>
> Formats such as *Avro*, *Parquet*, and *ORC* support microsecond (and
> even nanosecond) precision. For example, Avro specifies support for
> timestamp-micros (spec link
> <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>).
> -
>
> Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi*
> offer *microsecond and nanosecond precision* for time-based fields,
> making millisecond precision inadequate for accurate data replication and
> analytics in some use cases.
>
> This gap can lead to *loss of precision* when transferring data through
> Kafka Connect, especially when interacting with systems that depend on
> high-resolution timestamps for change tracking, event ordering, or
> deduplication.
>
> The goal of this KIP is to:
>
> -
>
> Introduce microsecond-level timestamp handling in Kafka Connect schema
> and data representation.
> -
>
> Ensure connectors (both source and sink) can leverage this precision
> when supported by the underlying data systems.
> -
>
> Maintain backward compatibility with existing millisecond-based
> configurations and data.
>
> We welcome community feedback on:
>
> -
>
> Potential implementation concerns or edge cases we should address
> -
>
> Suggestions for schema enhancements or conversion strategies
> -
>
> Impacts on connector compatibility and testing
>
> Looking forward to your thoughts and input on this proposal!
>
> Thanks.
> Link to the KIP.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API>
>
>
Re: [DISCUSS] KIP-1153: Extending support for Microsecond Precision for Kafka Connect
Posted by pritam kumar <ku...@gmail.com>.
Hi Chia-Ping Tsai,
Thankyou very much for the review.
chia_00: the description "Workers running the old version will not
recognize schemas with the new logical type names and will treat them as
raw INT64, potentially causing data misinterpretation." seems to be wrong
since the old worker will throw a NoClassDefFoundError, right?
Ans :: Sorry for the confusion but I meant old worker in the sense of old
data plane but I totally agree that if a connector/converter compiled
against the new connect-api (its bytecode references TimestampMicros) is
deployed on an old worker, the first reference to TimestampMicros.SCHEMA
links against the worker's old connect-api, which lacks the class →
NoClassDefFoundError. I have updated the KIP to reflect the same.
chia_01: does this KIP handle timestamps before 1970? The test plan
mentions negative values, but the KIP does not define how
fromLogical/toLogical convert them.
Ans :: : Yes, negative (pre-1970) values are handled correctly, but the KIP
currently only implies it via the test plan. The encoded long is a signed
count since the Unix epoch; negative = before 1970. The reason it works
cleanly is that Instant stores an epochSecond (which may be negative) and a
nanoOfSecond that is always in [0, 999_999_999] — so getNano() is never
negative, even for pre-1970 instants. I have updated to KIP with same
including a sample worked pre-1970 example.
chia_02: I guess an old sink can use TimestampConverter to convert the new
types back to Timestamp to avoid compatibility issues? If so, it would be
nice to document that in the migration path.
Ans :: Yes totally agree and worth documenting.
I request you to please go through the update KIP once and review the
changes.
Thanks,
Pritam
On Mon, Sep 28, 2026 at 7:53 PM Chia-Ping Tsai <ch...@apache.org> wrote:
> Hi
>
> Thanks for this KIP. I have a couple of comments
>
> chia_00: the description "Workers running the old version will not
> recognize schemas with the new logical type names and will treat them as
> raw INT64, potentially causing data misinterpretation." seems to be wrong
> since the old worker will throw a NoClassDefFoundError, right?
>
> chia_01: does this KIP handle timestamps before 1970? The test plan
> mentions negative values, but the KIP does not define how
> fromLogical/toLogical convert them.
>
> chia_02: I guess an old sink can use TimestampConverter to convert the new
> types back to Timestamp to avoid compatibility issues? If so, it would be
> nice to document that in the migration path.
>
> Best,
> Chia-Ping
>
> On 2025/04/04 18:44:52 pritam kumar wrote:
> > Hi Kafka Community,
> >
> > I’d like to start a discussion on KIP-1153: Extending Support for
> > Microsecond Precision for Kafka Connect
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect
> >
> > .
> >
> > The primary motivation behind this KIP is to enhance the precision of
> > timestamp handling in Kafka Connect. Currently, Kafka Connect is limited
> to
> > millisecond-level precision for timestamps. However, many modern data
> > formats and platforms have moved beyond this limitation:
> >
> > -
> >
> > Formats such as *Avro*, *Parquet*, and *ORC* support microsecond (and
> > even nanosecond) precision. For example, Avro specifies support for
> > timestamp-micros (spec link
> > <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>).
> > -
> >
> > Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi*
> > offer *microsecond and nanosecond precision* for time-based fields,
> > making millisecond precision inadequate for accurate data replication
> and
> > analytics in some use cases.
> >
> > This gap can lead to *loss of precision* when transferring data through
> > Kafka Connect, especially when interacting with systems that depend on
> > high-resolution timestamps for change tracking, event ordering, or
> > deduplication.
> >
> > The goal of this KIP is to:
> >
> > -
> >
> > Introduce microsecond-level timestamp handling in Kafka Connect schema
> > and data representation.
> > -
> >
> > Ensure connectors (both source and sink) can leverage this precision
> > when supported by the underlying data systems.
> > -
> >
> > Maintain backward compatibility with existing millisecond-based
> > configurations and data.
> >
> > We welcome community feedback on:
> >
> > -
> >
> > Potential implementation concerns or edge cases we should address
> > -
> >
> > Suggestions for schema enhancements or conversion strategies
> > -
> >
> > Impacts on connector compatibility and testing
> >
> > Looking forward to your thoughts and input on this proposal!
> >
> > Thanks.
> > Link to the KIP.
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API
> >
> >
>
Re: [DISCUSS] KIP-1153: Extending support for Microsecond Precision for Kafka Connect
Posted by Chia-Ping Tsai <ch...@apache.org>.
Hi
Thanks for this KIP. I have a couple of comments
chia_00: the description "Workers running the old version will not recognize schemas with the new logical type names and will treat them as raw INT64, potentially causing data misinterpretation." seems to be wrong since the old worker will throw a NoClassDefFoundError, right?
chia_01: does this KIP handle timestamps before 1970? The test plan mentions negative values, but the KIP does not define how fromLogical/toLogical convert them.
chia_02: I guess an old sink can use TimestampConverter to convert the new types back to Timestamp to avoid compatibility issues? If so, it would be nice to document that in the migration path.
Best,
Chia-Ping
On 2025/04/04 18:44:52 pritam kumar wrote:
> Hi Kafka Community,
>
> I’d like to start a discussion on KIP-1153: Extending Support for
> Microsecond Precision for Kafka Connect
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect>
> .
>
> The primary motivation behind this KIP is to enhance the precision of
> timestamp handling in Kafka Connect. Currently, Kafka Connect is limited to
> millisecond-level precision for timestamps. However, many modern data
> formats and platforms have moved beyond this limitation:
>
> -
>
> Formats such as *Avro*, *Parquet*, and *ORC* support microsecond (and
> even nanosecond) precision. For example, Avro specifies support for
> timestamp-micros (spec link
> <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>).
> -
>
> Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi*
> offer *microsecond and nanosecond precision* for time-based fields,
> making millisecond precision inadequate for accurate data replication and
> analytics in some use cases.
>
> This gap can lead to *loss of precision* when transferring data through
> Kafka Connect, especially when interacting with systems that depend on
> high-resolution timestamps for change tracking, event ordering, or
> deduplication.
>
> The goal of this KIP is to:
>
> -
>
> Introduce microsecond-level timestamp handling in Kafka Connect schema
> and data representation.
> -
>
> Ensure connectors (both source and sink) can leverage this precision
> when supported by the underlying data systems.
> -
>
> Maintain backward compatibility with existing millisecond-based
> configurations and data.
>
> We welcome community feedback on:
>
> -
>
> Potential implementation concerns or edge cases we should address
> -
>
> Suggestions for schema enhancements or conversion strategies
> -
>
> Impacts on connector compatibility and testing
>
> Looking forward to your thoughts and input on this proposal!
>
> Thanks.
> Link to the KIP.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API>
>