You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "kumarpritam863 (via GitHub)" <gi...@apache.org> on 2026/10/19 11:48:03 UTC

[PR] [FLINK-41027][format/avro] Add an option to map time-millis and time-micros to INT and BIGINT [flink]

kumarpritam863 opened a new pull request, #29522:
URL: https://github.com/apache/flink/pull/29522

   ## What
   
   An option, off by default, to map the Avro logical types `time-millis` and `time-micros` to plain numbers (`INT` and `BIGINT`) instead of to `TIME(3)` and `TIME(6)`.
   
   ## Why
   
   - A Flink `TIME` stores milliseconds since midnight in an int, so the microseconds of a `time-micros` are lost as a `TIME`. As a `BIGINT` they are kept.
   - A `TIME(6)` has no Avro schema: `AvroSchemaConverter.convertToSchema` refuses a precision above 3. In `avro-confluent` with a `schema`, a `time-micros` field therefore could not be used at all: a `TIME(6)` column failed before the provided schema was looked at, and a `BIGINT` column did not match it.
   
   ## Change
   
   - `AvroSchemaConverter.convertToDataType(String, boolean, boolean)` and `convertToTypeInfo(String, boolean, boolean)`: with `timeAsNumber`, `time-millis` becomes `INT` / `Types.INT` and `time-micros` becomes `BIGINT` / `Types.LONG`, also nested in arrays, maps and nullable unions, with the legacy and the new timestamp mapping. The existing overloads keep the current mapping.
   - `AvroToVariantDataConverters.createVariantConverter(Schema, boolean)`: with `timeAsNumber`, `time-millis` stays the Avro int instead of being converted to microseconds of day and stored as a long. `time-micros` is a long either way.
   - `avro-confluent` option `time-mapping.as-number` (default `false`), used when the table schema is checked against the provided `schema`. Documented in the English and Chinese pages.
   - Related fix in `RegistryAvroFormatFactory`: the Avro schema of the table was built with `Optional.orElse`, which runs even when a `schema` is provided, so a `TIME(6)` column failed there. It now uses `orElseGet`.
   
   `debezium-avro-confluent` is not changed. Nothing else changes with the option off, apart from the `orElseGet` fix.
   
   ## Testing
   
   - `AvroSchemaConverterTest`: the default mapping, the option, nested and nullable fields, both timestamp mappings, type info, and that timestamps and dates are untouched.
   - `AvroToVariantDataConvertersTest`: default and option, in a record, a union and an array.
   - `RegistryAvroFormatFactoryTest`: INT/BIGINT columns with the option (encoder and decoder), the same columns without it (rejected), TIME columns without it (accepted, which needs the `orElseGet` fix; with `orElse` the test errors) and with it (rejected).
   
     ---
   
     ##### Was generative AI tooling used to co-author this PR?
   
     - [X] Yes (please specify the tool below)
   
     Generated-by: Claude Code (Anthropic Claude Sonnet 5.5)
   
     ---
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-41027][format/avro] Add an option to map time-millis and time-micros to INT and BIGINT [flink]

Posted by "kumarpritam863 (via GitHub) via issues" <is...@flink.apache.org>.
kumarpritam863 commented on PR #29522:
URL: https://github.com/apache/flink/pull/29522#issuecomment-6758735810

   Hi @twalthr can you please review this once. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-41027][format/avro] Add an option to map time-millis and time-micros to INT and BIGINT [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #29522:
URL: https://github.com/apache/flink/pull/29522#issuecomment-6266824879

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "c2d3c9acae26ae9642c7550e5e7d1222baff26ef",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c2d3c9acae26ae9642c7550e5e7d1222baff26ef",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c2d3c9acae26ae9642c7550e5e7d1222baff26ef UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-41027][format/avro] Add an option to map time-millis and time-micros to INT and BIGINT [flink]

Posted by "kumarpritam863 (via GitHub)" <gi...@apache.org>.
kumarpritam863 commented on PR #29522:
URL: https://github.com/apache/flink/pull/29522#issuecomment-6283522884

   @mxm can you please review this once.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org