You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/08/04 21:05:05 UTC

[PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

yihua opened a new pull request, #19512:
URL: https://github.com/apache/hudi/pull/19512

   ### Describe the issue this Pull Request addresses
   
   `HoodieRowParquetWriteSupport`, the Spark row write support used by bulk insert, clustering, and file-group-reader-based MOR compaction, sizes decimal `FIXED_LEN_BYTE_ARRAY` columns from `Decimal.minBytesForPrecision()[precision]` and discards the declared Avro `fixed` size. A Spark `DecimalType` carries only precision and scale, so an Avro `fixed(N)` decimal whose declared size is wider than the precision-minimal width (for example a `fixed(10)` decimal(20,2), whose minimal width is 9) is written as `FIXED_LEN_BYTE_ARRAY(9)`. This diverges from the Avro write path (inserts), which preserves the declared 10. Mixing the two paths (insert then compaction, or upsert then clustering) leaves a table with mixed-width decimal columns under one logical schema.
   
   ### Summary and Changelog
   
   Add a helper `decimalFixedLen(HoodieSchema resolvedSchema, int precision)`: when the resolved schema is a `HoodieSchema.Decimal` backed by an Avro `fixed` type (`isFixed()`), use its `getFixedSize()`; otherwise fall back to `Decimal.minBytesForPrecision()[precision]`. It is applied at both decimal branches that already receive the resolved `HoodieSchema`:
   
   - the output Parquet type in the schema converter (`convertField`), so the column is declared `FIXED_LEN_BYTE_ARRAY(N)`, and
   - the value writer (`makeWriter`), so the written value bytes are padded to the same `N`.
   
   The padding buffer is now resolved once per column instead of once per record. The shared `decimalBuffer` is only sized for the precision-minimal maximum, so a declared `fixed(N)` wider than that gets its own per-column buffer, which also closes a latent overflow when honoring a wide fixed size.
   
   No code copied.
   
   ### Impact
   
   Behavior change on the Spark row write path: a decimal declared as an Avro `fixed(N)` is now written as `FIXED_LEN_BYTE_ARRAY(N)`, matching the table schema and the Avro write path, instead of the precision-minimal width. Existing narrower files still read correctly (Spark decodes decimals by precision). The scope is limited to decimals backed by an Avro `fixed` type; `bytes`-backed decimals and pure-Spark tables (no Avro fixed size) are unaffected.
   
   ### Risk Level
   
   low
   
   The change is confined to the decimal branch and only diverges from prior behavior when the resolved schema is a `fixed`-backed decimal. A new unit test asserts an Avro `fixed(10)` decimal(20,2) is declared `FIXED_LEN(10)` and a `bytes` decimal(20,2) stays `FIXED_LEN(9)`. A new functional MOR test builds a table whose Avro schema declares a `fixed(10)` decimal, runs compaction and row-writer clustering over two file groups, asserts the rewritten base files stay `FIXED_LEN_BYTE_ARRAY(10)`, and asserts the commit-metadata table schema is unchanged after both operations. Both fixed-width assertions fail (expected 10 but was 9) when the fix is reverted.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5198980435

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106",
       "triggerID" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108",
       "triggerID" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "triggerType" : "PUSH"
     }, {
       "hash" : "854bc3133a435273dba657d5a85766025f51b954",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "854bc3133a435273dba657d5a85766025f51b954",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f0cb8b8ae961bbf32464c7c6d8495014beda3cee Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106) 
   * bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108) 
   * 854bc3133a435273dba657d5a85766025f51b954 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5185493261

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 94edcff35053f6db8255214f668fecfaf144f0a5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5198124077

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106",
       "triggerID" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 94edcff35053f6db8255214f668fecfaf144f0a5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065) 
   * f0cb8b8ae961bbf32464c7c6d8495014beda3cee Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5199368190

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106",
       "triggerID" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108",
       "triggerID" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "triggerType" : "PUSH"
     }, {
       "hash" : "854bc3133a435273dba657d5a85766025f51b954",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "854bc3133a435273dba657d5a85766025f51b954",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5fff3a60b9ca2f4477c1d25965b2a30d78316e00",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16109",
       "triggerID" : "5fff3a60b9ca2f4477c1d25965b2a30d78316e00",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108) 
   * 854bc3133a435273dba657d5a85766025f51b954 UNKNOWN
   * 5fff3a60b9ca2f4477c1d25965b2a30d78316e00 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16109) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5198835343

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106",
       "triggerID" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108",
       "triggerID" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f0cb8b8ae961bbf32464c7c6d8495014beda3cee Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106) 
   * bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5198020056

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 94edcff35053f6db8255214f668fecfaf144f0a5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065) 
   * f0cb8b8ae961bbf32464c7c6d8495014beda3cee UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5198790897

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106",
       "triggerID" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f0cb8b8ae961bbf32464c7c6d8495014beda3cee Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106) 
   * bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5200054021

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106",
       "triggerID" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108",
       "triggerID" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "triggerType" : "PUSH"
     }, {
       "hash" : "854bc3133a435273dba657d5a85766025f51b954",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "854bc3133a435273dba657d5a85766025f51b954",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5fff3a60b9ca2f4477c1d25965b2a30d78316e00",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16109",
       "triggerID" : "5fff3a60b9ca2f4477c1d25965b2a30d78316e00",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e9b76040bad6293e6f990ea5f0da488ddc6052e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16113",
       "triggerID" : "5e9b76040bad6293e6f990ea5f0da488ddc6052e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 854bc3133a435273dba657d5a85766025f51b954 UNKNOWN
   * 5fff3a60b9ca2f4477c1d25965b2a30d78316e00 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16109) 
   * 5e9b76040bad6293e6f990ea5f0da488ddc6052e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16113) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5184802776

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 94edcff35053f6db8255214f668fecfaf144f0a5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5199405204

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106",
       "triggerID" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108",
       "triggerID" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "triggerType" : "PUSH"
     }, {
       "hash" : "854bc3133a435273dba657d5a85766025f51b954",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "854bc3133a435273dba657d5a85766025f51b954",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5fff3a60b9ca2f4477c1d25965b2a30d78316e00",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16109",
       "triggerID" : "5fff3a60b9ca2f4477c1d25965b2a30d78316e00",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e9b76040bad6293e6f990ea5f0da488ddc6052e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5e9b76040bad6293e6f990ea5f0da488ddc6052e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108) 
   * 854bc3133a435273dba657d5a85766025f51b954 UNKNOWN
   * 5fff3a60b9ca2f4477c1d25965b2a30d78316e00 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16109) 
   * 5e9b76040bad6293e6f990ea5f0da488ddc6052e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.

voonhous merged PR #19512:
URL: https://github.com/apache/hudi/pull/19512


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.

hudi-agent commented on code in PR #19512:
URL: https://github.com/apache/hudi/pull/19512#discussion_r3724562483


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -482,6 +482,20 @@ private void writeFields(InternalRow row, StructType schema, ValueWriter[] field
     }
   }
 
+  private static int decimalFixedLen(HoodieSchema resolvedSchema, int precision) {

Review Comment:
   🤖 nit: `decimalFixedLen` is a bit ambiguous -- "fixed" could refer to the Avro `fixed` type or just mean "determined length". Something like `resolveDecimalByteLength` might read more clearly at the two call sites.
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/table/functional/TestHoodieSparkMergeOnReadTableCompaction.java:
##########
@@ -442,4 +463,137 @@ private void commitToTable(String instant, List<WriteStatus> writeStatuses) {
         client.commitStats(instant, writeStats, Option.empty(), metaClient.getCommitActionType());
     assertTrue(committed);
   }
+
+  // Avro fixed(10) decimal(20,2). 10 is wider than the precision-minimal width (9 for precision
+  // 20), so Spark's own DecimalType->Avro conversion would emit fixed(9); the declared 10 only
+  // survives if the write path honors the Avro fixed size. Do not derive this schema from a
+  // DataFrame, which would drop the fixed size.
+  private static final String FIXED10_DECIMAL_SCHEMA =
+      "{\"type\":\"record\",\"name\":\"decimalRec\",\"fields\":["
+          + "{\"name\":\"_row_key\",\"type\":\"string\"},"
+          + "{\"name\":\"partition_path\",\"type\":\"string\"},"
+          + "{\"name\":\"ts\",\"type\":\"long\"},"
+          + "{\"name\":\"dec\",\"type\":{\"type\":\"fixed\",\"name\":\"decFixed\",\"size\":10,"
+          + "\"logicalType\":\"decimal\",\"precision\":20,\"scale\":2}}]}";
+
+  private static final String DECIMAL_PARTITION = "p1";
+  private static final int EXPECTED_DECIMAL_FIXED_LEN = 10;
+
+  @Test
+  void testDecimalFixedWidthPreservedAfterCompactionAndClustering() throws Exception {
+    Properties props = getPropertiesForKeyGen(true);
+    Properties rowWriterProps = new Properties();
+    rowWriterProps.put("hoodie.datasource.write.row.writer.enable", "true");
+    HoodieWriteConfig config = HoodieWriteConfig.newBuilder()
+        .forTable("test-decimal-fixed")
+        .withPath(basePath())
+        .withSchema(FIXED10_DECIMAL_SCHEMA)
+        .withParallelism(2, 2)
+        .withPreCombineField("ts")
+        .withProperties(rowWriterProps)
+        .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+            .withMaxNumDeltaCommitsBeforeCompaction(1)
+            .compactionSmallFileSize(0)
+            .withInlineCompaction(false)
+            .build())
+        .withClusteringConfig(HoodieClusteringConfig.newBuilder()
+            .withClusteringMaxNumGroups(10)
+            .withClusteringTargetPartitions(0)
+            .withInlineClustering(false)
+            .withInlineClusteringNumCommits(1)
+            .build())
+        .build();
+    props.putAll(config.getProps());
+
+    metaClient = getHoodieMetaClient(HoodieTableType.MERGE_ON_READ, props);
+    client = getHoodieWriteClient(config);
+
+    // two insert commits (small-file size 0 forces a fresh file group each) create two base-file
+    // groups, both written via the Avro path at fixed(10)
+    String instant1 = WriteClientTestUtils.createNewInstantTime();
+    writeData(instant1, buildDecimalRecords(0, 10, 1L, new BigDecimal("123456789.12")), true);
+    String instant2 = WriteClientTestUtils.createNewInstantTime();
+    writeData(instant2, buildDecimalRecords(10, 10, 1L, new BigDecimal("223456789.34")), true);
+    // update every key so both groups accumulate log files for compaction to merge
+    String instant3 = WriteClientTestUtils.createNewInstantTime();
+    writeData(instant3, buildDecimalRecords(0, 20, 2L, new BigDecimal("323456789.56")), true);
+
+    // precondition: both file groups must carry log files, else compaction is a no-op and would not
+    // exercise the row-writer merge path
+    metaClient = HoodieTableMetaClient.reload(metaClient);
+    HoodieTable hoodieTable = HoodieSparkTable.create(config, context(), metaClient);
+    hoodieTable.getHoodieView().sync();
+    List<FileSlice> latestSlices =
+        hoodieTable.getHoodieView().getLatestFileSlices(DECIMAL_PARTITION).collect(Collectors.toList());
+    assertEquals(2, latestSlices.size(), "expected two file groups before compaction");
+    assertTrue(latestSlices.stream().allMatch(slice -> slice.getLogFiles().findAny().isPresent()),
+        "each file group must have log files for compaction to merge");
+    HoodieSchema tableSchema = new TableSchemaResolver(metaClient).getTableSchema(false);
+
+    // compaction rewrites both base files through the Spark record type
+    String compactionInstant = (String) client.scheduleCompaction(Option.empty()).get();
+    HoodieWriteMetadata compactionResult = client.compact(compactionInstant);
+    client.commitCompaction(compactionInstant, compactionResult, Option.empty());

Review Comment:
   🤖 nit: `assertTrue(!clusterStats.isEmpty(), ...)` is a bit awkward -- `assertFalse(clusterStats.isEmpty(), ...)` reads more naturally.
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5199217573

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106",
       "triggerID" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108",
       "triggerID" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "triggerType" : "PUSH"
     }, {
       "hash" : "854bc3133a435273dba657d5a85766025f51b954",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "854bc3133a435273dba657d5a85766025f51b954",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5fff3a60b9ca2f4477c1d25965b2a30d78316e00",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16109",
       "triggerID" : "5fff3a60b9ca2f4477c1d25965b2a30d78316e00",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f0cb8b8ae961bbf32464c7c6d8495014beda3cee Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106) 
   * bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108) 
   * 854bc3133a435273dba657d5a85766025f51b954 UNKNOWN
   * 5fff3a60b9ca2f4477c1d25965b2a30d78316e00 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16109) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.

codecov-commenter commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5198088471

   ## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19512?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 77.16%. Comparing base ([`1f8e53a`](https://app.codecov.io/gh/apache/hudi/commit/1f8e53a703fa0d3adb1a9d33855c8e1b69bc8612?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f0cb8b8`](https://app.codecov.io/gh/apache/hudi/commit/f0cb8b8ae961bbf32464c7c6d8495014beda3cee?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 4 commits behind head on master.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #19512      +/-   ##
   ============================================
   - Coverage     77.17%   77.16%   -0.01%     
   + Complexity    33948    33944       -4     
   ============================================
     Files          2575     2576       +1     
     Lines        143397   143370      -27     
     Branches      17579    17575       -4     
   ============================================
   - Hits         110662   110636      -26     
   - Misses        24476    24477       +1     
   + Partials       8259     8257       -2     
   ```
   
   | [Components](https://app.codecov.io/gh/apache/hudi/pull/19512/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19512/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.39% <0.00%> (-0.02%)` | :arrow_down: |
   | [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19512/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.55% <ø> (+<0.01%)` | :arrow_up: |
   | [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19512/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.97% <ø> (+<0.01%)` | :arrow_up: |
   | [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19512/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.10% <ø> (ø)` | |
   | [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19512/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.63% <ø> (ø)` | |
   | [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19512/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.32% <ø> (ø)` | |
   | [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19512/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.74% <ø> (ø)` | |
   | [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19512/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.95% <ø> (ø)` | |
   | [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19512/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.60% <ø> (ø)` | |
   | [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19512/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.44% <ø> (ø)` | |
   | [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19512/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.98% <ø> (-0.02%)` | :arrow_down: |
   | [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19512/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
   
   | [Flag](https://app.codecov.io/gh/apache/hudi/pull/19512/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19512/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.75% <ø> (+0.05%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19512?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...i/io/storage/row/HoodieRowParquetWriteSupport.java](https://app.codecov.io/gh/apache/hudi/pull/19512?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2Fstorage%2Frow%2FHoodieRowParquetWriteSupport.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9yb3cvSG9vZGllUm93UGFycXVldFdyaXRlU3VwcG9ydC5qYXZh) | `80.13% <ø> (+0.04%)` | :arrow_up: |
   </details>
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5199590216

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106",
       "triggerID" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108",
       "triggerID" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "triggerType" : "PUSH"
     }, {
       "hash" : "854bc3133a435273dba657d5a85766025f51b954",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "854bc3133a435273dba657d5a85766025f51b954",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5fff3a60b9ca2f4477c1d25965b2a30d78316e00",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16109",
       "triggerID" : "5fff3a60b9ca2f4477c1d25965b2a30d78316e00",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e9b76040bad6293e6f990ea5f0da488ddc6052e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16113",
       "triggerID" : "5e9b76040bad6293e6f990ea5f0da488ddc6052e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108) 
   * 854bc3133a435273dba657d5a85766025f51b954 UNKNOWN
   * 5fff3a60b9ca2f4477c1d25965b2a30d78316e00 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16109) 
   * 5e9b76040bad6293e6f990ea5f0da488ddc6052e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16113) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5198745401

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106",
       "triggerID" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f0cb8b8ae961bbf32464c7c6d8495014beda3cee Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #19512:
URL: https://github.com/apache/hudi/pull/19512#discussion_r3725273941


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -482,6 +482,20 @@ private void writeFields(InternalRow row, StructType schema, ValueWriter[] field
     }
   }
 
+  private static int decimalFixedLen(HoodieSchema resolvedSchema, int precision) {

Review Comment:
   Done, renamed to `resolveDecimalByteLength`.



##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/table/functional/TestHoodieSparkMergeOnReadTableCompaction.java:
##########
@@ -442,4 +463,137 @@ private void commitToTable(String instant, List<WriteStatus> writeStatuses) {
         client.commitStats(instant, writeStats, Option.empty(), metaClient.getCommitActionType());
     assertTrue(committed);
   }
+
+  // Avro fixed(10) decimal(20,2). 10 is wider than the precision-minimal width (9 for precision
+  // 20), so Spark's own DecimalType->Avro conversion would emit fixed(9); the declared 10 only
+  // survives if the write path honors the Avro fixed size. Do not derive this schema from a
+  // DataFrame, which would drop the fixed size.
+  private static final String FIXED10_DECIMAL_SCHEMA =
+      "{\"type\":\"record\",\"name\":\"decimalRec\",\"fields\":["
+          + "{\"name\":\"_row_key\",\"type\":\"string\"},"
+          + "{\"name\":\"partition_path\",\"type\":\"string\"},"
+          + "{\"name\":\"ts\",\"type\":\"long\"},"
+          + "{\"name\":\"dec\",\"type\":{\"type\":\"fixed\",\"name\":\"decFixed\",\"size\":10,"
+          + "\"logicalType\":\"decimal\",\"precision\":20,\"scale\":2}}]}";
+
+  private static final String DECIMAL_PARTITION = "p1";
+  private static final int EXPECTED_DECIMAL_FIXED_LEN = 10;
+
+  @Test
+  void testDecimalFixedWidthPreservedAfterCompactionAndClustering() throws Exception {
+    Properties props = getPropertiesForKeyGen(true);
+    Properties rowWriterProps = new Properties();
+    rowWriterProps.put("hoodie.datasource.write.row.writer.enable", "true");
+    HoodieWriteConfig config = HoodieWriteConfig.newBuilder()
+        .forTable("test-decimal-fixed")
+        .withPath(basePath())
+        .withSchema(FIXED10_DECIMAL_SCHEMA)
+        .withParallelism(2, 2)
+        .withPreCombineField("ts")
+        .withProperties(rowWriterProps)
+        .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+            .withMaxNumDeltaCommitsBeforeCompaction(1)
+            .compactionSmallFileSize(0)
+            .withInlineCompaction(false)
+            .build())
+        .withClusteringConfig(HoodieClusteringConfig.newBuilder()
+            .withClusteringMaxNumGroups(10)
+            .withClusteringTargetPartitions(0)
+            .withInlineClustering(false)
+            .withInlineClusteringNumCommits(1)
+            .build())
+        .build();
+    props.putAll(config.getProps());
+
+    metaClient = getHoodieMetaClient(HoodieTableType.MERGE_ON_READ, props);
+    client = getHoodieWriteClient(config);
+
+    // two insert commits (small-file size 0 forces a fresh file group each) create two base-file
+    // groups, both written via the Avro path at fixed(10)
+    String instant1 = WriteClientTestUtils.createNewInstantTime();
+    writeData(instant1, buildDecimalRecords(0, 10, 1L, new BigDecimal("123456789.12")), true);
+    String instant2 = WriteClientTestUtils.createNewInstantTime();
+    writeData(instant2, buildDecimalRecords(10, 10, 1L, new BigDecimal("223456789.34")), true);
+    // update every key so both groups accumulate log files for compaction to merge
+    String instant3 = WriteClientTestUtils.createNewInstantTime();
+    writeData(instant3, buildDecimalRecords(0, 20, 2L, new BigDecimal("323456789.56")), true);
+
+    // precondition: both file groups must carry log files, else compaction is a no-op and would not
+    // exercise the row-writer merge path
+    metaClient = HoodieTableMetaClient.reload(metaClient);
+    HoodieTable hoodieTable = HoodieSparkTable.create(config, context(), metaClient);
+    hoodieTable.getHoodieView().sync();
+    List<FileSlice> latestSlices =
+        hoodieTable.getHoodieView().getLatestFileSlices(DECIMAL_PARTITION).collect(Collectors.toList());
+    assertEquals(2, latestSlices.size(), "expected two file groups before compaction");
+    assertTrue(latestSlices.stream().allMatch(slice -> slice.getLogFiles().findAny().isPresent()),
+        "each file group must have log files for compaction to merge");
+    HoodieSchema tableSchema = new TableSchemaResolver(metaClient).getTableSchema(false);
+
+    // compaction rewrites both base files through the Spark record type
+    String compactionInstant = (String) client.scheduleCompaction(Option.empty()).get();
+    HoodieWriteMetadata compactionResult = client.compact(compactionInstant);
+    client.commitCompaction(compactionInstant, compactionResult, Option.empty());

Review Comment:
   Done, switched to `assertFalse(clusterStats.isEmpty(), ...)`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5199156306

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16065",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106",
       "triggerID" : "f0cb8b8ae961bbf32464c7c6d8495014beda3cee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108",
       "triggerID" : "bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888",
       "triggerType" : "PUSH"
     }, {
       "hash" : "854bc3133a435273dba657d5a85766025f51b954",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "854bc3133a435273dba657d5a85766025f51b954",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5fff3a60b9ca2f4477c1d25965b2a30d78316e00",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5fff3a60b9ca2f4477c1d25965b2a30d78316e00",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f0cb8b8ae961bbf32464c7c6d8495014beda3cee Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16106) 
   * bd6e7e2b0a74bbee6ae14fc149272bb86e6c5888 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16108) 
   * 854bc3133a435273dba657d5a85766025f51b954 UNKNOWN
   * 5fff3a60b9ca2f4477c1d25965b2a30d78316e00 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(spark): preserve the Avro fixed-size decimal width in the Spark row write support [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19512:
URL: https://github.com/apache/hudi/pull/19512#issuecomment-5184707827

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "94edcff35053f6db8255214f668fecfaf144f0a5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 94edcff35053f6db8255214f668fecfaf144f0a5 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org