You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2025/10/14 15:33:06 UTC

[PR] fix(core): add table level validation for decimal evolution [hudi]

jonvex opened a new pull request, #14089:
URL: https://github.com/apache/hudi/pull/14089

   ### Describe the issue this Pull Request addresses
   
   Users currently can make lossy decimal evolutions
   
   ### Summary and Changelog
   
   validate that evolution is not lossy for decimal
   
   ### Impact
   
   prevent user from lossy decimal evolution
   
   ### Risk Level
   
   low
   
   ### Documentation Update
   
   N/A
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Enough context is provided in the sections above
   - [ ] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(core): add table level validation for decimal evolution [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #14089:
URL: https://github.com/apache/hudi/pull/14089#discussion_r2432500630


##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaCompatibility.java:
##########
@@ -522,15 +523,46 @@ private SchemaCompatibilityResult checkReaderEnumContainsAllWriterEnumSymbols(fi
 
     private SchemaCompatibilityResult checkFixedSize(final Schema reader, final Schema writer,
                                                      final Deque<LocationInfo> locations) {
-      SchemaCompatibilityResult result = SchemaCompatibilityResult.compatible();
       int actual = reader.getFixedSize();
       int expected = writer.getFixedSize();
       if (actual != expected) {
         String message = String.format("Fixed size field '%s' expected: %d, found: %d", getLocationName(locations, reader.getType()), expected, actual);
-        result = SchemaCompatibilityResult.incompatible(SchemaIncompatibilityType.FIXED_SIZE_MISMATCH, reader, writer,
+        return SchemaCompatibilityResult.incompatible(SchemaIncompatibilityType.FIXED_SIZE_MISMATCH, reader, writer,

Review Comment:
   Did we borrow the logic somewhere from Avro?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(core): add table level validation for decimal evolution [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14089:
URL: https://github.com/apache/hudi/pull/14089#issuecomment-3432462801

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "38f60edfd8b81434d86ed98bee459c21fb4fbb8f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9150",
       "triggerID" : "38f60edfd8b81434d86ed98bee459c21fb4fbb8f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "77596a996580814e703ab74fd5690c06f0382417",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9263",
       "triggerID" : "77596a996580814e703ab74fd5690c06f0382417",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 38f60edfd8b81434d86ed98bee459c21fb4fbb8f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9150) 
   * 77596a996580814e703ab74fd5690c06f0382417 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9263) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(core): add table level validation for decimal evolution [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14089:
URL: https://github.com/apache/hudi/pull/14089#issuecomment-3402919028

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "38f60edfd8b81434d86ed98bee459c21fb4fbb8f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9150",
       "triggerID" : "38f60edfd8b81434d86ed98bee459c21fb4fbb8f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 38f60edfd8b81434d86ed98bee459c21fb4fbb8f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9150) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(core): add table level validation for decimal evolution [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #14089:
URL: https://github.com/apache/hudi/pull/14089


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(core): add table level validation for decimal evolution [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14089:
URL: https://github.com/apache/hudi/pull/14089#issuecomment-3402451247

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "38f60edfd8b81434d86ed98bee459c21fb4fbb8f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "38f60edfd8b81434d86ed98bee459c21fb4fbb8f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 38f60edfd8b81434d86ed98bee459c21fb4fbb8f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(core): add table level validation for decimal evolution [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14089:
URL: https://github.com/apache/hudi/pull/14089#discussion_r2452168116


##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaCompatibility.java:
##########
@@ -522,15 +523,46 @@ private SchemaCompatibilityResult checkReaderEnumContainsAllWriterEnumSymbols(fi
 
     private SchemaCompatibilityResult checkFixedSize(final Schema reader, final Schema writer,
                                                      final Deque<LocationInfo> locations) {
-      SchemaCompatibilityResult result = SchemaCompatibilityResult.compatible();
       int actual = reader.getFixedSize();
       int expected = writer.getFixedSize();
       if (actual != expected) {
         String message = String.format("Fixed size field '%s' expected: %d, found: %d", getLocationName(locations, reader.getType()), expected, actual);
-        result = SchemaCompatibilityResult.incompatible(SchemaIncompatibilityType.FIXED_SIZE_MISMATCH, reader, writer,
+        return SchemaCompatibilityResult.incompatible(SchemaIncompatibilityType.FIXED_SIZE_MISMATCH, reader, writer,

Review Comment:
   I cross referenced documentation from deltalake and it aligns
   
   https://delta-docs-incubator.netlify.app/delta-type-widening/
   
   I am not sure where did Jon took inspiration from. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(core): add table level validation for decimal evolution [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14089:
URL: https://github.com/apache/hudi/pull/14089#issuecomment-3432456509

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "38f60edfd8b81434d86ed98bee459c21fb4fbb8f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9150",
       "triggerID" : "38f60edfd8b81434d86ed98bee459c21fb4fbb8f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "77596a996580814e703ab74fd5690c06f0382417",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "77596a996580814e703ab74fd5690c06f0382417",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 38f60edfd8b81434d86ed98bee459c21fb4fbb8f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9150) 
   * 77596a996580814e703ab74fd5690c06f0382417 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(core): add table level validation for decimal evolution [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14089:
URL: https://github.com/apache/hudi/pull/14089#issuecomment-3432911171

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "38f60edfd8b81434d86ed98bee459c21fb4fbb8f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9150",
       "triggerID" : "38f60edfd8b81434d86ed98bee459c21fb4fbb8f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "77596a996580814e703ab74fd5690c06f0382417",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9263",
       "triggerID" : "77596a996580814e703ab74fd5690c06f0382417",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 77596a996580814e703ab74fd5690c06f0382417 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9263) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(core): add table level validation for decimal evolution [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14089:
URL: https://github.com/apache/hudi/pull/14089#issuecomment-3402456450

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "38f60edfd8b81434d86ed98bee459c21fb4fbb8f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9150",
       "triggerID" : "38f60edfd8b81434d86ed98bee459c21fb4fbb8f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 38f60edfd8b81434d86ed98bee459c21fb4fbb8f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9150) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(core): add table level validation for decimal evolution [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #14089:
URL: https://github.com/apache/hudi/pull/14089#issuecomment-3404403760

   why in draft state? 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(core): add table level validation for decimal evolution [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #14089:
URL: https://github.com/apache/hudi/pull/14089#issuecomment-3433067463

   <img width="1236" height="571" alt="image" src="https://github.com/user-attachments/assets/b4fbc3cd-d10d-449e-b5e5-7f7c2a714419" />
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org