You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 12:42:45 UTC

[I] Flink 1.18 array values incorrect in fg reader test [hudi]

hudi-bot opened a new issue, #17092:
URL: https://github.com/apache/hudi/issues/17092

   {code:java}
    <{"timestamp": 0, "_row_key": "30f916be-06d6-4ef3-a9fb-9dddc40e8abd", "partition_path": "any_partition", "trip_type": "BLACK", "rider": "rider-001", "driver": "driver-001", "begin_lat": 0.7585664392201925, "begin_lon": 0.37789349980839115, "end_lat": 0.7757098794857531, "end_lon": 0.1730947142293684, "distance_in_meters": 1356260410, "seconds_since_epoch": -9002173267385180332, "weight": 0.9950906, "nation": "Canada", "current_date": 13, "current_ts": 1182257224, "height": [0, 0, 11, 121, -97], "city_to_state": {"LA": "CA"}, "fare": {"amount": 89.22144149794701, "currency": "USD"}, "tip_history": [{"amount": 78.67847272460095, "currency": "USD"}], "customField0": 1299566198, "customField1": 5724776041695989483, "customField2": 0.7481167, "customField3": 0.2553663812411848, "customField4": "25db73af-fdf9-4e04-b47d-ce44882bf362", "customField5": "877c2613-f2a1-4afd-af4a-863814dec38b", "_hoodie_is_deleted": false}>  <{"timestamp": 0, "_row_key": "30f916be-06d6-4ef3-a9fb-9dddc40e8abd
 ", "partition_path": "any_partition", "trip_type": "BLACK", "rider": "rider-001", "driver": "driver-001", "begin_lat": 0.7585664392201925, "begin_lon": 0.37789349980839115, "end_lat": 0.7757098794857531, "end_lon": 0.1730947142293684, "distance_in_meters": 1356260410, "seconds_since_epoch": -9002173267385180332, "weight": 0.9950906, "nation": "Canada", "current_date": 13, "current_ts": 1182257224, "height": [0, 0, 11, 121, -97], "city_to_state": {"LA": "CA"}, "fare": {"amount": 89.22144149794701, "currency": "USD"}, "tip_history": [{"amount": 4.5791476348245315, "currency": "USD"}], "customField0": 1299566198, "customField1": 5724776041695989483, "customField2": 0.7481167, "customField3": 0.2553663812411848, "customField4": "25db73af-fdf9-4e04-b47d-ce44882bf362", "customField5": "877c2613-f2a1-4afd-af4a-863814dec38b", "_hoodie_is_deleted": false}>
    {code}
   every field is the same except for the tip_history. This only happens on Flink 1.18 and not 1.20. I added a config to HoodieTestDataGenerator.SchemaEvolutionConfigs that can be enabled to expose this issue. I have marked it as a TODO in the code
   
   I read the data in spark to ensure it wasn't an issue with the writer:
   {code:java}
   scala> spark.read.format("hudi").load("/var/folders/d0/l7mfhzl1661byhh3mbyg5fv00000gn/T/junit-5387917324981749197/").select("_row_key","tip_history").show(100,false)
   25/07/18 14:12:07 WARN DFSPropertiesConfiguration: Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
   25/07/18 14:12:07 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
   # WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed.]
   +------------------------------------+---------------------------+              
   |_row_key                            |tip_history                |
   +------------------------------------+---------------------------+
   |30f916be-06d6-4ef3-a9fb-9dddc40e8abd|[{78.67847272460095, USD}] |
   |d0efa8ad-5f8e-4ee2-ace7-963585aa2b7f|[{94.05111350973606, USD}] |
   |f0dd358d-de15-4c67-9e3b-f006263d9286|[{30.4615021527871, USD}]  |
   |e6db310d-71ca-4147-9f4f-cbf7a2675016|[{3.853885568405202, USD}] |
   |27819509-b6ac-45ee-8a7a-a031b2ef9a05|[{4.5791476348245315, USD}]|
   +------------------------------------+---------------------------+ {code}
   The tip history matches the expected 78.67847272460095 for record 30f916be-06d6-4ef3-a9fb-9dddc40e8abd.
   
   Additionally, we can see that the value that is showing up in the record exactly matches the value in the last record `27819509-b6ac-45ee-8a7a-a031b2ef9a05`
   
    
   
   I went back and looked at all the expected records and they all have that same value in their tip history
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-9603
   - Type: Bug
   - Affects version(s):
     - 1.0.2
   - Fix version(s):
     - 1.1.1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org