You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vamshikrishnakyatham (via GitHub)" <gi...@apache.org> on 2025/09/30 01:38:42 UTC

[I] CDC Query results are not inconsistent [hudi]

vamshikrishnakyatham opened a new issue, #14016:
URL: https://github.com/apache/hudi/issues/14016

   ### Bug Description
   
   **What happened:**
   
   CDC query returns different number of rows when ORDER BY ts_ms is added. For e.g., without ORDER BY: 9 rows returned. With ORDER BY: 6 rows returned. Some records are missing in the ordered result.
   
   **What you expected:**
   
   CDC query should return the same number of rows regardless of whether ORDER BY clause is used. The ordering should not affect which records are included in the result set.
   
   **Steps to reproduce:**
   1. Create a Hudi table with CDC enabled and perform insert/update/delete operations
   2. Run CDC query without ORDER BY: SELECT op, ts_ms, ... FROM hudi_table_changes('table_path', 'cdc', 'earliest')
   3. Run the same CDC query with ORDER BY: SELECT op, ts_ms, ... FROM hudi_table_changes('table_path', 'cdc', 'earliest') ORDER BY ts_ms ASC
   4. Observe different row counts and results between the two queries
   
   Run results:
   
   ```
   spark-sql (default)> SELECT op, ts_ms, get_json_object(before, '$.ts') AS before_ts, get_json_object(before, '$.rider') as before_rider, get_json_object(after,  '$.rider') AS after_rider FROM hudi_table_changes('file:///tmp/hudi_test_table', 'cdc', 'earliest')
                      > ;
   i       20250924110448254       NULL    NULL    rider-E
   i       20250924110628340       NULL    NULL    rider-G
   u       20250924110905831       1695516137      rider-G rider-E
   i       20250924110448254       NULL    NULL    rider-C
   u       20250924110905831       1695091554      rider-C rider-E
   i       20250924110448254       NULL    NULL    rider-A
   i       20250924110448254       NULL    NULL    rider-F
   u       20250924110551107       1695516137      rider-F rider-E
   d       20250924110611165       1695516137      rider-E NULL
   Time taken: 0.141 seconds, Fetched 9 row(s)
   
   spark-sql (default)> SELECT op, ts_ms, get_json_object(before, '$.ts') AS before_ts, get_json_object(before, '$.rider') as before_rider, get_json_obj
   ect(after,  '$.rider') AS after_rider FROM hudi_table_changes('file:///tmp/hudi_test_table', 'cdc', 'earliest') order by ts_ms asc;
   i       20250924110448254       NULL    NULL    rider-A
   u       20250924110551107       1695516137      rider-F rider-E
   d       20250924110611165       1695516137      rider-E NULL
   i       20250924110628340       NULL    NULL    rider-G
   u       20250924110905831       1695516137      rider-G rider-E
   u       20250924110905831       1695091554      rider-C rider-E
   Time taken: 0.214 seconds, Fetched 6 row(s)
   ```
   
   ### Environment
   
   **Hudi version:** 1.1
   **Query engine:** (Spark/Flink/Trino etc)
   **Relevant configs:**
   
   
   ### Logs and Stack Trace
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CDC Query results are not inconsistent [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #14016:
URL: https://github.com/apache/hudi/issues/14016#issuecomment-3349765419

   @vamshikrishnakyatham can you make a quick test to help narrow down the troubleshooting scope by disable file group reader and run the tests again, 
   ```
   set hoodie.file.group.reader.enabled=false;
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CDC Query results are not inconsistent [hudi]

Posted by "vamshikrishnakyatham (via GitHub)" <gi...@apache.org>.
vamshikrishnakyatham commented on issue #14016:
URL: https://github.com/apache/hudi/issues/14016#issuecomment-3349786592

   https://github.com/apache/hudi/pull/14017 you can use this test @cshuo 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CDC Query results are not inconsistent [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua closed issue #14016: CDC Query results are not inconsistent
URL: https://github.com/apache/hudi/issues/14016


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org