[jira] [Resolved] (HUDI-2745) Record count does not match input after compaction is scheduled when running Hudi Kafka Connect sink

["Rajesh Mahindra (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rajesh Mahindra resolved HUDI-2745.
-----------------------------------

> Record count does not match input after compaction is scheduled when running Hudi Kafka Connect sink
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HUDI-2745
>                 URL: https://issues.apache.org/jira/browse/HUDI-2745
>             Project: Apache Hudi
>          Issue Type: Bug
>          Components: Compaction
>            Reporter: Ethan Guo
>            Assignee: Ethan Guo
>            Priority: Blocker
>             Fix For: 0.10.0
>
>
> Spark Shell command to do snapshot query:
> {code:java}
> val basePath = "/tmp/hoodie/hudi-test-topic"
> val df = spark.read.format("hudi").load(basePath)
> df.createOrReplaceTempView("hudi_test_table")
> spark.sql("select count(*) from hudi_test_table").show() {code}
> Two cases of count mismatch:
> (1) Compaction scheduled, more deltacommits later on: the count does not match input size.  After compaction is executed.  The count becomes correct.
> (2) Clustering scheduled, more deltacommits later on: the count is correct, equal to the input size.  After clustering is executed, the count drops and becomes incorrect.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)