Re: [PR] perf(spark): fast-path SELECT count(*) on COW tables via parquet footer row counts (#18769) [hudi]

["txwyy123 (via GitHub)" <gi...@apache.org>]

txwyy123 commented on PR #18770:
URL: https://github.com/apache/hudi/pull/18770#issuecomment-5902532002

   Independent validation on PR head `b8718e8` (Spark 3.5.5 / Scala 2.12 / JDK 17, macOS arm64): the 18-module reactor test-compiles successfully. Running `TestCountStarFastPath` gives 7 passed, 1 failed. `testCountStarFastPathReadsLessThanFullScan` fails reproducibly (including Surefire reruns): count(*) reads about 4,411,000 bytes vs SELECT * about 4,806,000 bytes, so the <1/2 assertion does not hold on this fixture.
   
   I used a local-only negative control, setting `canUseCountFastPath = false` and rerunning the same test: count(*) then read about 4,448,000 bytes vs SELECT * about 4,806,000 bytes. The fast path appears to save ~36 KB here, but the ~4.4 MB background I/O dominates the comparison. I also tried a wider, high-entropy payload: the ratio assertion passed both with the fast path enabled *and disabled*, so merely widening the fixture would make the test green without proving this path is exercised.
   
   Could we keep the correctness tests but replace this ratio-based CI gate with a deterministic routing test (and keep throughput comparison as a benchmark)? If continuing to use `SparkListener` for bytesRead, it would also be worth scoping events to the query and waiting for listener delivery before asserting. I made no changes to this PR; both controls were only in an isolated local copy. I cannot tell whether the historic Azure failure has this same cause because that build's logs are no longer available.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org