You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "mani-sethu (via GitHub)" <gi...@apache.org> on 2025/06/11 10:11:53 UTC

[I] Delta Stats getting wrongly populated after merge [incubator-gluten]

mani-sethu opened a new issue, #9945:
URL: https://github.com/apache/incubator-gluten/issues/9945

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   [Expected behavior] 
   I am running a merge job with gluten and without gluten and following are the merge statistics on delta log respectively
   
   **With Gluten:**
   `object
   numTargetRowsCopied: "1056"
   numTargetRowsDeleted: "0"
   numTargetFilesAdded: "1"
   numTargetBytesAdded: "3954"
   numTargetBytesRemoved: "3215"
   numTargetDeletionVectorsAdded: "0"
   numTargetRowsMatchedUpdated: "528"
   executionTimeMs: "158406"
   numTargetRowsInserted: "528"
   numTargetRowsMatchedDeleted: "0"
   numTargetDeletionVectorsUpdated: "0"
   scanTimeMs: "4326"
   numTargetRowsUpdated: "528"
   numOutputRows: "2112"
   numTargetDeletionVectorsRemoved: "0"
   numTargetRowsNotMatchedBySourceUpdated: "0"
   numTargetChangeFilesAdded: "0"
   numSourceRows: "528"
   numTargetFilesRemoved: "1"
   numTargetRowsNotMatchedBySourceDeleted: "0"
   rewriteTimeMs: "1489"`
   
   **Without Gluten**
   numTargetRowsCopied: "0"
   numTargetRowsDeleted: "0"
   numTargetFilesAdded: "1"
   numTargetBytesAdded: "3215"
   numTargetBytesRemoved: "3225"
   numTargetDeletionVectorsAdded: "0"
   numTargetRowsMatchedUpdated: "528"
   executionTimeMs: "132701"
   materializeSourceTimeMs: "114262"
   numTargetRowsInserted: "0"
   numTargetRowsMatchedDeleted: "0"
   numTargetDeletionVectorsUpdated: "0"
   scanTimeMs: "16822"
   numTargetRowsUpdated: "528"
   numOutputRows: "528"
   numTargetDeletionVectorsRemoved: "0"
   numTargetRowsNotMatchedBySourceUpdated: "0"
   numTargetChangeFilesAdded: "0"
   numSourceRows: "528"
   numTargetFilesRemoved: "1"
   numTargetRowsNotMatchedBySourceDeleted: "0"
   rewriteTimeMs: "1593"
   
   Where the second is actually correct. There is no change in the underlying data (both source and destination table). I have tested this for multiple jobs where this is the case.
   
   ### Gluten version
   
   main branch
   
   ### Spark version
   
   Spark-3.5.x
   
   ### Spark configurations
   
   _No response_
   
   ### System information
   
   Gluten Version: 1.5.0-SNAPSHOT
   Commit: c38c431951b3597aa203838bd7d0592974c15ceb
   CMake Version: 3.28.3
   System: Linux-6.8.0-1030-gcp
   Arch: x86_64
   CPU Name: Model name:                           Intel(R) Xeon(R) CPU @ 2.80GHz
   C++ Compiler: /usr/bin/c++
   C++ Compiler Version: 11.4.0
   C Compiler: /usr/bin/cc
   C Compiler Version: 11.4.0
   CMake Prefix Path: /usr/local;/usr;/;/usr/local/lib/python3.10/dist-packages/cmake/data;/usr/local;/usr/X11R6;/usr/pkg;/opt
   
   ### Relevant logs
   
   ```bash
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Delta Stats getting wrongly populated after merge [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on issue #9945:
URL: https://github.com/apache/gluten/issues/9945#issuecomment-5053160846

   @mani-sethu @zhouyuan this is a known issue: https://github.com/apache/gluten/issues/9003


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org