You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "mani-sethu (via GitHub)" <gi...@apache.org> on 2025/06/11 10:11:53 UTC
[I] Delta Stats getting wrongly populated after merge [incubator-gluten]
mani-sethu opened a new issue, #9945:
URL: https://github.com/apache/incubator-gluten/issues/9945
### Backend
VL (Velox)
### Bug description
[Expected behavior]
I am running a merge job with gluten and without gluten and following are the merge statistics on delta log respectively
**With Gluten:**
`object
numTargetRowsCopied: "1056"
numTargetRowsDeleted: "0"
numTargetFilesAdded: "1"
numTargetBytesAdded: "3954"
numTargetBytesRemoved: "3215"
numTargetDeletionVectorsAdded: "0"
numTargetRowsMatchedUpdated: "528"
executionTimeMs: "158406"
numTargetRowsInserted: "528"
numTargetRowsMatchedDeleted: "0"
numTargetDeletionVectorsUpdated: "0"
scanTimeMs: "4326"
numTargetRowsUpdated: "528"
numOutputRows: "2112"
numTargetDeletionVectorsRemoved: "0"
numTargetRowsNotMatchedBySourceUpdated: "0"
numTargetChangeFilesAdded: "0"
numSourceRows: "528"
numTargetFilesRemoved: "1"
numTargetRowsNotMatchedBySourceDeleted: "0"
rewriteTimeMs: "1489"`
**Without Gluten**
numTargetRowsCopied: "0"
numTargetRowsDeleted: "0"
numTargetFilesAdded: "1"
numTargetBytesAdded: "3215"
numTargetBytesRemoved: "3225"
numTargetDeletionVectorsAdded: "0"
numTargetRowsMatchedUpdated: "528"
executionTimeMs: "132701"
materializeSourceTimeMs: "114262"
numTargetRowsInserted: "0"
numTargetRowsMatchedDeleted: "0"
numTargetDeletionVectorsUpdated: "0"
scanTimeMs: "16822"
numTargetRowsUpdated: "528"
numOutputRows: "528"
numTargetDeletionVectorsRemoved: "0"
numTargetRowsNotMatchedBySourceUpdated: "0"
numTargetChangeFilesAdded: "0"
numSourceRows: "528"
numTargetFilesRemoved: "1"
numTargetRowsNotMatchedBySourceDeleted: "0"
rewriteTimeMs: "1593"
Where the second is actually correct. There is no change in the underlying data (both source and destination table). I have tested this for multiple jobs where this is the case.
### Gluten version
main branch
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
Gluten Version: 1.5.0-SNAPSHOT
Commit: c38c431951b3597aa203838bd7d0592974c15ceb
CMake Version: 3.28.3
System: Linux-6.8.0-1030-gcp
Arch: x86_64
CPU Name: Model name: Intel(R) Xeon(R) CPU @ 2.80GHz
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 11.4.0
C Compiler: /usr/bin/cc
C Compiler Version: 11.4.0
CMake Prefix Path: /usr/local;/usr;/;/usr/local/lib/python3.10/dist-packages/cmake/data;/usr/local;/usr/X11R6;/usr/pkg;/opt
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Delta Stats getting wrongly populated after merge [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on issue #9945:
URL: https://github.com/apache/gluten/issues/9945#issuecomment-5053160846
@mani-sethu @zhouyuan this is a known issue: https://github.com/apache/gluten/issues/9003
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org