You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "malinjawi (via GitHub)" <gi...@apache.org> on 2026/10/20 16:31:15 UTC
[PR] [VL][Delta] Fix base row ID filters and missing row tracking metadata in native scans [gluten]
malinjawi opened a new pull request, #13391:
URL: https://github.com/apache/gluten/pull/13391
## What changes are proposed in this pull request?
Fixes two row tracking read bugs in the native Delta scan, seen in the nightly Delta Lake job (Spark 4.1, Delta 4.4.1).
**Filters on constant `_metadata` fields.** A filter such as `_metadata.base_row_id = 10` was pushed into the native scan. Velox reads constant `_metadata` fields as synthesized split columns. Since facebookincubator/velox#16116 it fails the scan when a pushed filter on one of them rejects a split, because it expects the planner to have pruned that split. Delta's file index does not prune by these fields, so any such filter that selects a subset of files failed with `Synthesized column 'base_row_id' failed filter validation`. `DeltaScanTransformer` now leaves filters on constant metadata fields out of `scanFilters` and the pushed-down filters. The Filter above the scan evaluates them, still in Velox. The `_metadata` file fields, such as `file_name`, take the same path.
**Missing base row IDs.** Delta fails a read of a non-nullable `_metadata.base_row_id` or `_metadata.default_row_commit_version` from a file that has no value for it. The native scan returned NULL instead, so `_metadata.row_id` returned rows where Delta raises an error. Such a scan now fails validation and runs on Spark, so it fails as in Delta. The check looks at the scan's files only when the scan reads one of those two fields. Delta assigns both values to every file once row tracking is on, so only a table changed by hand gets here. Delta Lake's test builds one with a manual metadata commit. The check is per Delta version (`DeltaRowTrackingScanInfo`):
- On Delta 4.x, scans with `nullableRowTrackingConstantFields` stay native and read NULL, as Delta does.
- Delta 3.3 always fails such reads.
- Delta 2.4 has no row tracking fields.
Removes three `RowIdSuite` entries from the Delta Lake test baseline: `Filter by base Row IDs`, `Filter by base Row IDs in subquery` and `missing base row ids and default row commit versions`. The docs listed both bugs under read problems. They now describe the new behavior.
The other three row tracking entries are not scan bugs:
- `row ids can be read back` and `No dictionary filtering on _metadata.row_id` fail on the row-group count and encoding names of a plain Parquet file that the Velox writer writes for the test. Both fail before reading any row ID.
- `write and read table with all-null materialized columns` fails on a zero-byte file left by an empty native write, fixed in #13390.
## How was this patch tested?
- `DeltaRowTrackingScanInfoSuite` plans the queries on vanilla Spark. It checks which filters reach the native scan and which scans the new check flags. It passes locally on Spark 3.5 (Delta 3.3) and Spark 4.1 (Delta 4.4.1).
- `VeloxDeltaRowTrackingSuite` runs the queries with Gluten:
- Filters on `base_row_id` and `file_name`, also in a subquery, return the right rows from a native scan.
- Reading row IDs or row commit versions from a table with a file that lacks them throws.
- A read that data skipping limits to files with the values stays native.
It loads the native library, so it runs in CI.
- Both suites are in the Delta 3.3 tree and linked into the Delta 4 tree.
- Compiled main and test code on Spark 3.5 and 4.1 with `-Pbackends-velox,spark-<v>,delta,spark-ut`, and `gluten-delta` on Spark 3.4.
cc @rui-mo @felipepessoto
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: IBM Bob
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Fix base row ID filters and missing row tracking metadata in native scans [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13391:
URL: https://github.com/apache/gluten/pull/13391#issuecomment-6299678883
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Fix base row ID filters and missing row tracking metadata in native scans [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13391:
URL: https://github.com/apache/gluten/pull/13391#issuecomment-6312060528
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org