You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "weimingdiit (via GitHub)" <gi...@apache.org> on 2026/04/27 12:45:46 UTC
[PR] [AURON #2217] Support Iceberg _spec_id metadata in native scan [auron]
weimingdiit opened a new pull request, #2218:
URL: https://github.com/apache/auron/pull/2218
# Which issue does this PR close?
Closes #2217
# Rationale for this change
Native Iceberg scan currently supports projecting the `_file` metadata column, but falls back when `_spec_id` is requested. `_spec_id` is a file-level Iceberg metadata column and can be materialized as a per-file constant value in the native scan, similar to `_file`.
# What changes are included in this PR?
This PR adds native scan support for the Iceberg `_spec_id` metadata column.
- Allows `_spec_id` in Iceberg native scan metadata column validation.
- Materializes `_spec_id` from `FileScanTask.file().specId()` as a per-file partition value.
- Adds integration tests for `_spec_id` projection and mixed data/metadata projection with `_file` and `_spec_id`.
# Are there any user-facing changes?
Yes. Queries that project Iceberg `_spec_id` can now use the native Iceberg scan path instead of falling back to Spark.
# How was this patch tested?
CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2217] Support Iceberg _spec_id metadata in native scan [auron]
Posted by "ShreyeshArangath (via GitHub)" <gi...@apache.org>.
ShreyeshArangath commented on code in PR #2218:
URL: https://github.com/apache/auron/pull/2218#discussion_r3157438315
##########
thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeIcebergTableScanExec.scala:
##########
@@ -221,6 +226,15 @@ case class NativeIcebergTableScanExec(basedScan: BatchScanExec, plan: IcebergSca
.toMap
}
+ private def buildFileSpecIds(): Map[String, Int] = {
+ // Map file path to Iceberg partition spec id; tasks may split a file into multiple ranges.
+ fileTasks
Review Comment:
```suggestion
fileTasks.iterator
.map(task => task.file().location() -> task.file().specId())
.toMap
```
##########
thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeIcebergTableScanExec.scala:
##########
@@ -125,6 +126,10 @@ case class NativeIcebergTableScanExec(basedScan: BatchScanExec, plan: IcebergSca
field.name match {
case name if name == MetadataColumns.FILE_PATH.name() =>
NativeConverters.convertExpr(Literal.create(filePath, StringType)).getLiteral
+ case name if name == MetadataColumns.SPEC_ID.name() =>
+ NativeConverters
+ .convertExpr(Literal.create(fileSpecIds(filePath), IntegerType))
Review Comment:
Should we do something like to be safer?
```suggestion
case name if name == MetadataColumns.SPEC_ID.name() =>
NativeConverters
.convertExpr(Literal.create(fileSpecIds(filePath), field.dataType))
.getLiteral
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2217] Support Iceberg _spec_id metadata in native scan [auron]
Posted by "weimingdiit (via GitHub)" <gi...@apache.org>.
weimingdiit commented on PR #2218:
URL: https://github.com/apache/auron/pull/2218#issuecomment-4414772497
@ShreyeshArangath @yew1eb @slfan1989 Thanks for your review and merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2217] Support Iceberg _spec_id metadata in native scan [auron]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #2218:
URL: https://github.com/apache/auron/pull/2218#issuecomment-4414683845
@weimingdiit Thanks for the contribution! Merged. @ShreyeshArangath @yew1eb Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2217] Support Iceberg _spec_id metadata in native scan [auron]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2218:
URL: https://github.com/apache/auron/pull/2218#discussion_r3152262208
##########
thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeIcebergTableScanExec.scala:
##########
@@ -125,6 +126,10 @@ case class NativeIcebergTableScanExec(basedScan: BatchScanExec, plan: IcebergSca
field.name match {
case name if name == MetadataColumns.FILE_PATH.name() =>
NativeConverters.convertExpr(Literal.create(filePath, StringType)).getLiteral
+ case name if name == MetadataColumns.SPEC_ID.name() =>
+ NativeConverters
+ .convertExpr(Literal.create(fileSpecIds(filePath), IntegerType))
+ .getLiteral
Review Comment:
`fileSpecIds(filePath)` uses `Map.apply`, which will throw a `NoSuchElementException` with little context if the key isn’t present (e.g., due to path normalization differences between Iceberg `DataFile.location()` and Spark `PartitionedFile.filePath.toString`). Consider using `getOrElse` and throwing an `IllegalStateException` with a clear message, and/or normalizing the key the same way in both `buildFileSpecIds` and `metadataPartitionValues` to guarantee consistent lookups.
##########
thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeIcebergTableScanExec.scala:
##########
@@ -221,6 +226,15 @@ case class NativeIcebergTableScanExec(basedScan: BatchScanExec, plan: IcebergSca
.toMap
}
+ private def buildFileSpecIds(): Map[String, Int] = {
+ // Map file path to Iceberg partition spec id; tasks may split a file into multiple ranges.
+ fileTasks
+ .map(task => task.file().location() -> task.file().specId())
+ .groupBy(_._1)
+ .mapValues(_.head._2)
+ .toMap
Review Comment:
`buildFileSpecIds` currently does `map(...).groupBy(...).mapValues(_.head...)`, which allocates a full grouped collection of all tasks. Since duplicates are expected only because a single file can be split into multiple ranges, this can be built in a single pass (and optionally validated for consistency when the same path appears multiple times) to reduce memory and CPU overhead on large scans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2217] Support Iceberg _spec_id metadata in native scan [auron]
Posted by "weimingdiit (via GitHub)" <gi...@apache.org>.
weimingdiit commented on PR #2218:
URL: https://github.com/apache/auron/pull/2218#issuecomment-4376148955
> Approach LGTM, just a couple minor comments
@ShreyeshArangath Thanks for your review and comments. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2217] Support Iceberg _spec_id metadata in native scan [auron]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 merged PR #2218:
URL: https://github.com/apache/auron/pull/2218
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2217] Support Iceberg _spec_id metadata in native scan [auron]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #2218:
URL: https://github.com/apache/auron/pull/2218#issuecomment-4414682984
@ShreyeshArangath Thanks for the review! I had an offline discussion with @weimingdiit. He mentioned that he has several other Iceberg-related PRs to work on. Therefore, I plan to merge this PR first to avoid blocking his subsequent development. If you have any other questions or suggestions, we will address them in future PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2217] Support Iceberg _spec_id metadata in native scan [auron]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #2218:
URL: https://github.com/apache/auron/pull/2218#issuecomment-4384792313
@ShreyeshArangath Could you please take another look at this PR? Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2217] Support Iceberg _spec_id metadata in native scan [auron]
Posted by "weimingdiit (via GitHub)" <gi...@apache.org>.
weimingdiit commented on code in PR #2218:
URL: https://github.com/apache/auron/pull/2218#discussion_r3185693870
##########
thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeIcebergTableScanExec.scala:
##########
@@ -125,6 +126,10 @@ case class NativeIcebergTableScanExec(basedScan: BatchScanExec, plan: IcebergSca
field.name match {
case name if name == MetadataColumns.FILE_PATH.name() =>
NativeConverters.convertExpr(Literal.create(filePath, StringType)).getLiteral
+ case name if name == MetadataColumns.SPEC_ID.name() =>
+ NativeConverters
+ .convertExpr(Literal.create(fileSpecIds(filePath), IntegerType))
Review Comment:
Thanks for the suggestion. That makes sense.
I updated this to use the metadata field's dataType when building the _spec_id literal, instead of hard-coding IntegerType. This is more consistent with how native file scans build partition values from the partition schema, while preserving the same behavior for the current Iceberg _spec_id field.
Fixed in the latest update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org