You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mdomaradzki-cosmose (via GitHub)" <gi...@apache.org> on 2026/01/21 16:12:48 UTC
[I] Spark Parquet timestamp min/max statistics [iceberg]
mdomaradzki-cosmose opened a new issue, #15103:
URL: https://github.com/apache/iceberg/issues/15103
### Query engine
Spark 4.0.1
Iceberg 1.10.0
### Question
I have a table in Iceberg as below:
spark.sql("""
CREATE OR REPLACE TABLE my_db.my_table (
serverTime TIMESTAMP,
id LONG,
...
)
USING iceberg
PARTITIONED BY (days(serverTime), bucket(50, id));
"""
)
I run below queries:
spark.sql("SELECT count(*) FROM my_db.my_table WHERE serverTime >= '2026-01-15 00:00:00';")
spark.sql("SELECT count(*) FROM my_db.my_table WHERE serverTime > '2026-01-15 00:00:00';")
First query finishes almost instantly because it can filter whole partitions.
However second query takes a lot of time (around 15 minutes). I checked statistics per Parquet file and for date('2026-01-15') lower_bounds for serverTime is always bigger than '2026-01-15 00:00:00', shouldn't Spark/Iceberg use these metastatistics instead of reading all files?
I'm using Spark 4.0.1 and Iceberg 1.10.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Parquet timestamp min/max statistics [iceberg]
Posted by "mdomaradzki-cosmose (via GitHub)" <gi...@apache.org>.
mdomaradzki-cosmose commented on issue #15103:
URL: https://github.com/apache/iceberg/issues/15103#issuecomment-3876478209
Hi @RussellSpitzer, here are my plans:
Physical plan 1:
```
== Physical Plan ==
AdaptiveSparkPlan (10)
+- == Final Plan ==
ResultQueryStage (7), Statistics(sizeInBytes=8.0 EiB)
+- * HashAggregate (6)
+- ShuffleQueryStage (5), Statistics(sizeInBytes=16.0 B, rowCount=1)
+- Exchange (4)
+- * HashAggregate (3)
+- * Project (2)
+- * LocalTableScan (1)
+- == Initial Plan ==
HashAggregate (9)
+- Exchange (8)
+- HashAggregate (3)
+- Project (2)
+- LocalTableScan (1)
(1) LocalTableScan [codegen id : 1]
Output [1]: [count(*)#50L]
Arguments: [count(*)#50L]
(2) Project [codegen id : 1]
Output [1]: [count(*)#50L AS agg_func_0#49L]
Input [1]: [count(*)#50L]
(3) HashAggregate [codegen id : 1]
Input [1]: [agg_func_0#49L]
Keys: []
Functions [1]: [partial_sum(agg_func_0#49L)]
Aggregate Attributes [1]: [sum#51L]
Results [1]: [sum#52L]
(4) Exchange
Input [1]: [sum#52L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=23]
(5) ShuffleQueryStage
Output [1]: [sum#52L]
Arguments: 0
(6) HashAggregate [codegen id : 2]
Input [1]: [sum#52L]
Keys: []
Functions [1]: [sum(agg_func_0#49L)]
Aggregate Attributes [1]: [sum(agg_func_0#49L)#48L]
Results [1]: [sum(agg_func_0#49L)#48L AS count#26L]
(7) ResultQueryStage
Output [1]: [count#26L]
Arguments: 1
(8) Exchange
Input [1]: [sum#52L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=13]
(9) HashAggregate
Input [1]: [sum#52L]
Keys: []
Functions [1]: [sum(agg_func_0#49L)]
Aggregate Attributes [1]: [sum(agg_func_0#49L)#48L]
Results [1]: [sum(agg_func_0#49L)#48L AS count#26L]
(10) AdaptiveSparkPlan
Output [1]: [count#26L]
Arguments: isFinalPlan=true
```
Phisical plan 2:
```
== Physical Plan ==
AdaptiveSparkPlan (14)
+- == Final Plan ==
ResultQueryStage (8), Statistics(sizeInBytes=8.0 EiB)
+- * HashAggregate (7)
+- ShuffleQueryStage (6), Statistics(sizeInBytes=384.0 B, rowCount=24)
+- Exchange (5)
+- * HashAggregate (4)
+- * Project (3)
+- * Filter (2)
+- BatchScan my_db.my_table.files (1)
+- == Initial Plan ==
HashAggregate (13)
+- Exchange (12)
+- HashAggregate (11)
+- Project (10)
+- Filter (9)
+- BatchScan my_db.my_table.files (1)
(1) BatchScan my_db.my_table
Output [1]: [serverTime#5]
my_db.my_table (branch=null) [filters=serverTime IS NOT NULL, serverTime > 1768435200000000, groupedBy=]
(2) ColumnarToRow [codegen id : 1]
Input [1]: [serverTime#5]
(3) Filter [codegen id : 1]
Input [1]: [serverTime#5]
Condition : (serverTime#5 > 2026-01-15 00:00:00)
(4) Project [codegen id : 1]
Output: []
Input [1]: [serverTime#5]
(5) HashAggregate [codegen id : 1]
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#50L]
Results [1]: [count#51L]
(6) Exchange
Input [1]: [count#51L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=34]
(7) ShuffleQueryStage
Output [1]: [count#51L]
Arguments: 0
(8) HashAggregate
Input [1]: [count#51L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#48L]
Results [1]: [count(1)#48L AS count#26L]
(9) Filter
Input [1]: [serverTime#5]
Condition : (serverTime#5 > 2026-01-15 00:00:00)
(10) Project
Output: []
Input [1]: [serverTime#5]
(11) HashAggregate
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#50L]
Results [1]: [count#51L]
(12) Exchange
Input [1]: [count#51L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=15]
(13) HashAggregate
Input [1]: [count#51L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#48L]
Results [1]: [count(1)#48L AS count#26L]
(14) AdaptiveSparkPlan
Output [1]: [count#26L]
Arguments: isFinalPlan=false
```
I'm using Spark 4.0.1, because I cannot see iceberg-spark-runtime for 4.1 (https://repo1.maven.org/maven2/org/apache/iceberg/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Parquet timestamp min/max statistics [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #15103: Spark Parquet timestamp min/max statistics
URL: https://github.com/apache/iceberg/issues/15103
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Parquet timestamp min/max statistics [iceberg]
Posted by "mdomaradzki-cosmose (via GitHub)" <gi...@apache.org>.
mdomaradzki-cosmose commented on issue #15103:
URL: https://github.com/apache/iceberg/issues/15103#issuecomment-3876498882
I also made sure that lower_bounds for serverTime are always strictly bigger than '2026-01-15 00:00:00' for each Parquet file in that partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Parquet timestamp min/max statistics [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #15103:
URL: https://github.com/apache/iceberg/issues/15103#issuecomment-3873226640
I wrote a test and could not repo this on the main branch, spark 4.1. Let me know if you have a more concrete example as something else may be going on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Parquet timestamp min/max statistics [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15103:
URL: https://github.com/apache/iceberg/issues/15103#issuecomment-5234606877
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Parquet timestamp min/max statistics [iceberg]
Posted by "Kimura-S2001 (via GitHub)" <gi...@apache.org>.
Kimura-S2001 commented on issue #15103:
URL: https://github.com/apache/iceberg/issues/15103#issuecomment-3857991471
Thanks for the detailed question.
I tried to reproduce this behavior locally and was able to confirm it with a minimal, reproducible example.
---
## Environment
- Spark: 4.0.1
- Iceberg: 1.10.0
- Catalog: Iceberg REST Catalog
- Storage: S3-compatible storage (MinIO)
---
## Table definition
```sql
CREATE TABLE default.ts_15103 (
serverTime TIMESTAMP,
id BIGINT
)
USING iceberg
PARTITIONED BY (days(serverTime));
```
---
## Test data
```sql
INSERT INTO default.ts_15103 VALUES
('2026-01-15 00:00:01', 1),
('2026-01-15 12:00:00', 2),
('2026-01-15 23:59:59', 3);
```
---
## File-level statistics (Iceberg metadata)
```sql
SELECT
file_path,
record_count,
lower_bounds,
upper_bounds
FROM default.ts_15103.files;
```
This shows that for the Parquet file of partition `serverTime_day=2026-01-15`,
the `lower_bounds` of `serverTime` are strictly greater than
`2026-01-15 00:00:00`.
---
## Query with `>=` (partition pruning works)
```sql
SELECT count(*)
FROM default.ts_15103
WHERE serverTime >= TIMESTAMP '2026-01-15 00:00:00';
```
- Result: `3`
- Physical plan uses metadata-based pruning
- No Parquet file scan is required
---
## Query with `>` (unexpected full scan)
```sql
SELECT count(*)
FROM default.ts_15103
WHERE serverTime > TIMESTAMP '2026-01-15 00:00:00';
```
- Result: `3`
- Physical plan shows a `BatchScan`
- Parquet files are scanned even though
file-level `lower_bounds` should allow pruning
---
## Observation
Although the file-level Iceberg statistics indicate that:
```
lower_bounds(serverTime) > '2026-01-15 00:00:00'
```
Spark + Iceberg only applies metadata pruning for `>=`,
but not for `>`.
From a logical perspective, both predicates should allow
the file to be safely included without scanning its contents.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Parquet timestamp min/max statistics [iceberg]
Posted by "Kimura-S2001 (via GitHub)" <gi...@apache.org>.
Kimura-S2001 commented on issue #15103:
URL: https://github.com/apache/iceberg/issues/15103#issuecomment-3832893504
I'm interested in working on this, but I'm new to this project. It's okay if I give it a try?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Parquet timestamp min/max statistics [iceberg]
Posted by "Kimura-S2001 (via GitHub)" <gi...@apache.org>.
Kimura-S2001 commented on issue #15103:
URL: https://github.com/apache/iceberg/issues/15103#issuecomment-3869296269
I investigated this behavior and opened an improvement issue with reproducible details: #15270
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Parquet timestamp min/max statistics [iceberg]
Posted by "mdomaradzki-cosmose (via GitHub)" <gi...@apache.org>.
mdomaradzki-cosmose commented on issue #15103:
URL: https://github.com/apache/iceberg/issues/15103#issuecomment-3872826975
Thanks @Kimura-S2001 for investigating it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Parquet timestamp min/max statistics [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #15103:
URL: https://github.com/apache/iceberg/issues/15103#issuecomment-3872880596
That issue looks unrelated? I also see we have tests to make sure this works
https://github.com/apache/iceberg/blob/main/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java#L451-L478
So I'm not sure this is actually a bug as described? Can we see the actual explain plans for the queries causing an issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Parquet timestamp min/max statistics [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15103:
URL: https://github.com/apache/iceberg/issues/15103#issuecomment-5403187589
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org