You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mattfaltyn (via GitHub)" <gi...@apache.org> on 2026/08/23 16:14:45 UTC
[I] Mixed-range IN predicates can fail during scan planning [iceberg-go]
mattfaltyn opened a new issue, #1870:
URL: https://github.com/apache/iceberg-go/issues/1870
## Apache Iceberg version
`main` at `0f08e66e59cc1ec9b40898823d9eb991409bd080` (also present in v0.6.0)
## Please describe the bug 🐞
Binding an `IN` or `NOT IN` predicate to a narrower numeric field retains `AboveMaxLiteral` and `BelowMinLiteral` sentinels in the bound set. Metrics evaluators assume every set member is a regular `TypedLiteral[T]`, so a valid scan can fail during planning when its filter mixes representable and out-of-range values.
For example, an application may build an `int64` filter for an Iceberg `int32` column:
```go
iceberg.IsIn(iceberg.Reference("id"), int64(35), int64(math.MaxInt32)+1)
```
The second value can never equal an `int32` column value and should be discarded during binding. Instead, it becomes `aboveMaxLiteral[int32]`. Inclusive metrics evaluation reaches `removeBoundCmp`, asserts that every member implements `TypedLiteral[int32]`, and returns this error:
```text
interface conversion: iceberg.aboveMaxLiteral[int32] is not iceberg.TypedLiteral[int32]: missing method Comparator
```
This affects the normal scan-planning path because `Scan.collectManifestEntriesWithSchema` creates the inclusive metrics evaluator for the row filter.
### Expected behavior
- Impossible set members are removed when binding to the field type.
- The mixed predicate above behaves like `id IN (35)`.
- A set containing only impossible members simplifies to `AlwaysFalse` for `IN` and `AlwaysTrue` for `NOT IN`.
- Metrics evaluation completes without an error.
### Reproduction
The following regression test can be added to `table/evaluators_test.go`:
```go
func TestInclusiveMetricsMixedRangeIn(t *testing.T) {
lower, err := iceberg.Int32Literal(30).MarshalBinary()
require.NoError(t, err)
upper, err := iceberg.Int32Literal(79).MarshalBinary()
require.NoError(t, err)
schema := iceberg.NewSchema(0,
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int32},
)
file := &mockDataFile{
count: 1,
valueCounts: map[int]int64{1: 1},
nullCounts: map[int]int64{1: 0},
lowerBounds: map[int][]byte{1: lower},
upperBounds: map[int][]byte{1: upper},
}
filter := iceberg.IsIn(
iceberg.Reference("id"),
int64(35),
int64(math.MaxInt32)+1,
)
eval, err := newInclusiveMetricsEvaluator(schema, filter, true, true)
require.NoError(t, err)
shouldRead, err := eval(file)
require.NoError(t, err)
assert.True(t, shouldRead)
}
```
On current `main`, `eval(file)` returns the interface-conversion error above. An in-range control using `int64(35), int64(36)` succeeds.
### Root cause
`createBoundSetPredicate` calls `Literal.To(boundType)` and unconditionally inserts the result. Scalar literal binding already handles `AboveMaxLiteral` and `BelowMinLiteral` specially, but set binding does not.
Dropping those impossible members before the existing zero/one-member simplification should preserve `IN` and `NOT IN` semantics while preventing non-comparable sentinel values from reaching expression, manifest, metrics, and bloom-filter consumers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Mixed-range IN predicates can fail during scan planning [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #1870: Mixed-range IN predicates can fail during scan planning
URL: https://github.com/apache/iceberg-go/issues/1870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Mixed-range IN predicates can fail during scan planning [iceberg-go]
Posted by "mattfaltyn (via GitHub)" <gi...@apache.org>.
mattfaltyn commented on issue #1870:
URL: https://github.com/apache/iceberg-go/issues/1870#issuecomment-5386999797
I'd like to work on this. I reproduced it on current main and will keep the fix scoped to set-predicate binding and focused regression coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org