You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jiwen624 (via GitHub)" <gi...@apache.org> on 2026/10/13 01:21:04 UTC
[I] IN/NOT IN filters on identity-partitioned timestamp_ns columns fail or return wrong results [iceberg]
jiwen624 opened a new issue, #18479:
URL: https://github.com/apache/iceberg/issues/18479
### Apache Iceberg version
1.12.0 (latest release)
### Query engine
Other
### Please describe the bug 🐞
On a v3 table with a `timestamp_ns` column `ts` partitioned by `identity(ts)`, an `IN` or `NOT IN` filter with two or more values fails during scan planning:
```Java
table.newScan()
.filter(Expressions.in("ts", "2026-01-01T00:00:00.000000001", "2025-01-01T00:00:00.000000001"))
.planFiles();
// java.lang.ArithmeticException: long overflow
```
For timestamps near 1970-01-01 there is no error, but the results are wrong:
- `IN` skips files that contain matching rows.
- `newDelete().deleteFromRowFilter(notIn(...))` deletes files whose rows all match an excluded value.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] IN/NOT IN filters on identity-partitioned timestamp_ns columns fail or return wrong results [iceberg]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on issue #18479:
URL: https://github.com/apache/iceberg/issues/18479#issuecomment-6387604104
> 3\. I can reproduce this on main (JDK 21) with a unit test that doesn't need a table.
> Schema: optional timestamp_ns "ts", spec: identity("ts").
> Expression projected = Projections.inclusive(spec)
> .project(Expressions.in("ts", "2026-01-01T00:00:00.000000001",
> "2025-01-01T00:00:00.000000001"));
> Binder.bind(spec.partitionType(), projected, true);
> This throws ArithmeticException: long overflow:
> DateTimeUtil.microsToNanos
> <- Literals$TimestampLiteral.to (Literals.java:452)
> <- Literals$LongLiteral.to (Literals.java:311)
> <- UnboundPredicate.bindInOperation
> For values near the epoch there is no exception but the bound set is wrong:
> [1, 86400000000001] becomes [1000, 86400000000001000].
> Cause: Identity.projectStrict handles set predicates by passing the raw Long values
> from asSetPredicate().literalSet() to Expressions.predicate(op, name, Iterable),
> which wraps them in LongLiteral. At bind time LongLiteral.to(TIMESTAMP_NANO)
> assumes micros and multiplies by 1000. Single-value predicates avoid this because
> Identity passes the typed Literal.
> I'd like to work on this. The tricky part is that there is no public way to build
> an UnboundPredicate for a set from typed literals (that constructor is private and
> the literal classes are package-private). Which direction would you prefer?
> (1) a new Expressions/UnboundPredicate factory that takes typed literals, used by
> Identity,
> (2) changing how LongLiteral converts to timestamp_ns, or
> (3) something else?
> I have failing tests for the inclusive projection and can add strict projection
> (and NOT IN) as well.
Thanks for confirming the repro! There's already a fix in #18478. It wraps the IN/NOT IN values with Expressions.nanos(...) in Identity, so the existing public API covers it without a new factory or a change to LongLiteral. It has tests for both inclusive IN and strict NOT IN. Reviews welcome there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] IN/NOT IN filters on identity-partitioned timestamp_ns columns fail or return wrong results [iceberg]
Posted by "adarshomrey (via GitHub)" <gi...@apache.org>.
adarshomrey commented on issue #18479:
URL: https://github.com/apache/iceberg/issues/18479#issuecomment-6382676793
I can reproduce this on main (JDK 21) with a unit test that doesn't need a table.
Schema: optional timestamp_ns "ts", spec: identity("ts").
Expression projected = Projections.inclusive(spec)
.project(Expressions.in("ts", "2026-01-01T00:00:00.000000001",
"2025-01-01T00:00:00.000000001"));
Binder.bind(spec.partitionType(), projected, true);
This throws ArithmeticException: long overflow:
DateTimeUtil.microsToNanos
<- Literals$TimestampLiteral.to (Literals.java:452)
<- Literals$LongLiteral.to (Literals.java:311)
<- UnboundPredicate.bindInOperation
For values near the epoch there is no exception but the bound set is wrong:
[1, 86400000000001] becomes [1000, 86400000000001000].
Cause: Identity.projectStrict handles set predicates by passing the raw Long values
from asSetPredicate().literalSet() to Expressions.predicate(op, name, Iterable<T>),
which wraps them in LongLiteral. At bind time LongLiteral.to(TIMESTAMP_NANO)
assumes micros and multiplies by 1000. Single-value predicates avoid this because
Identity passes the typed Literal.
I'd like to work on this. The tricky part is that there is no public way to build
an UnboundPredicate for a set from typed literals (that constructor is private and
the literal classes are package-private). Which direction would you prefer?
(1) a new Expressions/UnboundPredicate factory that takes typed literals, used by
Identity,
(2) changing how LongLiteral converts to timestamp_ns, or
(3) something else?
I have failing tests for the inclusive projection and can add strict projection
(and NOT IN) as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] IN/NOT IN filters on identity-partitioned timestamp_ns columns fail or return wrong results [iceberg]
Posted by "adarshomrey (via GitHub)" <gi...@apache.org>.
adarshomrey commented on issue #18479:
URL: https://github.com/apache/iceberg/issues/18479#issuecomment-6387802946
Thanks, and sorry for the noise about the missing public API: I'd missed Expressions.nanos.
I've tested #18478 locally against my own test and it fixes the issue; I left a comment there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org