You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Peter Vary <pv...@cloudera.com.INVALID> on 2021/02/19 14:08:37 UTC
Timestamp / Date filtering
Hi Team,
Running tests against Iceberg backed Hive tables I was trying to run this query:
SELECT * from date_test WHERE d_date='1998-02-19'
The query fails with the Exception below. Basically complains that when filtering the Date field it expects an Integer in the Record, but finds a LocalDate object instead.
I am confused now, because my mental model was - until now - that Iceberg records stores data like:
LocalDate - Types.DateType,
LocalDateTime - Types.TimestampType.withoutZone,
OffsetDateTime - Types.TimestampType.withZone,
Debugging the query above I have found that the expression evaluation expects different types in the Record defined in org.apache.iceberg.types.Type where we define:
Date - Integer,
Timestamp - Long
Do I supposed to prepare the Record for expression evaluation, so the evaluator could find the correct type?
What part of the puzzle I miss?
Thanks,
Peter
The exception:
Caused by: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.IllegalStateException: Not an instance of java.lang.Integer: 2130-01-20
at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:499)
at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:307)
at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:878)
at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:559)
at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:551)
at org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:143)
... 62 more
Caused by: java.io.IOException: java.lang.IllegalStateException: Not an instance of java.lang.Integer: 2130-01-20
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:521)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:428)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2208)
at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:494)
... 67 more
Caused by: java.lang.IllegalStateException: Not an instance of java.lang.Integer: 2130-01-20
at org.apache.iceberg.data.GenericRecord.get(GenericRecord.java:123)
at org.apache.iceberg.Accessors$PositionAccessor.get(Accessors.java:71)
at org.apache.iceberg.Accessors$PositionAccessor.get(Accessors.java:58)
at org.apache.iceberg.expressions.BoundReference.eval(BoundReference.java:39)
at org.apache.iceberg.expressions.Evaluator$EvalVisitor.eq(Evaluator.java:132)
at org.apache.iceberg.expressions.Evaluator$EvalVisitor.eq(Evaluator.java:52)
at org.apache.iceberg.expressions.ExpressionVisitors$BoundVisitor.predicate(ExpressionVisitors.java:249)
at org.apache.iceberg.expressions.ExpressionVisitors.visitEvaluator(ExpressionVisitors.java:346)
at org.apache.iceberg.expressions.Evaluator$EvalVisitor.eval(Evaluator.java:57)
at org.apache.iceberg.expressions.Evaluator$EvalVisitor.access$100(Evaluator.java:52)
at org.apache.iceberg.expressions.Evaluator.eval(Evaluator.java:49)
at org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.lambda$applyResidualFiltering$0(IcebergInputFormat.java:288)
at org.apache.iceberg.io.CloseableIterable$3.shouldKeep(CloseableIterable.java:82)
at org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:67)
at org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:50)
at org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.nextKeyValue(IcebergInputFormat.java:202)
at org.apache.iceberg.mr.mapred.MapredIcebergInputFormat$MapredIcebergRecordReader.next(MapredIcebergInputFormat.java:104)
at org.apache.iceberg.mr.mapred.MapredIcebergInputFormat$MapredIcebergRecordReader.next(MapredIcebergInputFormat.java:81)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:488)
... 71 more
Re: Timestamp / Date filtering
Posted by Peter Vary <pv...@cloudera.com.INVALID>.
Created a PR with the fix, but not entirely comfortable with the changes in GenericRecord.java
https://github.com/apache/iceberg/pull/2254 <https://github.com/apache/iceberg/pull/2254>
If you have time and more knowledge around the expected behavior of the GenericRecord, please review.
Thanks,
Peter
> On Feb 19, 2021, at 15:08, Peter Vary <pv...@cloudera.com> wrote:
>
> Hi Team,
>
> Running tests against Iceberg backed Hive tables I was trying to run this query:
>
> SELECT * from date_test WHERE d_date='1998-02-19'
>
> The query fails with the Exception below. Basically complains that when filtering the Date field it expects an Integer in the Record, but finds a LocalDate object instead.
>
> I am confused now, because my mental model was - until now - that Iceberg records stores data like:
> LocalDate - Types.DateType,
> LocalDateTime - Types.TimestampType.withoutZone,
> OffsetDateTime - Types.TimestampType.withZone,
>
> Debugging the query above I have found that the expression evaluation expects different types in the Record defined in org.apache.iceberg.types.Type where we define:
> Date - Integer,
> Timestamp - Long
>
> Do I supposed to prepare the Record for expression evaluation, so the evaluator could find the correct type?
> What part of the puzzle I miss?
>
> Thanks,
> Peter
>
> The exception:
>
> Caused by: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.IllegalStateException: Not an instance of java.lang.Integer: 2130-01-20
> at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:499)
> at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:307)
> at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:878)
> at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:559)
> at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:551)
> at org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:143)
> ... 62 more
> Caused by: java.io.IOException: java.lang.IllegalStateException: Not an instance of java.lang.Integer: 2130-01-20
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:521)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:428)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2208)
> at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:494)
> ... 67 more
> Caused by: java.lang.IllegalStateException: Not an instance of java.lang.Integer: 2130-01-20
> at org.apache.iceberg.data.GenericRecord.get(GenericRecord.java:123)
> at org.apache.iceberg.Accessors$PositionAccessor.get(Accessors.java:71)
> at org.apache.iceberg.Accessors$PositionAccessor.get(Accessors.java:58)
> at org.apache.iceberg.expressions.BoundReference.eval(BoundReference.java:39)
> at org.apache.iceberg.expressions.Evaluator$EvalVisitor.eq(Evaluator.java:132)
> at org.apache.iceberg.expressions.Evaluator$EvalVisitor.eq(Evaluator.java:52)
> at org.apache.iceberg.expressions.ExpressionVisitors$BoundVisitor.predicate(ExpressionVisitors.java:249)
> at org.apache.iceberg.expressions.ExpressionVisitors.visitEvaluator(ExpressionVisitors.java:346)
> at org.apache.iceberg.expressions.Evaluator$EvalVisitor.eval(Evaluator.java:57)
> at org.apache.iceberg.expressions.Evaluator$EvalVisitor.access$100(Evaluator.java:52)
> at org.apache.iceberg.expressions.Evaluator.eval(Evaluator.java:49)
> at org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.lambda$applyResidualFiltering$0(IcebergInputFormat.java:288)
> at org.apache.iceberg.io.CloseableIterable$3.shouldKeep(CloseableIterable.java:82)
> at org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:67)
> at org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:50)
> at org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.nextKeyValue(IcebergInputFormat.java:202)
> at org.apache.iceberg.mr.mapred.MapredIcebergInputFormat$MapredIcebergRecordReader.next(MapredIcebergInputFormat.java:104)
> at org.apache.iceberg.mr.mapred.MapredIcebergInputFormat$MapredIcebergRecordReader.next(MapredIcebergInputFormat.java:81)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:488)
> ... 71 more