[GitHub] [iceberg] yyanyy commented on a change in pull request #3320: ORC: Fix importing ORC files with float and double columns

[GitBox <gi...@apache.org>]

yyanyy commented on a change in pull request #3320:
URL: https://github.com/apache/iceberg/pull/3320#discussion_r732344704



##########
File path: orc/src/main/java/org/apache/iceberg/orc/OrcMetrics.java
##########
@@ -196,11 +196,17 @@ private static Metrics buildOrcMetrics(final long numOfRows, final TypeDescripti
         min = Math.toIntExact((long) min);
       }
     } else if (columnStats instanceof DoubleColumnStatistics) {
-      // since Orc includes NaN for upper/lower bounds of floating point columns, and we don't want this behavior,
-      // we have tracked metrics for such columns ourselves and thus do not need to rely on Orc's column statistics.
-      Preconditions.checkNotNull(fieldMetrics,
-          "[BUG] Float or double type columns should have metrics being tracked by Iceberg Orc writers");
-      min = fieldMetrics.lowerBound();
+      if (fieldMetrics != null) {
+        // since Orc includes NaN for upper/lower bounds of floating point columns, and we don't want this behavior,
+        // we have tracked metrics for such columns ourselves and thus do not need to rely on Orc's column statistics.
+        min = fieldMetrics.lowerBound();
+      } else {
+        // imported files will not have metrics that were tracked by Iceberg, so fall back to the file's metrics.
+        min = ((DoubleColumnStatistics) columnStats).getMinimum();
+        if (type.typeId() == Type.TypeID.FLOAT) {
+          min = ((Double) min).floatValue();

Review comment:
       wondering if we want to reassign min to negative infinite if the this value tracked by ORC is NaN




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org