[GitHub] [arrow-rs] crepererum commented on a change in pull request #256: fix NaN handling in parquet statistics

[GitBox <gi...@apache.org>]

crepererum commented on a change in pull request #256:
URL: https://github.com/apache/arrow-rs/pull/256#discussion_r625605221



##########
File path: parquet/src/column/writer.rs
##########
@@ -922,11 +922,14 @@ impl<T: DataType> ColumnWriterImpl<T> {
     }
 
     fn update_page_min_max(&mut self, val: &T::T) {
-        if self.min_page_value.as_ref().map_or(true, |min| min > val) {
-            self.min_page_value = Some(val.clone());
-        }
-        if self.max_page_value.as_ref().map_or(true, |max| max < val) {
-            self.max_page_value = Some(val.clone());
+        // simple "isNaN" check that works for all types
+        if val == val {

Review comment:
       Should we introduce some generic `T::is_nan` function into the data type abstraction layer (that basically returns `False` for every non-float/double type and calls `.is_nan()` for float/double) or is this local workaround good enough?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org