You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/12/30 09:50:40 UTC

[GitHub] [arrow] sanjibansg opened a new pull request, #15125: ARROW-12264: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

sanjibansg opened a new pull request, #15125:
URL: https://github.com/apache/arrow/pull/15125

   This PR fixes the issue of handling NaNs in the Parquet predicate push-down. 
   While computing the valid bounds for a column, if the max or min of the column is null, the range should ignore that. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] westonpace merged pull request #15125: GH-28074: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace merged PR #15125:
URL: https://github.com/apache/arrow/pull/15125


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sanjibansg commented on a diff in pull request #15125: ARROW-12264: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on code in PR #15125:
URL: https://github.com/apache/arrow/pull/15125#discussion_r1060318928


##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -143,18 +147,25 @@ std::optional<compute::Expression> ColumnChunkStatisticsAsExpression(
       }
       return compute::or_(std::move(single_value), is_null(std::move(field_expr)));
     }
-
+    
     auto lower_bound =
-        compute::greater_equal(field_expr, compute::literal(std::move(min)));
-    auto upper_bound = compute::less_equal(field_expr, compute::literal(std::move(max)));
+        compute::greater_equal(field_expr, compute::literal(min));
+    auto upper_bound = compute::less_equal(field_expr, compute::literal(max));
+    auto in_range = compute::Expression();

Review Comment:
   made the changes, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sanjibansg commented on a diff in pull request #15125: ARROW-12264: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on code in PR #15125:
URL: https://github.com/apache/arrow/pull/15125#discussion_r1061883442


##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -135,6 +146,10 @@ std::optional<compute::Expression> ColumnChunkStatisticsAsExpression(
     min = maybe_min.MoveValueUnsafe();
     max = maybe_max.MoveValueUnsafe();
 
+    if (isNan(*min) && isNan(*max)) {

Review Comment:
   Added a comment mentioning the NaN handling.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #15125: ARROW-12264: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15125:
URL: https://github.com/apache/arrow/pull/15125#issuecomment-1367826137

   https://issues.apache.org/jira/browse/ARROW-12264


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sanjibansg commented on pull request #15125: GH-28074: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by "sanjibansg (via GitHub)" <gi...@apache.org>.
sanjibansg commented on PR #15125:
URL: https://github.com/apache/arrow/pull/15125#issuecomment-1416676688

   > Looks like the parquet tests are failing because they can't find the parquet file. Do you maybe need to update the submodule version? I haven't done this recently and am not sure of the right commands.
   @westonpace 
   I think by mistake, I updated the `arrow-testing` submodule as well while updating `parquet-testing`. Will it be okay?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] westonpace commented on a diff in pull request #15125: ARROW-12264: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by GitBox <gi...@apache.org>.
westonpace commented on code in PR #15125:
URL: https://github.com/apache/arrow/pull/15125#discussion_r1060115894


##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -143,18 +147,25 @@ std::optional<compute::Expression> ColumnChunkStatisticsAsExpression(
       }
       return compute::or_(std::move(single_value), is_null(std::move(field_expr)));
     }
-
+    
     auto lower_bound =
-        compute::greater_equal(field_expr, compute::literal(std::move(min)));
-    auto upper_bound = compute::less_equal(field_expr, compute::literal(std::move(max)));
+        compute::greater_equal(field_expr, compute::literal(min));
+    auto upper_bound = compute::less_equal(field_expr, compute::literal(max));
+    auto in_range = compute::Expression();

Review Comment:
   ```suggestion
       compute::Expression in_range;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sanjibansg commented on a diff in pull request #15125: ARROW-12264: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on code in PR #15125:
URL: https://github.com/apache/arrow/pull/15125#discussion_r1061883069


##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -98,6 +98,17 @@ Result<std::shared_ptr<SchemaManifest>> GetSchemaManifest(
   return manifest;
 }
 
+bool isNan(const Scalar& value) {
+  if (value.type->Equals(*float32())) {
+    const FloatScalar& float_scalar = checked_cast<const FloatScalar&>(value);
+    return std::isnan(float_scalar.value);

Review Comment:
   Checking for validity now.



##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -98,6 +98,17 @@ Result<std::shared_ptr<SchemaManifest>> GetSchemaManifest(
   return manifest;
 }
 
+bool isNan(const Scalar& value) {
+  if (value.type->Equals(*float32())) {
+    const FloatScalar& float_scalar = checked_cast<const FloatScalar&>(value);
+    return std::isnan(float_scalar.value);

Review Comment:
   Checking for validity now first.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sanjibansg commented on a diff in pull request #15125: GH-28074: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by "sanjibansg (via GitHub)" <gi...@apache.org>.
sanjibansg commented on code in PR #15125:
URL: https://github.com/apache/arrow/pull/15125#discussion_r1101756899


##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -306,6 +282,65 @@ Result<bool> IsSupportedParquetFile(const ParquetFileFormat& format,
 
 }  // namespace
 
+std::optional<compute::Expression> ParquetFileFragment::EvaluateStatisticsAsExpression(
+    const Field& field, const parquet::Statistics& statistics) {
+  auto field_expr = compute::field_ref(field.name());
+
+  // Optimize for corner case where all values are nulls
+  if (statistics.num_values() == 0 && statistics.null_count() > 0) {
+    return is_null(std::move(field_expr));
+  }
+
+  std::shared_ptr<Scalar> min, max;
+  if (!StatisticsAsScalars(statistics, &min, &max).ok()) {
+    return std::nullopt;
+  }
+
+  auto maybe_min = min->CastTo(field.type());
+  auto maybe_max = max->CastTo(field.type());
+
+  if (maybe_min.ok() && maybe_max.ok()) {
+    min = maybe_min.MoveValueUnsafe();
+    max = maybe_max.MoveValueUnsafe();
+
+    // Since the minimum & maximum values are NaN, useful statistics
+    // cannot be extracted for checking the presence of a value within
+    // range
+    if (IsNan(*min) && IsNan(*max)) {
+      return std::nullopt;
+    }
+
+    if (min->Equals(max)) {
+      auto single_value = compute::equal(field_expr, compute::literal(std::move(min)));
+
+      if (statistics.null_count() == 0) {
+        return single_value;
+      }
+      return compute::or_(std::move(single_value), is_null(std::move(field_expr)));
+    }
+
+    auto lower_bound = compute::greater_equal(field_expr, compute::literal(min));
+    auto upper_bound = compute::less_equal(field_expr, compute::literal(max));
+    compute::Expression in_range;
+
+    // If either minimum or maximum is NaN, it should be ignored for the
+    // range computation
+    if (IsNan(*min)) {

Review Comment:
   made the change, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sanjibansg commented on a diff in pull request #15125: ARROW-12264: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on code in PR #15125:
URL: https://github.com/apache/arrow/pull/15125#discussion_r1061882696


##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -98,6 +98,17 @@ Result<std::shared_ptr<SchemaManifest>> GetSchemaManifest(
   return manifest;
 }
 
+bool isNan(const Scalar& value) {
+  if (value.type->Equals(*float32())) {

Review Comment:
   Made the change, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sanjibansg commented on a diff in pull request #15125: ARROW-12264: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on code in PR #15125:
URL: https://github.com/apache/arrow/pull/15125#discussion_r1061882548


##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -98,6 +98,17 @@ Result<std::shared_ptr<SchemaManifest>> GetSchemaManifest(
   return manifest;
 }
 
+bool isNan(const Scalar& value) {

Review Comment:
   Thanks for the review, apologies for missing that out. Made the change!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] westonpace commented on pull request #15125: GH-28074: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on PR #15125:
URL: https://github.com/apache/arrow/pull/15125#issuecomment-1424647888

   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] ursabot commented on pull request #15125: GH-28074: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #15125:
URL: https://github.com/apache/arrow/pull/15125#issuecomment-1424922273

   Benchmark runs are scheduled for baseline = 0a7e7fb1e656bc01502720beb71b24da72a4a572 and contender = 518fc51e313c4ed886cb947d71fa3c8cd5a86187. 518fc51e313c4ed886cb947d71fa3c8cd5a86187 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Failed :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/f7f2abf9eec643c6a1af703a8b2d36d7...42be167fc3704faa872fff7461f785f5/)
   [Failed] [test-mac-arm](https://conbench.ursa.dev/compare/runs/6125aa97e0a34d2d8c6071dc8124c79d...c1ab4cc2083943b8ae5b7710b1da680f/)
   [Failed] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/d3adcdca29324a578db34ba1ecadf42e...28299edce2f6428fa36c83c0f6841526/)
   [Finished :arrow_down:0.1% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/c94702b580554f3982557a14bf78c973...b9d9ba45526e45fc80b10ca08567bb99/)
   Buildkite builds:
   [Finished] [`518fc51e` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/2352)
   [Failed] [`518fc51e` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/2380)
   [Finished] [`518fc51e` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/2350)
   [Finished] [`518fc51e` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/2371)
   [Failed] [`0a7e7fb1` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/2351)
   [Failed] [`0a7e7fb1` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/2379)
   [Failed] [`0a7e7fb1` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/2349)
   [Finished] [`0a7e7fb1` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/2370)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] westonpace commented on a diff in pull request #15125: GH-28074: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #15125:
URL: https://github.com/apache/arrow/pull/15125#discussion_r1097473086


##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -306,6 +277,69 @@ Result<bool> IsSupportedParquetFile(const ParquetFileFormat& format,
 
 }  // namespace
 
+std::optional<compute::Expression> ParquetFileFragment::EvaluateStatisticsAsExpression(
+    std::shared_ptr<Field> field, std::shared_ptr<parquet::Statistics> statistics) {

Review Comment:
   ```suggestion
   std::optional<compute::Expression> ParquetFileFragment::EvaluateStatisticsAsExpression(
       const Field& field, const parquet::Statistics& statistics) {
   ```
   
   Minor nit: prefer accepting `const X&` over `std::shared_ptr<X>` or even `const std::shared_ptr<X>&` if you can.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #15125: GH-28074: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15125:
URL: https://github.com/apache/arrow/pull/15125#issuecomment-1385294408

   :warning: GitHub issue #28074 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #15125: GH-28074: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15125:
URL: https://github.com/apache/arrow/pull/15125#issuecomment-1385294367

   * Closes: #28074


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sanjibansg commented on pull request #15125: GH-28074: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by "sanjibansg (via GitHub)" <gi...@apache.org>.
sanjibansg commented on PR #15125:
URL: https://github.com/apache/arrow/pull/15125#issuecomment-1421396319

   > This looks good. I don't see a problem with updating the arrow testing repo but it might be nice to keep changes isolated if you can.
   
   Yes, of course, and sorry, I will be more careful next time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #15125: ARROW-12264: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #15125:
URL: https://github.com/apache/arrow/pull/15125#discussion_r1060707883


##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -98,6 +98,17 @@ Result<std::shared_ptr<SchemaManifest>> GetSchemaManifest(
   return manifest;
 }
 
+bool isNan(const Scalar& value) {

Review Comment:
   Please, let's follow the coding conventions
   ```suggestion
   bool IsNan(const Scalar& value) {
   ```



##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -98,6 +98,17 @@ Result<std::shared_ptr<SchemaManifest>> GetSchemaManifest(
   return manifest;
 }
 
+bool isNan(const Scalar& value) {
+  if (value.type->Equals(*float32())) {
+    const FloatScalar& float_scalar = checked_cast<const FloatScalar&>(value);
+    return std::isnan(float_scalar.value);

Review Comment:
   Do we know for sure that `float_scalar.valid` is true?



##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -144,17 +159,23 @@ std::optional<compute::Expression> ColumnChunkStatisticsAsExpression(
       return compute::or_(std::move(single_value), is_null(std::move(field_expr)));
     }
 
-    auto lower_bound =
-        compute::greater_equal(field_expr, compute::literal(std::move(min)));
-    auto upper_bound = compute::less_equal(field_expr, compute::literal(std::move(max)));
+    auto lower_bound = compute::greater_equal(field_expr, compute::literal(min));

Review Comment:
   Perhaps the code could be better organized, to that NaN handling happens in a single place?
   (note that `min->Equals(max)` can only be false if there's a NaN...)



##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -98,6 +98,17 @@ Result<std::shared_ptr<SchemaManifest>> GetSchemaManifest(
   return manifest;
 }
 
+bool isNan(const Scalar& value) {
+  if (value.type->Equals(*float32())) {

Review Comment:
   Doing a full type equality check is unnecessarily slow here
   ```suggestion
     if (value.type->id() == Type::FLOAT) {
   ```



##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -135,6 +146,10 @@ std::optional<compute::Expression> ColumnChunkStatisticsAsExpression(
     min = maybe_min.MoveValueUnsafe();
     max = maybe_max.MoveValueUnsafe();
 
+    if (isNan(*min) && isNan(*max)) {

Review Comment:
   Please add a comment explaining the reasons why NaN needs to be handled explicitly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] wjones127 commented on pull request #15125: GH-28074: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #15125:
URL: https://github.com/apache/arrow/pull/15125#issuecomment-1400941192

   I believe I have generated a file that could be used for a test case here: https://github.com/apache/parquet-testing/pull/35
   
   Does that seem sufficient?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sanjibansg commented on a diff in pull request #15125: ARROW-12264: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on code in PR #15125:
URL: https://github.com/apache/arrow/pull/15125#discussion_r1061965235


##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -144,17 +159,23 @@ std::optional<compute::Expression> ColumnChunkStatisticsAsExpression(
       return compute::or_(std::move(single_value), is_null(std::move(field_expr)));
     }
 
-    auto lower_bound =
-        compute::greater_equal(field_expr, compute::literal(std::move(min)));
-    auto upper_bound = compute::less_equal(field_expr, compute::literal(std::move(max)));
+    auto lower_bound = compute::greater_equal(field_expr, compute::literal(min));

Review Comment:
   Apologies, I did not fully understand that. Can you elaborate on considering the inequality for min and max, and determining NaNs? Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] westonpace commented on pull request #15125: GH-28074: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on PR #15125:
URL: https://github.com/apache/arrow/pull/15125#issuecomment-1416396036

   Looks like the parquet tests are failing because they can't find the parquet file.  Do you maybe need to update the submodule version?  I haven't done this recently and am not sure of the right commands.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sanjibansg commented on a diff in pull request #15125: GH-28074: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by "sanjibansg (via GitHub)" <gi...@apache.org>.
sanjibansg commented on code in PR #15125:
URL: https://github.com/apache/arrow/pull/15125#discussion_r1099173966


##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -306,6 +277,69 @@ Result<bool> IsSupportedParquetFile(const ParquetFileFormat& format,
 
 }  // namespace
 
+std::optional<compute::Expression> ParquetFileFragment::EvaluateStatisticsAsExpression(
+    std::shared_ptr<Field> field, std::shared_ptr<parquet::Statistics> statistics) {

Review Comment:
   Made the change, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on pull request #15125: ARROW-12264: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #15125:
URL: https://github.com/apache/arrow/pull/15125#issuecomment-1369937342

   > Unfortunately, `ColumnChunkStatisticsAsExpression` isn't really a public method and so it might be tricky coming up with a test case without reproducing an offending parquet file which may not be easy to do.
   
   I agree we should ideally unit test `ColumnChunkStatisticsAsExpression`. Perhaps it can be exposed as an internal API?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] westonpace commented on a diff in pull request #15125: GH-28074: [C++][Dataset] Handle NaNs correctly in Parquet predicate push-down

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #15125:
URL: https://github.com/apache/arrow/pull/15125#discussion_r1100385860


##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -306,6 +282,65 @@ Result<bool> IsSupportedParquetFile(const ParquetFileFormat& format,
 
 }  // namespace
 
+std::optional<compute::Expression> ParquetFileFragment::EvaluateStatisticsAsExpression(
+    const Field& field, const parquet::Statistics& statistics) {
+  auto field_expr = compute::field_ref(field.name());
+
+  // Optimize for corner case where all values are nulls
+  if (statistics.num_values() == 0 && statistics.null_count() > 0) {
+    return is_null(std::move(field_expr));
+  }
+
+  std::shared_ptr<Scalar> min, max;
+  if (!StatisticsAsScalars(statistics, &min, &max).ok()) {
+    return std::nullopt;
+  }
+
+  auto maybe_min = min->CastTo(field.type());
+  auto maybe_max = max->CastTo(field.type());
+
+  if (maybe_min.ok() && maybe_max.ok()) {
+    min = maybe_min.MoveValueUnsafe();
+    max = maybe_max.MoveValueUnsafe();
+
+    // Since the minimum & maximum values are NaN, useful statistics
+    // cannot be extracted for checking the presence of a value within
+    // range
+    if (IsNan(*min) && IsNan(*max)) {
+      return std::nullopt;
+    }
+
+    if (min->Equals(max)) {
+      auto single_value = compute::equal(field_expr, compute::literal(std::move(min)));
+
+      if (statistics.null_count() == 0) {
+        return single_value;
+      }
+      return compute::or_(std::move(single_value), is_null(std::move(field_expr)));
+    }
+
+    auto lower_bound = compute::greater_equal(field_expr, compute::literal(min));
+    auto upper_bound = compute::less_equal(field_expr, compute::literal(max));
+    compute::Expression in_range;
+
+    // If either minimum or maximum is NaN, it should be ignored for the
+    // range computation
+    if (IsNan(*min)) {

Review Comment:
   ```suggestion
       if (min->Equals(max)) {
         auto single_value = compute::equal(field_expr, compute::literal(std::move(min)));
   
         if (statistics.null_count() == 0) {
           return single_value;
         }
         return compute::or_(std::move(single_value), is_null(std::move(field_expr)));
       }
   
       auto lower_bound = compute::greater_equal(field_expr, compute::literal(min));
       auto upper_bound = compute::less_equal(field_expr, compute::literal(max));
       compute::Expression in_range;
   
       // Since the minimum & maximum values are NaN, useful statistics
       // cannot be extracted for checking the presence of a value within
       // range
       if (IsNan(*min) && IsNan(*max)) {
         return std::nullopt;
       }
   
       // If either minimum or maximum is NaN, it should be ignored for the
       // range computation
       if (IsNan(*min)) {
   ```
   Per @pitrou 's earlier request, let's try and keep all the NaN handling grouped together.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org