You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "kou (via GitHub)" <gi...@apache.org> on 2024/12/21 05:50:03 UTC

[PR] GH-44579: [C++] Use array type to compute min/max statistics Arrow type [arrow]

kou opened a new pull request, #45094:
URL: https://github.com/apache/arrow/pull/45094

   ### Rationale for this change
   
   `arrow::ArrayStatistics` uses raw C++ types such as `int64_t` and `std::string` for min/max types. We need to convert raw C++ types to Arrow types when we use `arrow::ArrayStatistics` for generating statistics array. (GH-45038)
   
   We can't map `std::string` to an Arrow type. Because it may be `arrow::binary()`, `arrow::utf8()` or something.
   
   ### What changes are included in this PR?
   
   Use `arrow::DataType` information of associated array when we convert `arrow::ArrayStatistics`'s min/max raw C++ types to Arrow types.
   
   ### Are these changes tested?
   
   Yes.
   
   ### Are there any user-facing changes?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-44579: [C++] Use array type to compute min/max statistics Arrow type [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou merged PR #45094:
URL: https://github.com/apache/arrow/pull/45094


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-44579: [C++] Use array type to compute min/max statistics Arrow type [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #45094:
URL: https://github.com/apache/arrow/pull/45094#discussion_r1894566543


##########
cpp/src/arrow/array/statistics.h:
##########
@@ -34,22 +34,38 @@ namespace arrow {
 /// as Apache Parquet may have statistics. Statistics associated with
 /// data source can be read unified API via this class.
 struct ARROW_EXPORT ArrayStatistics {
+  /// \brief The type for maximum and minimum values. If the target
+  /// value exists, one of them is used. `std::nullopt` is used
+  /// otherwise.
   using ValueType = std::variant<bool, int64_t, uint64_t, double, std::string>;
 
   static const std::shared_ptr<DataType>& ValueToArrowType(
-      const std::optional<ValueType>& value) {
+      const std::optional<ValueType>& value,
+      const std::shared_ptr<DataType>& array_type) {
     if (!value.has_value()) {
       return null();
     }
 
     struct Visitor {
+      const std::shared_ptr<DataType>& array_type;
+
       const std::shared_ptr<DataType>& operator()(const bool&) { return boolean(); }
       const std::shared_ptr<DataType>& operator()(const int64_t&) { return int64(); }
       const std::shared_ptr<DataType>& operator()(const uint64_t&) { return uint64(); }
       const std::shared_ptr<DataType>& operator()(const double&) { return float64(); }
-      // GH-44579: How to support binary data?
-      const std::shared_ptr<DataType>& operator()(const std::string&) { return utf8(); }
-    } visitor;
+      const std::shared_ptr<DataType>& operator()(const std::string&) {
+        switch (array_type->id()) {
+          case Type::STRING:
+          case Type::BINARY:
+          case Type::FIXED_SIZE_BINARY:

Review Comment:
   Should we use `arrow::binary()` or `arrow::large_binary()` for `Type::FIXED_SIZE_BINARY`?
   Approximate min/max values may not be a fixed size binary...?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-44579: [C++] Use array type to compute min/max statistics Arrow type [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #45094:
URL: https://github.com/apache/arrow/pull/45094#issuecomment-2558011704

   :warning: GitHub issue #44579 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-44579: [C++] Use array type to compute min/max statistics Arrow type [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #45094:
URL: https://github.com/apache/arrow/pull/45094#issuecomment-2561592257

   I'll merge this in the next week if nobody objects this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-44579: [C++] Use array type to compute min/max statistics Arrow type [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #45094:
URL: https://github.com/apache/arrow/pull/45094#issuecomment-2585849693

   After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit fd1bf8e6886646ccd71a03de940190d5baca1ad2.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/35494965854) has more details. It also includes information about 10 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org