You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "kou (via GitHub)" <gi...@apache.org> on 2024/08/23 06:47:21 UTC

[PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

kou opened a new pull request, #43801:
URL: https://github.com/apache/arrow/pull/43801

   ### Rationale for this change
   
   If we can attach associated statistics to an array via `ArrayData`, we can use it in later processes such as query planning.
   
   If `ArrayData` not `Array` has statistics, we can use statistics in computing kernels.
   
   There was a concern that associated `arrow::ArrayStatistics` may be outdated if `arrow::ArrayData` is mutated after attaching `arrow::ArrayStatistics`. But `arrow::ArrayData` isn't mutable after the first population. So `arrow::ArrayStatistics` will not be outdated. We can require mutators to take responsibility for statistics.
   
   ### What changes are included in this PR?
   
   * Embed `arrow::ArrayStatistics` into `arrow::ArrayData` because `arrow::ArrayStatistics` is a lightweight data
   * Add `arrow::Array::statistics()` to get statistics attached in `arrow::ArrayData`
   
   This doesn't provide a new `arrow::ArrayData` constructor (`arrow::ArrayData::Make()`) that accepts `arrow::ArrayStatistics`. We can change `arrow::ArrayData::statistics` after we create `arrow::ArrayData`.
   
   ### Are these changes tested?
   
   Yes.
   
   ### Are there any user-facing changes?
   
   Yes.
   
   `arrow::Array::statistics()` is a new public API.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43801:
URL: https://github.com/apache/arrow/pull/43801#discussion_r1739920758


##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +197,7 @@ std::shared_ptr<ArrayData> ArrayData::Slice(int64_t off, int64_t len) const {
   } else {
     copy->null_count = null_count != 0 ? kUnknownNullCount : 0;
   }
+  copy->statistics = nullptr;

Review Comment:
   It's a good idea. I've add it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou merged PR #43801:
URL: https://github.com/apache/arrow/pull/43801


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43801:
URL: https://github.com/apache/arrow/pull/43801#issuecomment-2322850753

   If nobody objects this, I'll merge this in the next week.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #43801:
URL: https://github.com/apache/arrow/pull/43801#issuecomment-2317362619

   What just raise my curiousity is that `subslice` for array should discard the statistics?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43801:
URL: https://github.com/apache/arrow/pull/43801#discussion_r1733722587


##########
cpp/src/arrow/array/data.h:
##########
@@ -390,6 +395,9 @@ struct ARROW_EXPORT ArrayData {
 
   // The dictionary for this Array, if any. Only used for dictionary type
   std::shared_ptr<ArrayData> dictionary;
+
+  // The statistics for this Array.
+  ArrayStatistics statistics{};

Review Comment:
   I think that "lazy" means that "we can 'allocate' `ArrayStatistics` only when we use it" here.
   ("lazy" may not mean that lazy "computed".)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "bkietz (via GitHub)" <gi...@apache.org>.
bkietz commented on PR #43801:
URL: https://github.com/apache/arrow/pull/43801#issuecomment-2318900206

   Most statistics would be invalidated by slicing, such as the distinct and null counts. The minimum and maximum could be preserved, but would have to be demoted to inexact until recomputed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #43801:
URL: https://github.com/apache/arrow/pull/43801#issuecomment-2327032849

   After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 4ed5a149695644fe364466eabcae38d8dabfc090.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/29625176818) has more details. It also includes information about 29 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #43801:
URL: https://github.com/apache/arrow/pull/43801#discussion_r1737977798


##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +197,7 @@ std::shared_ptr<ArrayData> ArrayData::Slice(int64_t off, int64_t len) const {
   } else {
     copy->null_count = null_count != 0 ? kUnknownNullCount : 0;
   }
+  copy->statistics = nullptr;

Review Comment:
   A naive question, `if off == 0 && len == this->length`, should we keep this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43801:
URL: https://github.com/apache/arrow/pull/43801#discussion_r1823656006


##########
cpp/src/arrow/array/array_base.h:
##########
@@ -232,6 +232,14 @@ class ARROW_EXPORT Array {
   /// \return DeviceAllocationType
   DeviceAllocationType device_type() const { return data_->device_type(); }
 
+  /// \brief Return the statistics of this Array
+  ///
+  /// This just delegates to calling statistics on the underlying ArrayData
+  /// object which backs this Array.
+  ///
+  /// \return const ArrayStatistics&
+  std::shared_ptr<ArrayStatistics> statistics() const { return data_->statistics; }

Review Comment:
   I agree with you. I don't know why I missed `const` and `&` here...
   
   Let's add them: GH-44590



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #43801:
URL: https://github.com/apache/arrow/pull/43801#discussion_r1733114826


##########
cpp/src/arrow/array/data.h:
##########
@@ -390,6 +395,9 @@ struct ARROW_EXPORT ArrayData {
 
   // The dictionary for this Array, if any. Only used for dictionary type
   std::shared_ptr<ArrayData> dictionary;
+
+  // The statistics for this Array.
+  ArrayStatistics statistics{};

Review Comment:
   I'm interested in how can this being lazy computed 🤔 ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #43801:
URL: https://github.com/apache/arrow/pull/43801#discussion_r1823574725


##########
cpp/src/arrow/array/array_base.h:
##########
@@ -232,6 +232,14 @@ class ARROW_EXPORT Array {
   /// \return DeviceAllocationType
   DeviceAllocationType device_type() const { return data_->device_type(); }
 
+  /// \brief Return the statistics of this Array
+  ///
+  /// This just delegates to calling statistics on the underlying ArrayData
+  /// object which backs this Array.
+  ///
+  /// \return const ArrayStatistics&
+  std::shared_ptr<ArrayStatistics> statistics() const { return data_->statistics; }

Review Comment:
   The return type should be `std::shared_ptr<ArrayStatistics>&` and we should probably add `const ArrayStatistics` to the `shared_ptr` so that callers can't mutate the statistics through the shared pointer.



##########
cpp/src/arrow/array/array_base.h:
##########
@@ -232,6 +232,14 @@ class ARROW_EXPORT Array {
   /// \return DeviceAllocationType
   DeviceAllocationType device_type() const { return data_->device_type(); }
 
+  /// \brief Return the statistics of this Array
+  ///
+  /// This just delegates to calling statistics on the underlying ArrayData
+  /// object which backs this Array.
+  ///
+  /// \return const ArrayStatistics&
+  std::shared_ptr<ArrayStatistics> statistics() const { return data_->statistics; }

Review Comment:
   @kou 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "bkietz (via GitHub)" <gi...@apache.org>.
bkietz commented on code in PR #43801:
URL: https://github.com/apache/arrow/pull/43801#discussion_r1733087728


##########
cpp/src/arrow/array/data.h:
##########
@@ -390,6 +395,9 @@ struct ARROW_EXPORT ArrayData {
 
   // The dictionary for this Array, if any. Only used for dictionary type
   std::shared_ptr<ArrayData> dictionary;
+
+  // The statistics for this Array.
+  ArrayStatistics statistics{};

Review Comment:
   ```suggestion
     std::unique_ptr<ArrayStatistics> statistics;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43801:
URL: https://github.com/apache/arrow/pull/43801#issuecomment-2313870664

   OK. I've changed to a pointer instead of embedding `ArrayStatistics` to `ArrayData`.
   But I used `std::shared_ptr` not `std::unique_ptr` because we want to export `ArrayStatistics` from `arrow::Array`. If we use `std::unique_ptr`, `arrow::Array::statistics()` also needs to return `std::unique_ptr`. It will be difficult to use. So I used `std::shared_ptr`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43801:
URL: https://github.com/apache/arrow/pull/43801#issuecomment-2306407813

   :warning: GitHub issue #43797 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #43801:
URL: https://github.com/apache/arrow/pull/43801#discussion_r1739707043


##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +197,7 @@ std::shared_ptr<ArrayData> ArrayData::Slice(int64_t off, int64_t len) const {
   } else {
     copy->null_count = null_count != 0 ? kUnknownNullCount : 0;
   }
+  copy->statistics = nullptr;

Review Comment:
   Could we add comment in `statistics` or subslice? This would helps user on this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43801:
URL: https://github.com/apache/arrow/pull/43801#discussion_r1739651361


##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +197,7 @@ std::shared_ptr<ArrayData> ArrayData::Slice(int64_t off, int64_t len) const {
   } else {
     copy->null_count = null_count != 0 ? kUnknownNullCount : 0;
   }
+  copy->statistics = nullptr;

Review Comment:
   Hmm. I think that we don't need to do it:
   
   * "Slice() always discards statistics" is simple and easy to understand
   * In general, `Slice(0, length)` will not be used because it's just a `Copy()` and an user should use `Copy()` instead of `Slice(0, length)`
   
   If it may be useful, we can revisit this later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43801:
URL: https://github.com/apache/arrow/pull/43801#issuecomment-2320047981

   Good catch! I forgot the `Slice()` case...
   I've discarded statistics on `Slice()`. If an user wants to re-use (min/max) statistics of the original `ArrayData`, the user is responsible it.
   
   I noticed that `ArrayData::CopyTo()` needs to copy statistics. I've done it too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43797: [C++] Attach `arrow::ArrayStatistics` to `arrow::ArrayData` [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43801:
URL: https://github.com/apache/arrow/pull/43801#issuecomment-2311635975

   @pitrou @bkietz @felipecrv What do you think about this approach?
   See also the discussion at https://github.com/apache/arrow/pull/43705#issuecomment-2302096692 and comments followed by it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org