You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/09/14 14:09:19 UTC

[GitHub] [arrow] lidavidm opened a new pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

lidavidm opened a new pull request #11152:
URL: https://github.com/apache/arrow/pull/11152


   This is intended to make it easier to use in bindings.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#issuecomment-921037609


   Squashed/rebased, thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] edponce commented on a change in pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
edponce commented on a change in pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#discussion_r709393805



##########
File path: cpp/src/arrow/compute/kernels/hash_aggregate_test.cc
##########
@@ -1425,6 +1425,51 @@ TEST(GroupBy, MinMaxDecimal) {
   }
 }
 
+TEST(GroupBy, MinOrMax) {
+  auto table =

Review comment:
       I agree (thus the "maybe"), if it already hitting those code paths independently which seems it is.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#issuecomment-920214068


   > LGTM. I added a few comments related to code organization.
   
   Thanks so much! I'll push some cleanups here in a bit. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] edponce commented on pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
edponce commented on pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#issuecomment-920213666


   LGTM. I added a few comments related to code organization.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#issuecomment-919188656


   https://issues.apache.org/jira/browse/ARROW-13849


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] pitrou commented on pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#issuecomment-921032344


   Now conflicts need to be resolved :-)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] edponce commented on a change in pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
edponce commented on a change in pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#discussion_r709524790



##########
File path: cpp/src/arrow/compute/kernels/hash_aggregate_test.cc
##########
@@ -1425,6 +1425,51 @@ TEST(GroupBy, MinMaxDecimal) {
   }
 }
 
+TEST(GroupBy, MinOrMax) {
+  auto table =
+      TableFromJSON(schema({field("argument", float64()), field("key", int64())}), {R"([
+    [1.0,   1],
+    [null,  1]
+                        ])",
+                                                                                    R"([
+    [0.0,   2],
+    [null,  3],
+    [4.0,   null],
+    [3.25,  1],

Review comment:
       Extend test case to get min/max between `NaN, Inf, null, value`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#issuecomment-920086207


   @edponce might you be able to take a look here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on a change in pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#discussion_r709430488



##########
File path: cpp/src/arrow/compute/kernels/aggregate_basic.cc
##########
@@ -282,6 +282,43 @@ Result<std::unique_ptr<KernelState>> MinMaxInit(KernelContext* ctx,
   return visitor.Create();
 }
 
+// For "min" and "max" functions: dispatch to the MinMax kernel.
+Result<std::unique_ptr<KernelState>> MinOrMaxInit(KernelContext* ctx,
+                                                  const KernelInitArgs& args) {
+  std::vector<ValueDescr> inputs = args.inputs;
+
+  ARROW_ASSIGN_OR_RAISE(auto func, GetFunctionRegistry()->GetFunction("min_max"));
+  ARROW_ASSIGN_OR_RAISE(auto kernel, func->DispatchBest(&inputs));
+
+  KernelInitArgs new_args{kernel, inputs, args.options};
+  return kernel->init(ctx, new_args);
+}
+
+Result<ValueDescr> MinOrMaxType(KernelContext*, const std::vector<ValueDescr>& descrs) {
+  // any[T] -> scalar[T]
+  return ValueDescr::Scalar(descrs.front().type);
+}
+
+template <uint8_t index>
+Status MinOrMaxFinalize(KernelContext* ctx, Datum* out) {
+  Datum temp;
+  RETURN_NOT_OK(checked_cast<ScalarAggregator*>(ctx->state())->Finalize(ctx, &temp));
+  *out = temp.scalar_as<StructScalar>().value[index];
+  return Status::OK();

Review comment:
       Good idea. I added this + moved some things to lambdas instead of free functions.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] edponce commented on a change in pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
edponce commented on a change in pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#discussion_r709377405



##########
File path: cpp/src/arrow/compute/kernels/aggregate_basic.cc
##########
@@ -782,6 +826,17 @@ void RegisterScalarAggregateBasic(FunctionRegistry* registry) {
 
   DCHECK_OK(registry->AddFunction(std::move(func)));
 
+  // Add min/max as convenience functions
+  func = std::make_shared<ScalarAggregateFunction>("min", Arity::Unary(), &min_or_max_doc,
+                                                   &default_scalar_aggregate_options);
+  aggregate::AddMinOrMaxAggKernel</*index=*/0>(func.get());
+  DCHECK_OK(registry->AddFunction(std::move(func)));

Review comment:
       If the comment of using enum is accepted, then use enum as template parameter.

##########
File path: cpp/src/arrow/compute/kernels/hash_aggregate_test.cc
##########
@@ -1425,6 +1425,51 @@ TEST(GroupBy, MinMaxDecimal) {
   }
 }
 
+TEST(GroupBy, MinOrMax) {
+  auto table =

Review comment:
       Maybe include tests of only `hash_min` and only `hash_max`.

##########
File path: cpp/src/arrow/compute/kernels/aggregate_basic.cc
##########
@@ -282,6 +282,43 @@ Result<std::unique_ptr<KernelState>> MinMaxInit(KernelContext* ctx,
   return visitor.Create();
 }
 
+// For "min" and "max" functions: dispatch to the MinMax kernel.
+Result<std::unique_ptr<KernelState>> MinOrMaxInit(KernelContext* ctx,
+                                                  const KernelInitArgs& args) {
+  std::vector<ValueDescr> inputs = args.inputs;
+
+  ARROW_ASSIGN_OR_RAISE(auto func, GetFunctionRegistry()->GetFunction("min_max"));
+  ARROW_ASSIGN_OR_RAISE(auto kernel, func->DispatchBest(&inputs));
+
+  KernelInitArgs new_args{kernel, inputs, args.options};
+  return kernel->init(ctx, new_args);
+}
+
+Result<ValueDescr> MinOrMaxType(KernelContext*, const std::vector<ValueDescr>& descrs) {
+  // any[T] -> scalar[T]
+  return ValueDescr::Scalar(descrs.front().type);
+}
+
+template <uint8_t index>
+Status MinOrMaxFinalize(KernelContext* ctx, Datum* out) {
+  Datum temp;
+  RETURN_NOT_OK(checked_cast<ScalarAggregator*>(ctx->state())->Finalize(ctx, &temp));
+  *out = temp.scalar_as<StructScalar>().value[index];
+  return Status::OK();

Review comment:
       It feels like the `index` value should not come from a template but an internal option, but given that we currently do not have a mechanism to extend options, I would suggest to use an enum instead of an explicit integer to make the code a bit more expressive.
   ```c++
   enum class MinMaxIndices : int8_t {
      MinIndex = 0,
      MaxIndex = 1,
   };
   ```

##########
File path: cpp/src/arrow/compute/kernels/hash_aggregate.cc
##########
@@ -1864,6 +1860,40 @@ Result<std::unique_ptr<KernelState>> MinMaxInit(KernelContext* ctx,
   return std::move(impl);
 }
 
+template <uint8_t index>
+Result<std::unique_ptr<KernelState>> MinOrMaxInit(KernelContext* ctx,
+                                                  const KernelInitArgs& args) {
+  std::vector<ValueDescr> inputs = args.inputs;
+
+  ARROW_ASSIGN_OR_RAISE(auto func, GetFunctionRegistry()->GetFunction("hash_min_max"));
+  ARROW_ASSIGN_OR_RAISE(auto kernel, func->DispatchBest(&inputs));
+
+  KernelInitArgs new_args{kernel, inputs, args.options};
+  return kernel->init(ctx, new_args);
+}
+
+template <uint8_t index>
+HashAggregateKernel MakeMinOrMaxKernel() {
+  HashAggregateKernel kernel;
+  kernel.init = MinOrMaxInit<index>;
+  kernel.signature = KernelSignature::Make(
+      {InputType(ValueDescr::ANY), InputType::Array(Type::UINT32)},
+      OutputType([](KernelContext* ctx,
+                    const std::vector<ValueDescr>& descrs) -> Result<ValueDescr> {
+        return ValueDescr::Array(descrs[0].type);
+      }));

Review comment:
       Nit: Here `OutputType` is expressed as a lambda function, so should [`MinMaxType`](https://github.com/apache/arrow/pull/11152/files#diff-fead2cb629ddc5a706a9383d4e64e24064cb3a2103e5e078bde7a11b57fd9654R297) be also a lambda.

##########
File path: cpp/src/arrow/compute/kernels/hash_aggregate.cc
##########
@@ -1864,6 +1860,40 @@ Result<std::unique_ptr<KernelState>> MinMaxInit(KernelContext* ctx,
   return std::move(impl);
 }
 
+template <uint8_t index>
+Result<std::unique_ptr<KernelState>> MinOrMaxInit(KernelContext* ctx,
+                                                  const KernelInitArgs& args) {
+  std::vector<ValueDescr> inputs = args.inputs;
+
+  ARROW_ASSIGN_OR_RAISE(auto func, GetFunctionRegistry()->GetFunction("hash_min_max"));
+  ARROW_ASSIGN_OR_RAISE(auto kernel, func->DispatchBest(&inputs));
+
+  KernelInitArgs new_args{kernel, inputs, args.options};
+  return kernel->init(ctx, new_args);
+}
+
+template <uint8_t index>
+HashAggregateKernel MakeMinOrMaxKernel() {
+  HashAggregateKernel kernel;
+  kernel.init = MinOrMaxInit<index>;
+  kernel.signature = KernelSignature::Make(
+      {InputType(ValueDescr::ANY), InputType::Array(Type::UINT32)},
+      OutputType([](KernelContext* ctx,
+                    const std::vector<ValueDescr>& descrs) -> Result<ValueDescr> {
+        return ValueDescr::Array(descrs[0].type);
+      }));
+  kernel.resize = HashAggregateResize;
+  kernel.consume = HashAggregateConsume;
+  kernel.merge = HashAggregateMerge;
+  kernel.finalize = [](KernelContext* ctx, Datum* out) {
+    ARROW_ASSIGN_OR_RAISE(Datum temp,
+                          checked_cast<GroupedAggregator*>(ctx->state())->Finalize());
+    *out = temp.array_as<StructArray>()->field(index);

Review comment:
       Similar comment as above on lambda vs `MinOrMaxFinalize`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on a change in pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#discussion_r710108171



##########
File path: cpp/src/arrow/compute/kernels/aggregate_basic.cc
##########
@@ -282,6 +282,43 @@ Result<std::unique_ptr<KernelState>> MinMaxInit(KernelContext* ctx,
   return visitor.Create();
 }
 
+// For "min" and "max" functions: dispatch to the MinMax kernel.
+Result<std::unique_ptr<KernelState>> MinOrMaxInit(KernelContext* ctx,
+                                                  const KernelInitArgs& args) {
+  std::vector<ValueDescr> inputs = args.inputs;
+
+  ARROW_ASSIGN_OR_RAISE(auto func, GetFunctionRegistry()->GetFunction("min_max"));

Review comment:
       Since we register min_max first, I've changed it to just pass the function to the init function, skipping the second lookup.

##########
File path: docs/source/cpp/compute.rst
##########
@@ -198,6 +198,10 @@ the input to a single output value.
 +---------------+-------+-------------+------------------------+----------------------------------+-------+
 | mean          | Unary | Numeric     | Scalar Decimal/Float64 | :struct:`ScalarAggregateOptions` |       |
 +---------------+-------+-------------+------------------------+----------------------------------+-------+
+| max           | Unary | Numeric     | Scalar Struct          | :struct:`ScalarAggregateOptions` |       |

Review comment:
       Thanks. I also fixed the output type column.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm closed pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
lidavidm closed pull request #11152:
URL: https://github.com/apache/arrow/pull/11152


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on a change in pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#discussion_r709391696



##########
File path: cpp/src/arrow/compute/kernels/hash_aggregate_test.cc
##########
@@ -1425,6 +1425,51 @@ TEST(GroupBy, MinMaxDecimal) {
   }
 }
 
+TEST(GroupBy, MinOrMax) {
+  auto table =

Review comment:
       I'm not sure there's a need to test the kernels separately, since they're being run separately by the nodes - I group them mostly because it's a lot of vertical visual noise to duplicate these tests otherwise (we should probably clean them up at some point).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] edponce commented on a change in pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
edponce commented on a change in pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#discussion_r709393805



##########
File path: cpp/src/arrow/compute/kernels/hash_aggregate_test.cc
##########
@@ -1425,6 +1425,51 @@ TEST(GroupBy, MinMaxDecimal) {
   }
 }
 
+TEST(GroupBy, MinOrMax) {
+  auto table =

Review comment:
       I agree (thus the "maybe"), if it is already hitting those code paths independently which seems it is.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] pitrou commented on a change in pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#discussion_r709906988



##########
File path: cpp/src/arrow/compute/kernels/aggregate_basic.cc
##########
@@ -282,6 +282,43 @@ Result<std::unique_ptr<KernelState>> MinMaxInit(KernelContext* ctx,
   return visitor.Create();
 }
 
+// For "min" and "max" functions: dispatch to the MinMax kernel.
+Result<std::unique_ptr<KernelState>> MinOrMaxInit(KernelContext* ctx,
+                                                  const KernelInitArgs& args) {
+  std::vector<ValueDescr> inputs = args.inputs;
+
+  ARROW_ASSIGN_OR_RAISE(auto func, GetFunctionRegistry()->GetFunction("min_max"));

Review comment:
       For the record, it's a bit of a pity to do a second registry lookup when arriving here.

##########
File path: cpp/src/arrow/compute/kernels/aggregate_basic.cc
##########
@@ -282,6 +282,43 @@ Result<std::unique_ptr<KernelState>> MinMaxInit(KernelContext* ctx,
   return visitor.Create();
 }
 
+// For "min" and "max" functions: dispatch to the MinMax kernel.
+Result<std::unique_ptr<KernelState>> MinOrMaxInit(KernelContext* ctx,
+                                                  const KernelInitArgs& args) {
+  std::vector<ValueDescr> inputs = args.inputs;
+
+  ARROW_ASSIGN_OR_RAISE(auto func, GetFunctionRegistry()->GetFunction("min_max"));
+  ARROW_ASSIGN_OR_RAISE(auto kernel, func->DispatchBest(&inputs));
+
+  KernelInitArgs new_args{kernel, inputs, args.options};
+  return kernel->init(ctx, new_args);
+}
+
+// For "min" and "max" functions: override finalize and return the actual value
+template <MinOrMax min_or_max>
+void AddMinOrMaxAggKernel(ScalarAggregateFunction* func) {
+  auto sig = KernelSignature::Make(
+      {InputType(ValueDescr::ANY)},
+      OutputType([](KernelContext*,
+                    const std::vector<ValueDescr>& descrs) -> Result<ValueDescr> {
+        // any[T] -> scalar[T]
+        return ValueDescr::Scalar(descrs.front().type);
+      }));
+
+  auto finalize = [](KernelContext* ctx, Datum* out) -> Status {
+    Datum temp;
+    RETURN_NOT_OK(checked_cast<ScalarAggregator*>(ctx->state())->Finalize(ctx, &temp));
+    *out = temp.scalar_as<StructScalar>().value[static_cast<uint8_t>(min_or_max)];

Review comment:
       DCHECK that the result is valid?

##########
File path: docs/source/cpp/compute.rst
##########
@@ -198,6 +198,10 @@ the input to a single output value.
 +---------------+-------+-------------+------------------------+----------------------------------+-------+
 | mean          | Unary | Numeric     | Scalar Decimal/Float64 | :struct:`ScalarAggregateOptions` |       |
 +---------------+-------+-------------+------------------------+----------------------------------+-------+
+| max           | Unary | Numeric     | Scalar Struct          | :struct:`ScalarAggregateOptions` |       |

Review comment:
       "max" before "mean"




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on a change in pull request #11152: ARROW-13849: [C++] Wrap min_max with min/max functions

Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #11152:
URL: https://github.com/apache/arrow/pull/11152#discussion_r709538359



##########
File path: cpp/src/arrow/compute/kernels/hash_aggregate_test.cc
##########
@@ -1425,6 +1425,51 @@ TEST(GroupBy, MinMaxDecimal) {
   }
 }
 
+TEST(GroupBy, MinOrMax) {
+  auto table =
+      TableFromJSON(schema({field("argument", float64()), field("key", int64())}), {R"([
+    [1.0,   1],
+    [null,  1]
+                        ])",
+                                                                                    R"([
+    [0.0,   2],
+    [null,  3],
+    [4.0,   null],
+    [3.25,  1],

Review comment:
       Done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org